A- A+
Alt. Display

# Ontology Usability Scale: Context-aware Metrics for the Effectiveness, Efficiency and Satisfaction of Ontology Uses

#### Xiaogang Ma,

##### Department of Computer Science, University of Idaho, 875 Perimeter Drive, MS 1010, Moscow, ID 83844-1010, US

Xiaogang (Marshall) Ma is an assistant professor of computer science at the University of Idaho. He received his Ph.D. degree of Earth Systems Science and GIScience from University of Twente, Netherlands in 2011. Before joining UIdaho he was an associate research scientist of Data Science and Semantic eScience at Rensselaer Polytechnic Institute. His research focuses on deploying data science in the Semantic Web to support cross-disciplinary collaboration and scientific discovery, with broad interests in participatory knowledge engineering, data interoperability and provenance, and visualized exploratory analysis of Big and Small Data. Ma is active in several international societies of data science and geoinformatics, and is experienced in research management and community service. He received the IAMG Vistelius Research Award in 2015 and the inaugural ICSU-WDS Data Stewardship Award in 2014. He won the ESIP Funding Friday Competition Award twice in 2013 and 2012. He can be reached on twitter at @MarshallXMa.

## Abstract

Both ontology builders and users need a way to evaluate ontologies in terms of usability, but existing ontology evaluation approaches do not fit this purpose. We propose the Ontology Usability Scale (OUS), a ten-item Likert scale derived from statements prepared according to a semiotic framework and an online poll in the Semantic Web community to provide a practical way of ontology usability evaluation. Case studies were conducted to bookkeep current usability evaluation results for ontologies expecting revisions in the future, and discussions of the poll results are presented to help proper use and customization of the OUS.

Keywords:
How to Cite: Ma, X., Fu, L., West, P. and Fox, P., 2018. Ontology Usability Scale: Context-aware Metrics for the Effectiveness, Efficiency and Satisfaction of Ontology Uses. Data Science Journal, 17, p.10. DOI: http://doi.org/10.5334/dsj-2018-010
Published on 10 May 2018
Accepted on 09 Apr 2018            Submitted on 16 Feb 2018

## 1 Introduction

Along with the rapid development of the Linked Open Data, ontologies have been increasingly built and used in various fields (Bikakis et al., 2013). Ontology usability arises as an issue of interest for many stakeholders in the Linked Open Data campaign, including both ontology builders and users. For instance, ontology builders may want to hear evaluation and feedback on the usability of their ontologies and then take actions on the revision. Such evaluation and feedback may also be beneficial to ontology users because they help identify the suitable ontologies and estimate the costs of using them for specific applications. In a broader perspective, the evaluation of ontology usability is a way of communication for developing better ontologies.

We need a set of criteria for the evaluation of ontology usability, through which people will be able to describe and assess an ontology from different aspects. The items in the criteria, however, are decided by the understandings about the meaning of usability. We adopt the definition given in the international standard ISO 9241-11 (ISO, 1998), which has received endorsements from various domains: “[Usability is] the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.” The background of this definition is Human-Computer Interaction and the definition represents a user-centered point of view (Jokela et al., 2003). The definition indicates that usability is not about the product itself (or, its quality), but about the activity of a user using it, so usability depends on the goal of the user and the context of the use. With satisfaction as one of its attributes, defined as “freedom from discomfort, and positive attitude to the use of the product” (Jokela et al., 2003), usability is inevitably subjective.

Therefore, existing criteria for assessing ontology quality do not necessarily fit in the evaluation of ontology usability. Nevertheless, some of them can serve as strong usability indicators. For example, Gruber (1995) discussed several elements for ontology design: clarity, coherence, extendibility, minimal encoding bias, and minimal ontological commitment. Those elements are all good principles for ontology quality evaluation. In a recent work, Fox and Lynnes (2015) added a few other items, namely contextual relevance, maturity, intended use, and fitness for use, to Gruber’s list to cover the contextual and subjective aspects of ontology usability. In this paper we propose the Ontology Usability Scale (OUS) for the evaluation of ontology usability. The evaluation metrics are reflected in a short list of statements, which were derived from an online poll in the Semantic Web community. In the reminder of the paper, we will introduce our thoughts on semiotics when preparing a long list of statements for the poll. By using the outputs of the poll, we will also analyze the community’s concerns on ontology usability.

## 2 Related Work

Although it is hard to find related work specifically on evaluating ontology usability, the general evaluation of ontologies, however, has already received attention even before the introduction of Resource Description Framework (RDF) (Miller, 1998) and Web Ontology Language (OWL) (McGuinness and Van Harmelen, 2004). For example, Gruninger and Fox (1995) proposed a method to check the completeness of an ontology with respect to a set of competency questions. Competency questions are the questions that the ontology is designed to answer, so this method checks whether “the right things are done”, and does not benefit people who later want to reuse the ontology.

With the major Semantic Web standards and heavily reused upper ontologies being available, representing domain knowledge with ontologies became a common practice and it became more and more likely to see several different conceptualizations of the same domain. Therefore, ontology evaluation methods for the purpose of selecting and reusing existing ontologies for new applications were greatly needed.

A popular approach to do this kind of evaluation is to define several criteria for decision making, evaluate the ontology in question on each criterion by giving a numerical score, and then compute the overall score for the ontology as a weighted sum of the per-criterion scores. Such methods are called multiple-criteria approaches in Brank et al. (2005). For example, Fox et al. (1995) proposed a set of criteria including generality, completeness, perspicuity, etc. Gomez-Perez (2001) in her paper published in 2001 pointed out the lack of interest in evaluation issues in the ontological engineering community at that time. She also pointed out that tools, tutorials and case studies are critical for ontology engineers to assess the usability of an existing ontology. Nevertheless, the paper does not give an example of ontology usability evaluation, it instead evaluates the Standard-Unit Ontology in terms of consistency, completeness and conciseness, so the evaluation result is not directly related to the effectiveness, efficiency and satisfaction of the users when they reuse a certain ontology. An ontology may be consistent (i.e. without any contradictory assertion), complete (i.e. without any missing definition) and concise (i.e. without any unnecessary definition), but still be unusable or very cumbersome to use (e.g. due to bad documentation).

Lozano-Tello et al. presented the ONTOMETRIC method (Lozano-Tello and Gomez-Perez, 2004; Lozano-Tello et al., 2003), which compares ontologies with a taxonomy of 160 characteristics organized in a multilevel tree-shaped framework. The final score for an ontology is calculated as the weighted sum of the scores given to each of the leaf node characteristics, through aggregations at each of the internal nodes governing aspects and sub-aspects of the ontology characteristics. It could be imagined that the scoring and weight assignment would take a lot of time and be easily biased by the viewpoint of the scorer, as pointed out by Hartmann et al. (2005). Similar with ONTOMETRIC, our approach also requires a scorer to get a single numerical score for each ontology in question, but we aimed to provide the scorer a Likert scale (a questionnaire asking for degrees of agreement with a list of statements) consisting of around 10 items, instead of a huge scoring form with pending weights for each item. We found a small portion of the 160 characteristics directly related to usability, and drew them out as our candidate usability metrics.

Different from the studies mentioned above, Burton-Jones et al. (2005) proposed a set of 10 attributes that fall into four metrics suites (i.e. syntactic, semantic, pragmatic and social quality) of a semiotic framework. They tried to find objective indicators to assess each recognized metric to eliminate the need of human reviewers. For example, the Lawfulness dimension of the Syntax metric is indicated by the percentage of correct syntax per class and property. The authors suggested that the evaluation could be extended to cover application-centered assessment of the quality of an ontology for use in a specific task. We create our list of candidate usability metrics by thinking in the aspects of syntax, semantics and pragmatics. We assume that the difference of social quality (authority and history) between two ontologies would dwarf the overall impact of these three other aspects. In fact, only ontologies of similar social quality are worth comparing. If one of them has significantly more authority (i.e. more ontologies rely on it) or longer history and more usage, it would be an obvious winner. We also found it hard, if not impossible, to assess some of the important metrics with objective indicators. For example, the quality of documentation is very important to the usability of the ontology, but it is hardly possible to be measured in any objective way, so we argue that it requires the participation of human reviewers to get meaningful usability assessments.

The semiotic framework was also used in Gangemi et al. (2006). The authors organized the criteria for ontology evaluation and selection with a semiotic meta-ontology called O2 and the oQual evaluation ontology which involves concepts and relations relevant to ontology evaluation and selection. Therefore, evaluation based on the oQual ontology goes beyond the mere calculation of a weighted sum, but also contains reasoning based on the evaluation ontology. Goals of reusing the ontology and trade-off rules among conflicting goals must be defined formally and in a way that connects with O2 and oQual in order to follow their approach, which limits its application to ontology experts only. The target user group of our metrics is anyone who wants to select an ontology suitable to an application (meaning the user can apply the selected ontology to his/her use case with the most aggregated effectiveness, efficiency and satisfaction among all the candidate ontologies), so we require much less ontological expertise of our users than their approach.

Casellas (2009) used the System Usability Scale (SUS, (Brooke, 1996)) to evaluate the usability of the Ontology of Professional Judicial Knowledge (See Chapter 2 of Casellas, 2009) for an introduction of the ontology). Our approach differs with Casellas (2009) in how we select the statements in the questionnaire. Casellas directly tailored the 10 items in SUS, but we think that the set of statements that best indicate the usability of ontologies may need to consider more aspects. We created a pool of 29 candidate statements by adapting several resources, including those used in Casellas (2009). Then we built the usability scale from those candidate statements through some community efforts, i.e. we gathered preferences among the Semantic Web community through an online poll. The result of the poll verified our thoughts since it differed a lot from the statement set in Casellas (2009).

## 3 Approach

The intuition behind our approach to evaluating ontology usability comes from SUS, a ten-item Likert scale whose usage is recommended by the UsabilityNet project as “it is very robust and has been extensively used and adapted. Of all the public domain questionnaires, this is the most strongly recommended”1. We hope to have such a concise scale that is applicable for ontology usability evaluation.

A direct adaptation of SUS for ontology (i.e. replace “system” with “ontology” in the questionnaire) does not cover all the aspects in our understanding of the ontology usability. Therefore, besides those items adapted from SUS, we collected usability evaluation statements from ONTOMETRIC (Lozano-Tello and Gomez-Perez, 2004) and Casellas (2009) to set up a large pool of statements. When collecting those statements we also refer to the semiotic framework discussed in Burton-Jones et al. (2005) and Gangemi et al. (2006), but our understanding of the syntax, semantics and pragmatics is slightly different because in our work the focus is the usability. We consider syntax is relevant to the machine-readable encoding and logic of the content of an ontology; semantics is relevant to the conceptual model and documentation; and pragmatics is relevant to the first hand experience of using the ontology in practice. Table 1 shows the grouped statements in the large pool. Note that we changed some statements originally in negative forms to positive forms so that all the statements are desired features of a highly usable ontology. Each feature represented by one of the statements can be represented in either the positive form or the negative form. Both forms are found in our original statement set. Two of these statements, numbered 23 and 26 in Table 1, are even in the two forms of exactly the same feature. To eliminate bias caused by the statement representation form and avoid listing the same feature twice (in both positive and negative forms) in the questionnaire, we decided to normalize every statement to its positive form.

Table 1

The large pool of statements for ontology usability evaluation.

 Syntax (Content structure) 1. I found the various concepts in this ontology well integrated 2. The ontology misses some important concepts – Changed to “The ontology has all the important concepts included” in the survey 3. The ontology has unnecessary concepts – Changed to “The ontology does not have unnecessary concepts” 4. I found the ontology unnecessarily complex – Changed to “I found the ontology brief but comprehensive” 5. I thought there was too much inconsistency in this ontology – Changed to “I found the various parts of this ontology well integrated” 6. I found the formal specification of concepts in this ontology coincides with their descriptions in natural language 7. I found the formal specification of relations in this ontology coincides with their descriptions in natural language 8. I think the attributes in this ontology describe the concepts well 9. I think the relations in this ontology relate appropriate concepts 10. I found the subclasses in this ontology are properly defined 11. I found the disjoint classes in this ontology are properly asserted Semantics (Documentation) 12. The purpose of this ontology is clear 13. I am confident I understand the conceptualization of the ontology 14. I need to ask a lot of questions before I could understand the conceptualization of the ontology – Changed to “I could understand the conceptualization of the ontology without asking a lot of questions” 15. I find the ontology easy to understand 16. The annotations are helpful 17. I found the concepts in this ontology properly described in natural language 18. I found the relations in this ontology properly described in natural language 19. I need further theoretical support to understand this ontology – Changed to “I do not need further theoretical support to be able to understand this ontology” 20. I would imagine that most domain experts would understand this ontology very quickly Pragmatics (First-hand experience) 21. I would imagine that most people would learn to use this ontology very quickly 22. I needed to learn a lot of things before I could get going with this ontology – Changed to “I do not need to learn any extra things before I could get going with this ontology” 23. I thought the ontology was easy to use 24. It is clear to me how to use this ontology 25. I felt very confident using the ontology 26. I found the ontology very cumbersome to use – Deleted in the survey since it is the negative form of statement 23 and we decided to present all the statements in their positive forms only. 27. I think that I would need the support of a person experienced with this ontology to be able to use it – Changed to “I do not need the support of a person experienced with this ontology to be able to use it” 28. I need some more examples than provided in the documentation to make sure how to use the ontology – Changed to “I think the documentation provides sufficient examples for me to make sure how to use the ontology” 29. I think that I would like to use this ontology frequently 30. I think that I could contribute to this ontology

The long list of statements can help provide a comprehensive usability evaluation, but the burden caused by the number of statements can be an issue, so we reached out to the semantic web community to ask for a poll for selecting 10 representative statements. We sent out invitations to the semantic web working group of the Federation of Earth Science Information Partners, the semantic web group on Facebook, and also colleagues at Tetherless World Constellation at Rensselaer Polytechnic Institute. To avoid confusion in reading the statements, we changed the forms of a few of them (see notes in Table 1) to make all the statements have a positive form, i.e. towards the goodness of an ontology instead of shortcomings. Moreover, we mixed the sequence of those statements in the survey and did not show the three groups of syntax, semantics and pragmatics of those statements in order to avoid any bias that may be led to the survey participants. We received 18 valid responses in 7 days, and the top 11 statements from the poll and the votes they each received are shown in Table 2.

Table 2

The top 11 statements resulted from an online poll and the votes to each statement received.

1 I think the documentation provides sufficient examples for me to make sure how to use the ontology. Pragmatics 14
2 The purpose of this ontology is clear. Semantics 9
3 I found the concepts in this ontology properly described in natural language. Semantics 9
4 I think the relations in this ontology relate appropriate concepts. Syntax 8
5 I am confident I understand the conceptualization of the ontology. Semantics 8
6 I would imagine that most domain experts would understand this ontology very quickly. Semantics 7
7 I think the attributes in this ontology describe the concepts well. Syntax 7
8 I found the subclasses in this ontology are properly defined. Syntax 7
9 I found the relations in this ontology properly described in natural language. Semantics 7
10 I found the formal specification of concepts in this ontology coincides with their descriptions in natural language. Syntax 7
11 I do not need the support of a person experienced with this ontology to be able to use it. Pragmatics 7

In Table 2 we can see that 5 of the top 11 statements are about semantics, 4 for syntax and the left 2 for pragmatics, which is a strong indication that usability of ontology is mostly about semantics and syntax. This also supports our above discussion that the SUS probably would not work well on ontologies because it is mostly about pragmatics.

We compiled our 10-item Likert scale for ontology usability evaluation based on the above result, as shown in Table 3.

Table 3

Ontology usability evaluation questionnaire (draft).

Number Statement 1 2 3 4 5

1 I think the documentation provides sufficient examples for me to make sure how to use the ontology.
2 The purpose of this ontology is clear.
3 I found the concepts and relations in this ontology properly described in natural language.
4 I think the relations in this ontology relate appropriate concepts.
5 I am confident I understand the conceptualization of the ontology.
6 I would imagine that most domain experts would understand this ontology very quickly.
7 I think the attributes in this ontology describe the concepts well.
8 I found the subclasses in this ontology are properly defined.
9 I found the formal specification of concepts and relations in this ontology coincides with their descriptions in natural language.
10 I do not need the support of a person experienced with this ontology to be able to use it.

In Table 3, closely related statements about concepts and relations were merged together. i.e., statements 3 and 9 in Table 2 were merged to “I found the concepts and relations in this ontology properly described in natural language”, and statement 10 was changed to “I found the formal specification of concepts and relations in this ontology coincides with their descriptions in natural language”. We did this based on the assumption that concepts are equally important as and considered together with relations in an ontology.

A scale from 1 to 5 was used to indicate “strongly disagree”, “disagree”, “neutral”, “agree” and “strongly agree” against each statement in Table 3, so higher scores mean better usability. To assess the usability of an ontology using this form, a scorer needs to give a score indicating his/her degree of agreement for each statement, denoted as s1, s2, …, s10, then the total score st is calculated as:

(1)
${s}_{t}=\sum _{i=1}^{10}{S}_{i}$

which ranges from 10 to 50.

To further improve the questionnaire, we use positive and negative forms of statements in Table 3 alternatingly to make scorers more attentive when they fill out the form. The adapted and reorganized statements are shown in Table 4.

Table 4

Recommended ontology usability scale.

Number Statement 1 2 3 4 5

1 The purpose of this ontology is clear.
2 I need more examples than provided in the documentation to make sure how to use the ontology.
3 I found the concepts and relations in this ontology properly described in natural language.
4 There is inconsistency between the formal specification of concepts and relations in this ontology and their descriptions in natural language.
5 I would imagine that most domain experts would understand this ontology very quickly.
6 I think that I would need the support of a person experienced with this ontology to be able to use it.
7 I am confident I understand the conceptualization of the ontology.
8 The attributes in this ontology fail to describe the concepts properly.
9 I think the relations in this ontology relate appropriate concepts.
10 I think the class hierarchy of this ontology needs better organization.

In Table 4, the statements at odd numbered positions are all in a positive form and those at even numbered positions are all in a negative form. To use this form, Equation 1 needs to be changed to:

(2)

which still ranges from 10 to 50. A higher score indicates a higher usability.

## 4 Case Study and Evaluation

We used the developed OUS, i.e. statements in Table 4 for a case study of ontology usability evaluation within the Tetherless World Constellation at Rensselaer Polytechnic Institute. The case study was carried out as an anonymous online survey, in which each participant was asked to choose an ontology and assign a score to each statement. The outputs of the survey are listed in Table 5. Since revisions are currently undergoing to update the ontologies of Deep Carbon Observatory (DCO) and Global Change Information System (GCIS), we will be able to apply the developed OUS to their later versions to check if the revisions are effective in terms of usability. Comparisons among ontologies with similar intended uses and different versions of the same ontology are made easy with OUS since each ontology is given an overall score calculated from the answers given by each reviewer. Scoring an OUS form (Table 4) does not require much of the reviewers’ time, but the collected answers provide simple yet comprehensive assessments of the usability of the ontologies in question. In addition to the total score, we can also analyze the scores on the same statement from several evaluation cases of a same ontology.

Table 5

Results of ontology usability evaluation in a case study.

Evaluation case Ontology Score on each statement
Total score st
S1 S2 S3 S4 S5 S6 S7 S8 S9 S10

Case 1 DCO 5 4 3 2 4 3 5 2 4 2 38
Case 2 DCO 5 4 4 1 3 2 5 1 5 2 42
Case 3 DCO 3 1 4 3 2 2 2 4 3 5 29
Case 4 GCIS 4 2 4 2 4 3 4 2 4 3 38
Case 5 GCIS 5 3 4 1 5 2 5 1 5 3 44
Case 6 SIO 5 5 5 2 5 5 2 1 5 3 36
Case 7 VSTO 1 5 1 3 2 4 4 2 4 2 26

## 5 Discussion

The resulting ontology usability scale of this study covers topics of syntax, semantics and pragmatics and addresses the issue of evaluating the usability of an ontology in a certain context. The list of statements in the current scale (Table 4) is based on a survey. It has a concise structure and is easy to use in practice. The scale can be used by all stakeholders who participated in the development, application, and revision of an ontology, and the result can be used to improve the ontology.

Semiotics is the study of signs. It is applicable to ontologies because ontologies are sign systems to represent knowledge. The division of semiotics into semantics, syntactics and pragmatics was contributed by Morris (1938). According to Morris, semantics is the study of the relation of signs to the things they refer to (their designata); syntactics is the study of the relation of signs to one another; pragmatics studies the relation of signs to their interpreters. In the case of ontology, we define semantics as the mapping of domain knowledge to ontological elements such as classes and relations, or the meaning of these elements, conveyed through the conceptual model and documentation. Syntactics (or syntax according to Burton-Jones et al., 2005) is defined as the way the ontological elements are organized, usually with terms in RDF Schema (RDFS) (Brickley and Guha, 2014) or OWL. Pragmatics is the relation of ontologies to their users, so it is about the activity of a user using an ontology.

According to the above definitions, it seems only the pragmatical dimension of semiotics is relevant to ontology usability since it is the only dimension about the ontology using activity rather than the ontology itself, and usability is about certain attributes of the activity of using a certain product rather than attributes of the product itself according to ISO 9241-11 (ISO, 1998). However, unlike simple signs which their users interpret out of intuition and experience, ontological terms require users to learn their semantical and syntactical features in order to interpret, typically through learning the term organization and reading the documentation. Therefore, all the three dimensions of semiotics are relevant to ontology usability. In fact, the survey result shown in Table 2 even indicated that the semantical and syntactical aspects may be more important than the pragmatical aspect, since only 2 out of the 11 top selections fall in the pragmatics group.

ISO 9241-11 (ISO, 1998) listed the following three aspects of usability, effectiveness is about accuracy and completeness of the result of that activity, efficiency is the resources expended during the process, such as time and effort of learning and creating solutions, and satisfaction is the freedom from discomfort and the positive attitude towards the use of the product. As we tried to decide which semiotical aspect impact which usability aspect, we found that each semiotical aspect may impact every usability aspect. For example, missing or insufficiently illustrated ontological terms (semantical aspect) may cause the user unable to complete his or her tasks (effectiveness), to spend more time learning the conceptualization (efficiency), and/or to feel discomfortable (satisfaction). Inconsistent or counter-intuitive organization of the ontological elements (syntactical) may have the same effect in terms of effectiveness, efficiency and satisfaction, and the pragmatical aspect aligns well with the overall usability. Therefore, the semiotical aspects and the usability aspects are closely related, so it is reasonable to classify usability criteria with semiotical aspects.

The survey result brings feedback and inspiration to ontology developers. As indicated by the top selections in Table 2, stating the purpose of the ontology explicitly, describing classes and relations in detail and providing abundant examples in the documentation will greatly help users understand and use the ontology. The purpose of an ontology is important probably because of the close relationship between usability and purpose. Usability is inherently associated with fitness, and fitness, as summed up by Terry Pratchett in his novel “Moving Pictures”, means “appropriateness to a purpose” (Pratchett, 1990) (quoted in Brooke, 1996). Besides the statements pool in Table 1, in the online survey we also invited participants to write down any additional statements from their point of view. One suggestion is about the provenance of components in an ontology, such as the cited source in the definition of a class or property, and the person who asserts the definition, etc. Another issue is about the serialization language of ontologies. It was mentioned that if an ontology is not serialized in a simple format such as Turtle, it can be difficult for a user to read and understand. Another participant proposed the issue of ontology maintenance/sustainability, such as the stability of the ontology over time and the level of maintenance support that the ontology has. In a previous publication (Ma and Fox, 2013) we discussed that to achieve better ontology applications, people need to balance the expressivity, implementability and maintainability of the ontology. The maintainability or sustainability is relevant to the usability of an ontology in a long-term period.

In the online survey we also received active feedback about the organization and form of the statements. In the survey preparation we tried to group the statements into three categories following a semiotic framework, while in the survey we did not show the groups and listed the statements in a random order. Our intention is to avoid any bias that may be caused by those pre-defined groups. It was interesting to see that several participants suggested the statement pool should be categorized, especially from the point of view of an ontologist. Another participant suggested that there can be a separation of statements for subject matter experts and end users.

Survey participants also commented on the positive and negative forms of the statements. From the comments we could see that it is okay to use both forms in a questionnaire, but we should avoid duplicated statements, such as “I thought the ontology was easy to use” and “I found the ontology very cumbersome to use.” Considering the feedback, we organized two sets of statements for ontology usability evaluation in Tables 3 and 4, one with only statements in positive form and the other with both positive and negative forms.

Besides the top 11 mostly voted statements in Table 2, we also took a review of the least voted statements in the online poll (Table 6). Most of them have either an extreme or vague meaning. Statement 1 in Table 6 is adapted from its negative form “I needed to learn a lot of things before I could get going with this ontology”; statement 2 was originally “The ontology has unnecessary concepts”; statement 3 overlaps a lot in meaning with “I found the concepts/relations in this ontology properly described in natural language” but is less clear; statement 4 is very similar to “I would imagine that most domain experts would understand this ontology very quickly”, and statement 5 similar to “I found the various concepts in this ontology well integrated” (which was selected 4 times).

Table 6

The least voted statements in the online poll.

1 I do not need to learn any extra things before I could get going with this ontology. Pragmatics 0
2 The ontology does not have unnecessary concepts. Syntax 2
3 The annotations are helpful. Semantics 2
4 I would imagine that most people would learn to use this ontology very quickly. Pragmatics 2
5 I found the various parts of this ontology were well integrated. Syntax 2

We also collected information about what ontologies the surveyees had worked with, the top ones are shown in Table 7.

Table 7

Ontologies that the online surveyees worked with.

Ontology name Number of people worked with it

SWEET 11
Dublin Core 8
FOAF 7
SKOS 5
PROV-O 3
GCIS 3

We found that users of domain ontologies such as SWEET have different preferences on statements from users of general ontologies such as Dublin Core. Top 12 selections for SWEET users listed in Table 8 are very similar with those in Table 2, with only a few minor differences. For example, statement 6 in Table 2 (“I would imagine that most domain experts would understand this ontology very quickly”) is missing in Table 8, but it ranks 13th among SWEET users; statement 10 and 12 in Table 8 is missing in Table 2, but they rank 12th and 13th among all the statements so did not make it to the top 11 in Table 2.

Table 8

Top 12 statements selected by SWEET users.

1 The purpose of this ontology is clear. 8
2 I think the documentation provides sufficient examples for me to make sure how to use the ontology. 8
3 I think the relations in this ontology relate appropriate concepts. 7
4 I found the concepts in this ontology properly described in natural language. 6
5 I am confident I understand the conceptualization of the ontology. 6
6 I think the attributes in this ontology describe the concepts well. 5
7 I found the subclasses in this ontology are properly defined. 5
8 I do not need the support of a person experienced with this ontology to be able to use it. 5
9 I found the relations in this ontology properly described in natural language. 4
10 I found the ontology brief but comprehensive. 4
11 I found the formal specification of concepts in this ontology coincides with their descriptions in natural language. 4
12 I find the ontology easy to understand. 4

But among the top three selections by Dublin Core users shown in Table 9, two of them (“I think that I would like to use this ontology frequently” and “It is clear to me how to use this ontology”) are missing in Table 2, so for ontologies designed to be used across different domains, statements 2 and 3 in Table 9 could be considered for the usability scale.

Table 9

Top 3 statements selected by Dublin Core users.

1 I think the documentation provides sufficient examples for me to make sure how to use the ontology. 7
2 I think that I would like to use this ontology frequently. 5
3 It is clear to me how to use this ontology. 4

There are several issues that can be explored in future works. The first work is to have more case studies using the statements in Table 3 and Table 4. In this study we only carried out case studies of the ontology usability scale in the GCIS and the DCO projects. Although the feedback is positive, we want to hear more feedback and suggestions on the statements themselves, including both their forms and the orders, as well the topics covered. The second potential work is relevant to the ontology types. As Table 7 shows, among the ontologies that the survey participants had worked with, there are both upper ontologies (e.g. Dublin Core, PROV-O) and domain ontologies (e.g. SWEET, GCIS). The former are applicable across a range of domain and the latter are only used for a specific application or domain. Therefore, we may organize corresponding statements for the usability evaluation of those two types of ontologies, and we can take further surveys to see if there are any differences between the community’s concerns on the two ontology types. In the current survey we did not ask the participants to specify their roles and experiences in the ontology work. The third potential work is that, if we are going to have new surveys, we can ask people about their roles (e.g. ontology developer, database curator, application developer, etc.) and their experience with ontology use (e.g. number of years). Last, the fourth potential work is to enrich, update and reorganize the statement pool from the point of view on expressivity, implementability and maintainability of ontologies. This framework is slightly different from the semiotic framework on syntax, semantics and pragmatics and covers new aspects of ontology usability.

## 6 Conclusions

In this paper, we followed the approach presented by Brooke (1996) to create a usability scale for ontologies. We considered candidate statements for this Likert scale from the syntactical, semantical and pragmatical aspects and conducted an online poll among the Semantic Web community to decide which are the most important statements. The usability scale was then used to evaluate domain ontologies under revision for the sake of comparing with their updated versions in the future. The goal of our work is to create a robust ontology usability scale. The evaluation of the scale itself, however, requires it being used and even adapted extensively across different domains. In this sense, this work is quite preliminary and without much data to validate its effectiveness, but we expect much more usage data from ontology users since the proposed usability scale is easy to score and is applicable to any domain ontologies.

## Acknowledgements

The research presented in the paper is partially supported by the Alfred P. Sloan Foundation through the Deep Carbon Observatory and a new faculty start-up grant from the University of Idaho.

## Competing Interests

The authors have no competing interests to declare.

## References

1. Bikakis, N, Tsinaraki, C, Gioldasis, N, Stavrakantonakis, I and Christodoulakis, S. 2013. The XML and semantic web worlds: technologies, interoperability and integration: a survey of the state of the art. In: Semantic Hyper/Multimedia Adaptation. Springer, pp. 319–360. DOI: https://doi.org/10.1007/978-3-642-28977-4_12

2. Brank, J, Grobelnik, M and Mladenic, D. 2005. A survey of ontology evaluation techniques.

3. Brickley, D and Guha, R. 2014. RDF Schema 1.1. W3C Recommendation. Available at: https://www.w3.org/TR/rdf-schema/ (Accessed on April 17, 2018).

4. Brooke, J. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189: 4–7.

5. Burton-Jones, A, Storey, V, Sugumaran, V and Ahluwalia, P. 2005. A semiotic metrics suite for assessing the quality of ontologies. Data & Knowledge Engineering, 55(1): 84–102. DOI: https://doi.org/10.1016/j.datak.2004.11.010

6. Casellas, N. 2009. Ontology Evaluation through Usability Measures. In: Meersman, R, Herrero, P and Dillon, T. (Eds.) On the Move to Meaningful Internet Systems: OTM 2009 Workshops, LNCS, Vol. 5872. Springer, Berlin, pp. 594–603. DOI: https://doi.org/10.1007/978-3-642-05290-3_73

7. Fox, M, Barbuceanu, M and Gruninger, M. 1995. An organisation ontology for enterprise modelling: preliminary concepts for linking structure and behaviour. In: Enabling Technologies: Proceedings of the Fourth Workshop on Infrastructure for Collaborative Enterprises. IEEE, pp. 71–81. DOI: https://doi.org/10.1109/ENABL.1995.484550

8. Fox, P and Lynnes, C. 2015. Additional items for ontology evaluation. Available at: http://tw.rpi.edu/web/project/SeSF/workinggroups/OntologyEvaluation (Accessed on Sept. 9, 2015).

9. Gangemi, A, Catenacci, C, Ciaramita, M and Lehmann, J. 2006. Qood grid: A metaontology-based framework for ontology evaluation and selection. In: Proceedings of the EON2006 Workshop.

10. Gomez-Perez, A. 2001. Evaluation of ontologies. International Journal of intelligent systems 16(3): 391–409. DOI: https://doi.org/10.1002/1098-111X(200103)16:3<391::AID-INT1014>3.0.CO;2-2

11. Gruber, T. 1995. Toward principles for the design of ontologies used for knowledge sharing. International Journal of Human-Computer Studies 43(5–6): 907–928. DOI: https://doi.org/10.1006/ijhc.1995.1081

12. Gruninger, M and Fox, M. 1995. Methodology for the design and evaluation of ontologies. Workshop on Basic Ontological Issues in Knowledge Sharing, IJCAI-95, Montreal, Canada. 10pp.

13. Hartmann, J, Spyns, P, Giboin, A, Maynard, D, Cuel, R, Suarez-Figueroa, M and Sure, Y. 2005. D1. 2.3 methods for ontology evaluation. EU-IST Network of Excellence (NoE) IST-2004-507482 KWEB Deliverable D, 1.

14. ISO. 1998. ISO 9241-11 Ergonomic requirements for office work with visual display terminals (VDT)s – Part 11 Guidance on usability. International Organization for Standardization.

15. Jokela, T, Iivari, N, Matero, J and Karukka, M. 2003. The standard of user-centered design and the standard definition of usability. In: CLIHC ’03 Proceedings of the Latin American conference on Human-computer interaction, pp. 53–60. DOI: https://doi.org/10.1145/944519.944525

16. Lozano-Tello, A and Gomez-Perez, A. 2004. Ontometric: A method to choose the appropriate ontology. Journal of Database Management 2(15): 1–18. DOI: https://doi.org/10.4018/jdm.2004040101

17. Lozano-Tello, A, Gomez-Perez, A and Sosa, E. 2003. Selection of ontologies for the semantic web. In: Web Engineering. Springer, pp. 413–416. DOI: https://doi.org/10.1007/3-540-45068-8_77

18. Ma, X and Fox, P. 2013. Recent progress on geologic time ontologies and considerations for future works. Earth Science Informatics 6(1): 31–46. DOI: https://doi.org/10.1007/s12145-013-0110-x

19. McGuinness, D and Van Harmelen, F. 2004. Owl web ontology language overview. W3C recommendation Available at: https://www.w3.org/TR/owl-features/ (Accessed on April 17, 2018).

20. Miller, E. 1998. An introduction to the resource description framework. Bulletin of the American Society for Information Science and Technology 25(1): 15–19. DOI: https://doi.org/10.1002/bult.105

21. Morris, C. 1938. Foundations of the Theory of Signs. University of Chicago Press. Chicago, IL.

22. Pratchett, T. 1990. Moving Pictures. Gollancz, London.