3.1 RI definition as a ‘wicked problem’

Rittel and Webber (1973) define wicked problems as complex design problems “for which no single computational formulation of the problem is sufficient, for which different stakeholders do not even agree on what the problem really is, and for which there are no right or wrong answers, only answers that are better or worse from different points of view” (Introne et al. 2013: 45). Wicked problems are to be distinguished from ‘tame’ problems. ‘Tame’ problems have a well-defined statement. It is clear what they are and what they are caused by. They therefore require a systematic methodology typical of engineering or scientific inquiry, because they belong to a class of problems that can be solved in a similar way each time. Furthermore, they have a definite stopping point and problem solving has no bearing on future options (Jentoft and Chuenpagdee 2009, Rittel and Webber 1973). In contrast to these problems, Rittel and Webber (Rittel and Webber 1973) identify characteristics of a ‘wicked problem’, which may be divided into Robert’s (Roberts 2000) two problem dimensions:

The problem’s definition: (1) There is no unique formulation of a wicked problem; (2) Wicked problems do not have a stopping rule; (3) Every wicked problem is unique; (4) Every wicked problem can be regarded as a symptom of another problem; (5) The choice of explanation determines the nature of the problem’s resolution; (6) The multiple stakeholders cause social complexity.

The problem’s solution: (7) Solutions to wicked problems are not true-or-false, but good-or-bad; (8) There is no test of a solution to a wicked problem; (9) Since it is impossible to learn by trial-and-error, every solution attempt has irreversible consequences; (10) Wicked problems do not have an enumerable set of potential solutions, nor is there a well-described set of permissible operations; (11) The planner has no right to be wrong.

Preceding qualitative research supports the interpretation of defining RI as a ‘wicked problem’(Riechert et al. 2016). As argumentation modelling has been specifically designed to address wicked problems, we introduce it as a way to document, model and manage RI definition.

3.2 Modelling argumentations in design processes

In the literature on Computer-Supported Argumentation Visualisation (CSAV), a number of modelling approaches have been developed. In their extensive review, Scheuer et al. (2010) discussed 50 different past and present argumentation systems and 13 empirical studies analysing the effects of their usage. Out of them, the research stream following Rittel’s IBIS directly addresses socially complex or wicked problems, which are IBIS, DRL and QOC. In the following we will briefly discuss their differences and the focus of their application.

The Issue-Based Information Systems (IBIS) notation was developed by Kunz and Rittel (Kunz and Rittel 1970) to support complex design processes with multiple stakeholders with conflicting aims. In IBIS, issues, positions and arguments are set in relation to one another (see Section 3.3 for details). Maps in IBIS notation do not evaluate the strength or weakness of arguments. Direct connections between arguments are explicitly disallowed (Conklin 2005). Therefore, the arguments have to be examined on their own. This is beneficial for depersonalising the discussion (Shum, Selvin, Sierhuis, Conklin, Haley, et al. 2006) but has shown itself to be an issue in contexts like decision argumentation modelling. The Decision Representation Language (DRL) (Lee 1989) addressed this need for documenting decisions in IBIS, and added Goal and Procedure node types and contained a more explicit grammar for relation types. According to Lee, making goals explicit by modular representation enables different viewpoints. Furthermore, arguments are not provided in the form of nodes but as a set of claims.

Design Space Analysis uses a semiformal notation called QOC (Questions, Options, and Criteria) and was developed mainly by MacLean and McKerlie (MacLean et al. 1991, 1993, MacLean and McKerlie 1995, McKerlie and MacLean 1994) to represent Design Rationale. Similarly to IBIS, QOC focuses on basic concepts like Questions and Options. In contrast to IBIS, criteria are used to evaluate and choose an option. Furthermore, DSA emphasises retrospectively rationalising the DR to clarify the dimensions defining the space, as opposed to recording the design process for a single design (Shum and Hammond 1994). Shum stated that both IBIS and QOC evolved into the current Compendium approach and tool (Shum, Selvin, Sierhuis, Conklin, Haley, et al. 2006: 2).

The research stream around IBIS addresses complex design processes with multiple stakeholders. Using DRL instead of IBIS would additionally allow for argument weighting. However, the depersonalisation effect is important for our use case. As such we will focus on IBIS and its current representation in Compendium in the following.

3.3 Issue-Based Information Systems (IBIS), Graphical IBIS (gIBIS) and Compendium

IBIS aims at guiding the identification, structuring, and settling of issues raised by problem-solving groups, and provides information pertinent to the discourse. Although Kunz and Rittel stated that IBIS has the elements topics, issues, questions of fact, positions, arguments, and model problems (Kunz and Rittel 1970), only issues, positions, and arguments were used in later tools (Conklin and Begeman 1988, Shum 2003).

In IBIS, each Issue can have many Positions. A Position is a statement or assertion which responds to the Issue. Often Positions will be mutually exclusive of each other, but the method does not require this. Each of the Issue’s Positions, in turn, may have one or more Arguments which either support that Position or object to it. Thus each separate Issue is the root of a (possibly empty) tree, with the children of the Issue being Positions and the children of the Positions being Arguments (Conklin and Begeman 1988). The relationships between the Issues are shown in Figure 1.

Later tools like “Graphical IBIS” (gIBIS) (Conklin and Begeman 1988) implemented the proposed elements of IBIS and provided a graphical interface for modelling the argumentation process. The tool is “designed to support the collaborative construction of these networks by any number of cooperating team members spread across a local area network” (Conklin and Begeman 1988). The aim of gIBIS was to address the interface problems inherent in capturing large amounts of informal design information and in providing effective methods for indexing and retrieval within that information (Conklin and Begeman 1988). Table 1 lists the properties of IBIS elements and their relationships as stated by Kunz and Rittel (1970) and Conklin and Begeman (1988).

The gIBIS java software was licensed by the Open University’s Knowledge Media Institute to further develop and release the software application and code. Called Compendium (Selvin et al. 2001), this tool was developed to provide an open environment for IBIS modelling, which can be integrated into design workflows. In 2006, Compendium was used on over 100 projects. We refer to the overview by Shum et. al. for empirical evidence of the approach’s learnability and effectiveness (Shum, Selvin, Sierhuis, Conklin, Haley, et al. 2006). Figure 2 shows the basic IBIS node types as used in Compendium (Shum, Selvin, Sierhuis, Conklin, Rowley, et al. 2006). Compendium provides the concept mapping functionality of gIBIS (implemented in the tool Questmap) but adds a more intuitive user interface and tagging functionality, a sub-map node type, an open architecture as well as API access (Shum, Selvin, Sierhuis, Conklin, Haley, et al. 2006).

The main visual representation in Compendium is a hierarchy of maps. Starting from the base map, it is possible to model multiple facets or hierarchy levels of the present model content. The exporting function to web maps allows the generation of interactive web maps of the Compendium maps. Additionally, the maps can be exported as XML, JPEG, or in a summarised outline form.

In our use case of documenting RI definitions (see Section 4) we modelled over 300 definitions with over 600 arguments. Consequently, the resulting network diagrams became very difficult to navigate and understand. We therefore discuss a simplified adaption of IBIS in Sections 5 and 6.

Functionality

Added value (R1): The structures IBIS provides form a valuable basis for modelling and visualising the argumentation of definitions. For modelling definitions, the general construct issue is specialised into three types of sub-issue. By using formalised sub-questions, information representation efficiency can be improved, because the resulting network structure follows semantic rules which allow for leaving out elements (as described in Section 5). For this definition case, the sub-questions for attributes and categories of differentiation are addressed:

• Possible characteristics of a defined element can be made explicit. For example, the definition for the element “gender” only has three characteristics: “female”, “male” or “n/a”.
• The second question we constantly face concerns the differentiation of the definition elements: For example, when defining “staff”, differentiation among the categories “full-time equivalents” and “head count” can be defined. They again could be differentiated among the categories nationality, gender, funding, qualification, etc.

Interoperability (R2): The formal language needs to be compatible with the IBIS notation to allow for existing argumentation to be reused. This requires a transformation mechanism to convert IBIS to RIDAL and back.

Usability

The present IBIS notation seems to be a valuable basis for modelling definition argumentation. Additionally, there is excellent tool support with Compendium. However, the resulting maps become very complex when defining research information with their attributes and differentiations. For higher usability, we propose semantic rules to reduce the complexity of argumentation maps in IBIS notation. The agreement on a shared static semantic allows for a comparably simple formal language which forms the basis of application in multiple use cases and tools. A simplification is needed in two regards:

Objective usability (R3): At present there is no format in which argumentation maps can be exchanged without the need for a considerable amount of additional information. While Compendium allows for XML and Web Map export, the resulting files contain a lot of additional information. A formal language should reduce this complexity in terms of the space used and depth of the data structure tree required to describe the same contents.

Subjective usability (R4): Based on experience within the project and discussions with the research information standardisation community, the requirement for better data accessibility became evident. The present visual representation in Compendium is easy to use and intuitive for small maps. In the definition context of our project, with about 80 hours of discussion time the maps became too complex to be printed or even viewed on a screen, even when using sub-maps. We assert that when it comes to representing large-scale discussions, IBIS maps are not accessible enough for external documentation. The current export functionalities (XML, Web Maps) provide direct access to the underlying structure, but are too complex for printing or providing insight. A formal language that incorporates a shared static semantic has to be more easily understood by experts.

Table 2 provides an overview of the requirements addressed.

5.1 Case “Specification Project of the German Research Core Dataset”

The use case we employ is the standardisation project “Research Core Dataset” (RCD) for the German science system. The project was initiated in 2013 by the German Council of Science and Humanities to develop a shared set of definitions for research information (e.g. staff, publications, funding, patents) for the German science system. More than 48 different stakeholders are involved in the project. They include representatives of universities, non-university research institutions, ministries, research information system vendors and scientific societies. Furthermore, four pilot universities are integrated in the project. The definition process is organised in four project groups, each of which has eight experts. Each of the project groups holds up to six meetings, with 1–2 days of discussion time per meeting (with eight hours of discussion time per day). The project group “definitions and data formats” defines research information for all six research information areas. The discussions are structured and moderated by using CSAV on a central screen. After an initial discussion phase, a feedback phase is conducted. All pilot universities, the non-university research institutions and software vendors are asked for feedback concerning the applicability of the definitions. After the feedback round, another discussion phase is conducted to integrate the various external feedback into the definition specification.

This case allows us to analyse the value of argumentation modelling for reaching a definition consensus in a particularly complicated case (or ‘extreme’ case; see Seawright and Gerring 2008). The German case exemplifies extreme characteristics because the country’s federal regulation of higher education and research institutes entails particularly fragmented and diverse reporting processes and requirements (Biesenbender and Hornbostel 2016), resulting in particularly complex stakeholder positions. Testing the language in this complex case will result in higher applicability in less complex cases.

5.2 Modelling the definition of research information in IBIS

The IBIS notation as proposed by Kunz and Rittel had no explicit formal model (Kunz and Rittel 1970), but was a paper-based approach. The implicit notation was later implemented in gIBIS (Conklin and Begeman 1988). Conklin and Begeman provided the first semi-formal depiction of the elements and their relationships (see Figure 1 in chapter 3.2). The most recent successor is Compendium.

Section 3.3 introduced the modelling of IBIS in Compendium. All IBIS node types have a representation in Compendium. Issues are modelled as questions (and displayed as a node with a question mark). Positions are modelled as options (and displayed with an exclamation mark). Arguments are displayed as nodes with a plus and minus sign. Compendium additionally allows for sub-maps, encapsulating more detailed discussion content. In the following we will stick to IBIS terminology (issues and positions). Note that the Compendium terms (questions and options) could be used correspondingly.

When modelling definitions of research information in Compendium, all elements can be placed freely on a canvas. Furthermore, there are no restrictions about which kind of issues can be modelled. This offers a high degree of flexibility for a wide range of IBIS applications, but results in very complex and confusing diagrams. The possibility of using sub-maps allows for encapsulation, but is not very user-friendly if the number of sub-levels rises above three. In our case, we had up to 9 sub-levels, which is very challenging to keep track of for all persons trying to read the argumentation model of the RI definitions.

In order to be able to simplify the definition model without losing any information and to meet the usability requirements R3 and R4, in the next section we discuss the introduction of semantic modelling rules.

5.3 Introducing rules for definition modelling of research information in IBIS

Using the IBIS notation in Compendium, a sample definition for “full-time equivalent” could be modelled as shown in Figure 3. The root question node (=issue node in IBIS) asks for a suitable definition for the element “full-time equivalent” in the research information context. The definition itself is documented in the node. Its definition can therefore be shown by clicking on the node in Compendium. Two possible alternative options (equivalent to position nodes in IBIS) have been discussed in the definition process by the experts. Both alternative options have arguments supporting or challenging the option.

On the next level, each of the alternative options can have the sub-question “How can … be differentiated?”. Alternative options can be appended to this question on the next level. If the differentiations of a definition are an exhaustive list, attributes should be used instead of differentiations. This is represented by the question “What are possible attributes for…?”. Using this approach allows for freely scalable maps to model as detailed differentiations as necessary, with the option of providing and documenting arguments on each level of the definitions. It has to be noted that IBIS allows for any type of question to be modelled. By restricting the usage of IBIS questions (or issues) to the above types of question, a higher degree of formalisation can be achieved. This forms the basis of the simplification described in Section 5.4.

We formalise the semantic rules described for modelling definitions in IBIS:

• Node rule 1: ID usage in the definition label: In order to allow for fast searching and identification, a serial ID is added to the position nodes.
  In the example: “St” shows that the definition deals with staff. The number is a serial number for all elements defined. Alternative positions (options in Compendium) like St1 or St1a are appended with a,b,c,d, … for each alternative position.
• Node rule 2: Issue types: Issue nodes can be one of three types:
  
  • Type A: Definition Issue: “What is a suitable definition for …?”
  • Type B: Differentiation Issue: “How can … be differentiated?”
  • Type C: Attribute Issue: “What are possible attributes for …?”
  Note: In the following, positions addressing those issues are called type a (for addressing definition issues type A), type b (for addressing differentiation issues type B) and type c (for addressing attribute issues type C).
• Node rule 3: Issue type A (Definition) as tree root: Each definition tree starts with the issue (or question in Compendium) “What is a suitable definition for …?”.
  In the example: “What is a suitable definition for full-time equivalent” is the root node.
• Link rule 1: Position answers issue: Positions (or options in Compendium) addressing that issue are connected to that issue. The definition text of the element is provided in that position node.
  In the example: Two alternative options are provided as positions (St1 and St1a).
• Link rule 2: Differentiation and attributes as sub-issues of positions: Each position can be further defined by a sub-issue asking for differentiation (Node rule 3) or for attributes (Node rule 4). If no more detailed definition is required, stop here.
  In the example: “St1 full-time equivalent (calendar year)” is connected to a differentiation issue, and “St7 Gender” is connected to an attribute issue.
• Node rule 4: Sub-issue type B (Differentiation): “How can … be differentiated?”. Positions for that issue are added based on Link rule 1.
• Node rule 5: Sub-issue type C (Attributes): “What are possible attributes for …?” Positions for that issue are added based on Link rule 1.
• Node rule 6: Attributes as different node type: Positions for attributes receive a different node type. This can be achieved by tagging and using a different visual representation.
  In the example: “St48 Male” has a different node icon than “St10 Gender”, because it is an attribute and not a differentiation.
• Link rule 3: Restart from Node rule 2 for differentiation or attribute definition: Each differentiation or attribute position is defined by adding another root issue tree (i.e. by restarting the rules from Node rule 2 onwards).
  In the example: “St5 Nationality” is followed by another Root issue “What is a suitable definition for Nationality”. As the appended position (including the definition text) is detailed enough, no further differentiation or attributes are defined below.
• Link rule 4: Arguments support and challenge positions: Pro- and con- arguments can be added to positions (not to questions!) on every level.
• Node rule 7: Set usage status as tag: For each position (or option) it is decided whether the definition is to be used or not used (for example for alternative positions not being used). The relevant information can be tagged as “used”, “not used”, and “more definition work required” in Compendium. Additionally, this information can be visually represented by using text colours green, red and orange.
  In the example: “St1” is tagged as “used”, while its alternative position “St1a” is tagged as “not used”. This is visually represented by green and red text colour.

Additionally, we use placement rules for higher readability. These are only required if the diagrams are to be read without further transformation.

• Placement rule 1: Nodes on the same depth level at same x position: To improve readability of the diagram, all nodes on the same depth level are placed at the same x position.
  In the example: Both alternative positions of full-time equivalent St1 and St1a are on the same depth level and therefore placed at the same x position.
• Placement rule 2: Sub-issues and positions from left to right: To improve readability of the diagram, all sub-issues and positions are placed from left to right.
  In the example: All positions and sub-issues are placed to the right of the issue.
• Placement rule 3: Arguments on the right border of the diagram: To improve readability, all arguments are placed on the right side of the diagram.

Fehler! Verweisquelle konnte nicht gefunden werden. shows an overview of how to model in Compendium using the above rules.

5.4 Model transformation to RIDAL

Based on these rules, it is possible to relate the different positions (or options), differentiations and attributes without having to specify the underlying issues (or questions) as shown in Figure 4. By using the conversion rules stated below, an IBIS diagram modelled using the rules in Section 5.3 can be converted to RIDAL without any loss of information.

For each position type b (i.e. addressing a differentiation issue) and type c (addressing a attribute issue):

• Connect all linked argument nodes to the next position type a (addressing definition issue) two levels lower.
• Remove connection to the position type b.

For each position type a (i.e. addressing a definition issue):

• Connect all positions on the same level with an arrowless link with the label “Alternative” (see step 1 in Figure 5).
• Get all child positions type A (definition) four levels lower, remove all connections it has to its direct parent elements (issue type A) and connect it to the parent position type A (four levels higher). A special rule applies for reconnection: Only reconnect the positions if their status is not “not used”. If all of the alternative positions have the “not used status”, connect the position with the smallest y position anyway (so there is always at least one connected element) (see steps 2 and 3 in Figure 5).
• Remove all child issues type B (differentiation) and C (attributes) as well as their related links (see step 4 in Figure 5).

Finally:

• Remove the root issue type A (definition)

The resulting diagram is shown in Section 5.5 in Figure 6. Note that the transformation can be reversed by reversing the order of the steps and inverting them (i.e. “create” instead of “remove”, “remove” instead of “connect”).

5.5 RIDAL

By allowing only three question types and following the rules in Section 5.3, a much simpler representation of the content is possible without any loss of information by using the transformation specified in Section 5.4. The resulting model is shown in Figure 6.

In order to model directly in RIDAL, the rules from our IBIS modelling in Section 5.3 can be adapted:

• Node rule 1: ID usage in the definition label: All positions (or options in Compendium) are connected to that issue. The definition text of the element is provided in that position node.
  In the example: Two alternative options are provided as positions (St1 and St1a).
• Node rule 2: Position as tree root: Each definition tree starts with the position (or answer in Compendium) for the implicit question “What is a suitable definition for …?”. If there is more than one possible definition, each gets its own position node and they are connected with an arrowless link with the label “Alternative”.
  In the example: Position “St1 full-time equivalent (calendar year)” is the root node. Its alternative St1a is connected with an “Alternative” link.
• Link rule 1: Differentiation and attributes as sub-positions: Each position can be further defined by a sub-position asking for differentiation (Node rule 3) or for attributes (Node rule 4). If no more detailed definition is required, stop here.
  In the example: “St1 full-time equivalent (calendar year)” is connected to the differentiation positions St5, St5 and St7. “St7 Gender” is connected to the attribute positions St48 and St49.
• Node rule 3: Sub-position addressing differentiation: Definition text answering the implicit question “How can … be differentiated?”
• Node rule 4: Sub-issue addressing attributes: Definition text answering the implicit question “What are possible attributes for …?”
• Link rule 2: Restart from Link rule 1 for differentiation or attribute definition: Each differentiation or attribute is defined by adding links for differentiations or attributes below (i.e. by restarting the rules from Link rule 2 on).
  In the example: “St7 gender” is further defined by attribute positions below.
• Link rule 4: Arguments support and challenge positions: Pro- and con- arguments can be added to positions (not to questions!) on every level.
• Node rule 5: Set usage status as tag: For each position (or option) it is decided whether the definition is to be used or not used (for example for alternative positions not being used). The relevant information can be tagged as “used”, “not used”, and “more definition work required” in Compendium. Additionally this information can be visually represented by using text colours green, red and orange.
  In the example: “St1” is tagged as “used”, while its alternative position “St1a” is tagged as “not used”. This is visually represented by green and red text colour.

Figure 7 shows an overview of how to model research information definitions in RIDAL using the above rules. Note that in contrast to the IBIS, position nodes type a (alternatives) have to be linked to each other with the label “Alternative” to be able to distinguish between the position types.

6.1 Requirement 1: Added value

The central idea of language simplification is to use only three types of question: “What is a suitable definition for …”, “How can … be differentiated?”, and “What are possible attributes for …?”. By making them explicit, experts defining research information have a structure to guide them through the discussion. This structure also helps to ensure that each definition is checked if further differentiation is required and if alternative definitions might be better suited. This increased formalisation comes at the cost of less flexibility when other question aspects might be of interest. We consider this requirement to be met.

6.2 Requirement 2: Interoperability

The semantic rules for modelling definitions in IBIS and RIDAL described in Section 5 allow for conversion of IBIS definitions to RIDAL definitions and vice versa without any loss of information (see Section 5.4). By using the steps as a conversion algorithm, definition models in IBIS can be converted automatically to RIDAL and back. Therefore it is possible to use both representation forms depending on usage context. For example, it might be of benefit to use the shorter RIDAL in situations where modelling speed is of importance, for instance when modelling is performed on-screen while experts discuss the definitions in order to increase model completeness and discuss group agreement (as found in Selvin et al. 2001). By leaving out redundant “What is a suitable definition for …” questions but still being able to provide the full definition context based on the semantics, improves the experience for experts in the discussion group and enhances modelling speed. In other cases, having the redundant questions inside the model might be of benefit. For example, when using the diagrams to document what has been discussed, seeing questions might help external experts without knowledge of the semantic rules. Since conversion from and to RIDAL has been shown to be possible, we consider this requirement to be met.

6.3 Requirement 3: Objective usability

In our use case of defining a research information standard for the German science system, experts agreed on 305 definitions (including differentiations and attributes), and discussed 172 alternative definitions. Additionally, more than 600 arguments were documented. By using RIDAL, which is less complex, the resulting text-only XML files have a size of 4 megabytes. When using the Compendium Web Map export (with a visual interface), the total file size amounts to 26 megabytes.

When using the structured IBIS, the text-only XML export has a file size of 8.2 megabytes and the Web Map export has a file size of 50.8 megabytes. Therefore, documenting the same definition content in RIDAL requires 49.4 % of rule-based IBIS for text-only export, and 52.2 % for Web Map export with a visual interface. One might argue that storing files of such sizes is no longer a problem. But when it comes to presenting Web Maps online, this reduction of about 50 % does play a role in mobile applications or if internet access is limited.

As a second dimension of complexity we counted the maximum depth of argumentation tree (i.e. the number of definition sublevels). While the RIDAL has a maximum depth of 14 levels, the corresponding structured IBIS requires 28 levels. RIDAL therefore reduces the tree depth by 50%.

6.4 Requirement 4: Subjective usability

Much more important than the necessary space is the reduction in visual complexity. By using RIDAL instead of a rule-based IBIS notation, in our case the number of definition-related nodes is reduced by 38.4 % from 775 to 447. As the number of arguments stays the same, the total number of nodes required is reduced by 26.3 % from 1405 to 1107.

The second component of visual complexity in network diagrams is the number of links required by the visual representation. Using RIDAL instead of a rule-based IBIS notation reduced the number of links in our project by 55.2 %, from 3085 to 1383.

6.5 Evaluation summary

The evaluation shows that using RIDAL reduces the number of required nodes by 26.3 %, the number of required links by 55.2 %, the text-only space by 49.4 %, the necessary Web Map space by 52.2 %, and the tree depth by 50% without any loss of information. One limitation of using RIDAL is that the readers of the diagrams have to be familiar with the rules. On the other hand, having said rules explicitly available helps when modelling RI definitions, because the process of definition is more structured, thus resulting in higher data consistency compared to modelling completely freely in IBIS. By providing a transformation algorithm, it is possible to convert RIDAL back to rule-based IBIS when required. Finally, the introduction of rules allows for the automatic reuse of the documented information for transforming definitions into a data model or providing additional visual representations to address weaknesses of Compendium Web Maps.