ViC has designed, developed and tested a selection of tools (Figure 1 above) to investigate values in software production. All tools are designed for both physical (e.g. face to face) and digital use, an overview of the tools designed and used so far can be found in this short paper (presented at ICSE-NIER2019).
This page focuses on the Values Q-sort (V-QS), which supports the systematic measurement and elicitation of human values in software development. For an overview on how the V-QS has been used in practice, check our ESEM2018 paper “Measuring Values in SE” .
What is the Q-sort?
The Q-Sort is an established mixed method that involves asking participants to sort a series of statements onto a grid according to their level of agreement with, or preference for, each statement. The results of multiple Q-sorts can be statistically analysed and different “types” of subjective orientations extracted (see Fig 2. below).
In addition, the Q-sort is accompanied by a semi-structured interview, meaning that the method also yields qualitative data. This provides added depth compared with, for example, survey-based research. Similarly to a survey, the Q-sort asks participants for their responses to a series of statements. However, it differs from a survey as responses are not free-form but have to be considered in relation to each other, making the participants to choose between statements.
Why Use the Q-sort?
Q methodology allows for the systematic study of subjectivity and provides structure to subjective opinions, and it offers an appropriate approach to systematically studying something like values. One of the other main strengths of the Q-sort is that it is a mixed method, combining a qualitative interview for each Q-sort and quantitative analysis of multiple Q-sorts. It also produces statistically valid results with smaller samples than traditional surveys and is hence regarded as “a robust tool for investigating human subjectivity” .
ViC Q-Sort
One of the challenges of using the Q-sort for this project was the nature of the statements. It was important that the statements were relevant to people working within the software industry. We wanted the statements to relate to Schwartz’s values model, as this is one of the most well-known and widely-applied contributions to the study of human values. Whilst Schwartz’s model has been critiqued (i.e. claims of cross-culture universality), the model is seen to provide particular insight into the mental representations of values held by people (Maio 2010).
ViC Q-Sort design
We used the most recent revised model of the Schwartz’s value schema (Schwartz 2012) and drew on Webler et al.’s advice (Webler 2009) that card statements should draw on the language and ideas of the group that is to be studied. Accordingly, the statements in the most recent version of the ACM Code of Ethics (Feb 2018) were dual-coded according to the Schwartz value types. These codes were then compared; considerable similarity was found, and areas of discord were discussed, leading to an agreed-upon coding. The most appropriate Code of Ethics statement for each value type was then turned into a Q-sort statement. Not all value types were represented in the Code of Ethics and in such cases a statement was written.
Additional Material
- Watts, S. and Stenner, P. 2012. Doing Q Methodological Research: Theory, Method and Interpretation, SAGE
- Introduction to Q-methodology – by William Stephenson, the British physicist and psychologist who developed the Q methodology in 1930’s. Note: “This article is chapter 1 from an unpublished book manuscript originally entitled Operant Subjectivity: Q-Methodology, Quantum Theory, and Newton’s Fifth Rule (ca. 1984). The article is published with the permission of the literary executor of the William Stephenson estate”
ViC tools Guiding principles
As we aim for a more scientific understanding of values (e.g. at a system, abstract and instantiation level), we need tools fit for the task; we argue that such tools should have the following characteristics:
- Applicability – can be used in a different contexts and domains and capture comparable data.
- Data Diversity – supports the capture and analysis of different kinds of data, i.e. numbers & narratives.
- Availability – ready to use with minimal training and resources.
- Explainability – the underpinning scientific framework and resulting design process can be clearly described, justified, and documented.