Back

Evidence summary: Measuring wellbeing with educational technology. How can research on wellbeing inform better educational technology?

Santiago De Ossorno Garcia, Research Mentor, UCL EDUCATE
  • articles

The purpose of this paper is to help edtech designers understand wellbeing and how to use different measurement strategies in an educational/digital technology context.

Introduction

Measuring wellbeing is a long-standing challenge for science. When developing health-related educational technology (edtech) or interventions to improve wellbeing, designers and educational technologists should consider multiple complexities, including the challenges related to measuring wellbeing while conducting research.

Understanding the measurement techniques or tools available, and the context-specific needs of the measurement of wellbeing, will allow educational technologists to understand how to improve design and implementation. This will enable the development of more effective, research-informed edtech.

The purpose of this paper is to help edtech designers understand wellbeing and how to use different measurement strategies in an educational/digital technology context.

The summary of evidence available should help guide the design of research and technology. It will also encourage consideration of recommendations and critical factors that may affect measurements of wellbeing that are appropriate to the context.

Defining wellbeing

Wellbeing is a multi-faceted concept that transcends scientific disciplines (health, education, economics, psychology, social sciences, etc.) and it is an all-encompassing term in society. This complexity explains why it is so hard to measure. There is no agreed definition of wellbeing, nor even an agreement on its spelling (Dodge et al., 2012). Terms such as happiness (eudaimonia), quality of life, or life satisfaction are often linked to this construct.

In psychological sciences and research, a construct refers to an explanatory variable that is not directly observable. Therefore, the edtech designer’s first challenge around the measurement of wellbeing is to find the appropriate definition of this construct relevant to their context and purpose.

The Universal Education Foundation (UEF) (Awartani et al., 2007) provides a holistic definition wellbeing as the realisation of one’s physical, emotional, social, mental and spiritual potential. However, a working definition may not always be useful. For instance, some research describes wellbeing as defined by contributors within the research group or context, with the people involved agreeing what a “good life” means for them (Ereaut and Whiting, 2008), rather than using an imposed definition.

A useful and widely-used definition to conceptualise adolescent wellbeing, used by Columbo (1986), is “a multidimensional construct incorporating mental/psychological, physical and social dimensions” (p.288). However, this may not always apply. For example, a study intending to conceptualise wellbeing among adolescents in the workplace may not use the same working definition of wellbeing as a study looking at the quality of peer interactions between adolescents in the classroom. Each study may use different set of variables, factors and scales.

A systematic review exploring the different definitions of child wellbeing across scientific literature is presented in Table 1. In order to operationalise the definition of wellbeing for any specific research project, the complexities of this concept and the specific population of interest should always be considered. There may not be a definition of wellbeing in this literature that perfectly fits every research context.

Table 1 Extracted from (Pollard and Lee, 2003): Child wellbeing a systematic review of the literature

It is important to consider children’s wellbeing distinctly from young people’s and adults’ (Ryff, 1989; Ryff and Keyes, 1995; Clarke et al., 2000) due to empirical evidence, ecological frameworks and theories of human development (Bronfenbrenner, 1979) supporting their differences. These state that parental wellbeing has reciprocal influence on children’s wellbeing, and that contextual factors have an interrelated effect on both children and parents. These effects are not always linked with outcomes in the same area of wellbeing (Ungar, 2013). For instance, being bullied during the later years of primary school is strongly associated with lower attainment in secondary school and it is the strongest predictor for wellbeing (Gutman and Feinstein, 2008). On the other hand, involving students in decision-making at school seems to have a significant effect on improving wellbeing in students (Jamal et al., 2013).

A positive ethos and a supportive school environment are fundamental factors that promote students’ wellbeing. Misinterpretation of the ideas of Jean-Jacques Rousseau may assume that enjoyable learning experiences will drive academic achievement, but this disregards Rousseau’s view on suffering as a pedagogical tool for effective learning. There is a fast growing body of evidence supporting the positive effects of interventions and strategies in an educational context that raise achievement and also raise pupil happiness and a joy for learning through the construct and interventions focused on social-emotional learning and emotionality (Valiente, Swanson and Eisenberg, 2012), however the mechanisms that explain these benefits in order to improve academic achievement are still to be understood (Panayiotou, Humphrey and Wigelsworth, 2019). On the contrary, a think tank research report tends to support the idea of academic achievement and wellbeing as a trade-off relationship (Heller-Sahlgren, 2018). It is worth mentioning that 20 years of extensive research and evidence has shown that physical punishment increases the risk of broad and enduring negative developmental outcomes (Durran and Ensom, 2012). In addition, robust peer reviewed meta-analysis shows the beneficial effects of fostering social-emotional learning interventions within educational contexts (Durlak et al., 2011), especially in schools, showing improvements in students’ social skills, reduction of anti-social behaviour, better mental health outcomes, positive self-image, increased academic achievement and prosocial behaviour (Sklad et al., 2012).

As stated wellbeing and social-emotional links can contribute to positive students outcomes, but definitions and psychological and emotional constructs interplay with educational processes and phenomena. Developing a logic model and theory of change for edtech solutions can help to overcome the complexities of product evaluation focused on wellbeing. These considerations will help identify factors that may have a major influence on the effectiveness of the tool when looking at wellbeing as an outcome.

Linguistic studies examining the definition of wellbeing demonstrate that it is a dynamic concept that changes over time. Research on the word ‘wellbeing’ shows that this term is puzzling in comparison with other concepts. For instance, it is a word with no clear opposite, and it is not clear how it should be spelled (Ereaut and Whiting, 2008).

Wellbeing research identifies two clear perspectives on how this construct is used (Ryan, Deci, 2001; Ryff, 2013):

  • The hedonic approach: Focuses on wellbeing in terms of pleasure and pain-avoidance, mainly related to the concept of happiness.
  • The eudaimonic approach: Defines wellbeing in terms of purpose and self-realisation, taking the perspective of a person who is fully-functioning, as evaluated by subjective standards. It relates directly to humans flourishing.

Another distinction that can be useful to determine wellbeing in the context of edtech solutions is internal vs external (Alatartseva and Barysheva, 2015):

  • Internal (subjective) wellbeing: Also known as psychological wellbeing, associated with one’s personal characteristics and features. Dimensions that relate to this concept are personal growth, life purpose, autonomy, positive relationships, etc.
  • External (objective) wellbeing: Develops from an external perception and one’s evaluation of human society. External wellbeing is associated with material wealth or quality of life. It is strongly linked to and composed of sociological factors, such as the level and stability of income, conditions of residence, education access, natural and social environment, safety, civil rights, and needs.

Recently, Dodge and colleagues (2012) proposed a different approach, defining wellbeing as a balance between resources and challenges (See Figure 1).

Figure 1 Extracted from Dodge et al. 2012: The challenge of defining wellbeing

Dodge and colleagues argue that stable wellbeing is when people have enough psychological, social and physical resources to meet the psychological, social and physical challenges of life. This model conceives wellbeing as dynamic, and it is similar to models used to explain other phenomena – such as stress or coping mechanisms. This model, however, does not provide a clear definition of wellbeing, and provides a limited explanation of the fact that, even if ‘challenges’ are balanced by ‘resources’, this does not necessarily bring an increased sense of satisfaction within the individual.

The Organisation for Economic Co-operation and Development (OECD) has worked for a number of years to try to formalise the measurement of wellbeing. Specific advice is not yet available to researchers on how to fully operationalise this term. Despite this, a number of studies and resources have been produced since the OECD set the measurement of wellbeing as an international goal (OECD, 2007). In 2017 the OECD published a report, How’s Life?, which features a range of such studies and analyses of people’s wellbeing and how to measure it, including the interactive Better Life Index website that compares wellbeing across countries. This is based on 11 key topics that the OECD has identified as essential factors that contribute to wellbeing.

Assessments and measure of wellbeing – current evidence

Although there are complexities and sensitivities in defining wellbeing, there is a body of literature that can guide us in making informed decisions about how to improve the criteria, and a broad offering of tools to measure wellbeing. This section provides a summary of some of the systematic reviews of psychometrics tools and tests that attempt to measure subjective wellbeing and make recommendations on how to measure it.

As discussed previously, there are two broad divisions to the measurement of wellbeing: objective and subjective. Objective measures make assumptions about individuals’ needs in relation to their context. These assumptions lead to indicators that estimate the extent to which an individual’s needs are being met. They normally measure three main areas (Selwyn and Wood, 2015):

  • Economic: Most common measurements are Gross Domestic Product (GDP) or household income, or real household disposable income. Further indicators are shown regularly in the UK Office for National Statistics (ONS).
  • Quality of life: Often related to health, common indicators include life expectancy, educational attainment, impairments or functioning in daily life.
  • Environmental: Relevant to the environment of the individual, for example air pollution, transportation, water quality, etc.

Objective measurements are well documented in research that compares different nations’ profiles. Nonetheless, it is important to note that these measurements may not give accurate information about wellbeing without considering the subjective angle (Guillen-Royo and Velasco, 2005; Kahneman and Krueger, 2006). Subjective measurements allow people to assess their own wellbeing and how they feel (Hicks, 2011). These are not only subjective because of the self-report method, but because the perceptions of people are crucial to understanding their own conceptions of subjective wellbeing. For subjective wellbeing, there are three established approaches, led by the ONS (2010). These guidelines are:

  • Evaluative approach: Asks individuals to step back and reflect on their overall life satisfaction and make a cognitive assessment of how it is going overall on specific aspects such as health, job, school, etc. This is a very common approach, normally using Likert (rating) scales. You should have an explicit focus and timeframe in the questionnaire (e.g. your job in the last two weeks). Methodological research has shown that this improves the response rates and comprehension (Dolan et al., 2011).
     
  • Affective (hedonic) or experience approach: Requires focussing on the assessment of people’s positive and negative emotional experiences (e.g. happiness, sadness, anxiety, energy levels, etc) over a short timeframe (e.g. on a day-to-day basis or during the last week). The use of emotional diaries is common, but sometimes researchers fear that including negative feelings will lead to a reduced rate of response. This risks biasing such research (Hicks and Tinkler, 2011). Yet negative emotions are the most important to understand when designing for impact and system changes.
     
  • Eudemonic (psychological, functioning and flourishing) approach: This is an individual assessment of someone’s internal world, based on self-determination theory (Ryan and Deci, 2000). Even though there is some agreement on the core dimensions (self-efficacy, good relationships, purpose of life, personal growth, autonomy and environmental mastery – see Ryff, 2013), there is an ongoing debate in around 350 publications using scales on eudemonic wellbeing, including the dimensions and factors that influence it. Some factors – such as personality, age, family, work, clinical/other interventions, and health and biological research – influence how this measurement approach is used.

In addition to the OECD and ONS, scholars have conducted systematic reviews on wellbeing self-reported instruments for adults (See Table 2; Linton, Diepper and Medina-lara, 2016) and children (see Table 3; Pollard and Lee, 2002), as summarised below. The summary aims to provide an illustration of the diversity of instruments available in scientific literature.

Table 2 Extract table summary of Linton, Diepper and Medina-lara, 2016: Review of 99 self-report measures for assessing well-being in adults: exploring dimensions of well-being and developments over time
Table 3 Extract table summary of Pollard and Lee (2002): Child wellbeing: A systematic review of the literature

The challenge to measure wellbeing continues to puzzle researchers across scientific disciplines, despite efforts to reach a global consensus. Edtech entrepreneurs and designers should take this contemporary debate into consideration, being critical when selecting tools of measurement, or when working or designing tools around this multi-dimensional concept. Some recommendations and critical views for edtech are outlined in the next section.

Recommendations for edtech on measuring wellbeing

In recent years, an emerging field of research on ‘positive technology’ or ‘positive computing’ explores the use of technology for wellbeing and human potential (Sander, 2011; Botella et al., 2012; Riva et al., 2016b, 2017). It is a multidisciplinary approach that requires a combination of psychology, technology, design, computing and human-computer interactions (Lee et al., 2018).

This nascent field of research brings complexities and challenges for educational technologists, additional to the contextual factors in addressing wellbeing from an edtech perspective (Desmet and Hassenzahl, 2012; Desmet and Pohlmeyer, 2013; Pohlmeyer, 2012).

There is a lack of evidence for methodological considerations on the intersection between the digital environment and the physical environment of the user. This makes it difficult to evidence the role of technology in changes or effects that may occur in the physical environment of the users. It becomes even more sensitive when the intervention with technology aims for behavioural change in their users, as they require careful and ethical consideration of psychological factors (Hassenzahl and Laschke, 2014). Effectiveness of health technologies and interventions for human behavioural change must therefore be evaluated through robust randomised controlled trials conducted alongside mixed methods, which can give a better understanding of this new digital context of therapeutic application.

Therefore, wellbeing needs to be considered carefully and redefined to take account of the vast array of interrelated internal and external factors within the context of edtech. Research has shown that users are sensitive to the way edtech products “speak to them”, and the communication styles have emotional consequences that can jeopardise the intended positive change to the user (Niess and Diefenbach, 2016). Furthermore, scholars have identified a lack of psychological foundation in existing technological products designed for self-improvement (Conroy et al., 2014).

Behavioural markers and digital phenotypes include a set of observable characteristics in a user through tracking and monitoring technology. The information data could come from data processing the features of the technology itself (sensors, usage, tracking, etc.). To obtain evidence about this information, researchers need self-reported data from users, contextual information about the interaction with the tool, and contextual information about the digital phenotype data to find meaningful behavioural markers or indicators related to the wellbeing of the user.

Digital phenotypes are promising – although there is not a prescriptive or refined method for capturing and analysing various streams of digital health data (Jain et al., 2015), nor a way to test how reliable these markers are. A potential limitation of the success of these approaches are the professionals and systems that aim to integrate this data into their practice in a way that is ethical and upholds users’ privacy. Therefore, it is advisable to conduct pilots in naturalistic environments, and to speak with experts and professionals from the given context in order to co-design and test the assumptions of any logic model.

It is also necessary to understand the user-contextual model of technology and how it is used across population. In addition, an understanding of the user’s personal characteristics and demographics is required. You may need to use qualitative methods, surveys, interviews, and focus groups to understand your target population and to fully understand all uses, positive and negative, of your edtech (Sockolow et al., 2016).

The logic model and theory of change (Zhao, Yan, and Lei, 2008) may enable you to understand the social validity and fidelity of your system/tool/intervention, and also to re-design it in order to achieve the intended impact. This will allow you to be context-aware whilst working with the concept of wellbeing. Your target population will be crucial for the evaluation of your tool on wellbeing, as wellbeing is also culturally biased (WHO, 2015) and influenced by environment. Therefore, the instrument used to evaluate and measure wellbeing may require cross-cultural validation for your international set of users.

War (2012) developed a set of recommendations on how to think about measuring wellbeing. Below, we summarise the key concepts that are relevant to the domain of edtech:

  • Psychological, physiological or social emphasis: Defining the emphasis based on your intended impact may help to determine the main indicators and definitions that apply to the context of the edtech solution.
  • State or trait wellbeing: Whatever measurement is used, it is essential to review the target duration that applies to the question investigated. Wellbeing can be conceived as a stable feeling, but it may also a link to a temporary behaviour that depends on a situation or point in time. 
  • The scope of measurement: The broadest scope for research is context-free (such as global happiness, life satisfaction, etc.). These constructs are not context-dependent. Some studies are domain-specific – relating, for instance, to health, job, leisure, etc. The more granular measures are facet-specific, targeting a particular aspect of wellbeing, such as one’s satisfaction with their peers at the workplace. Your measurement tool will need to take this scope into consideration. 
  • Select the approach to measure wellbeing: You may need to decide if it is an evaluative, affective or eudemonic approach. This will be highly dependent on your research question and methodology for data collection.
  • Examine ambivalence: The similarities or differences between wellbeing elements may be studied across time or on a single occasion. In the first scenario, it is worth considering change and expected fluctuation levels from period to period. The second scenario should consider how someone can feel ‘good’ and ‘bad’ at the same point in time. This illustrates how ambivalent wellbeing can be, and why behavioural change interventions should consider the bittersweet experience of the user (Diefenbach, 2018). 
  • Content validation and related concepts: Develop a clear theory of the attribute to be measured and select tools based on this theory (Highhouse, 2009). Due to the complexity of wellbeing, a “discriminant validity” approach, adding similar measures, may be suitable in the context of edtech research.

More simple and pragmatic approaches to evaluate wellbeing can also be considered. For instance, Davies and colleagues (2017) proposed what they called “proportionate” methods to evaluate In Hand, a mental wellbeing smartphone app for adolescents. They used three different methods of data collection: (1) mobile analytical data, (2) a user survey adapted from a validated wellbeing measure, and (3) semi-structured interviews to a subset of the survey respondents. Despite this, there are several sampling limitations, as the survey respondents may not have been representative of the population (e.g. sample by convenience), interview rates were low, and the mobile analytics limited the statistical analyses for the study. It demonstrates a simple evaluation of wellbeing in an edtech context (see Figure 2). This specific research provided further understanding of how the app was used, providing insights about ways in which the tool can further support mental health wellbeing of adolescents and improve its effectiveness.

Figure 2 Diagram outlining number of user sessions and the flow of participants during the study. Extracted from Davies et al. (2017): Proportionate methods for evaluating a simple digital mental health tool

Conclusions

This summary of evidence gives us an understanding of the challenges and complexities of working with the concept of wellbeing. It requires a multidisciplinary approach, especially in the context of developing edtech. Wellbeing is a multi-dimensional construct that is dynamic in nature. Contemporary debate continues to seek consensus on definitions of wellbeing, and relevant institutions are addressing this at national and international levels. Resources are therefore available to support the measurement of wellbeing with edtech (OECD, 2017; ONS, 2010).

The definition of wellbeing is divisive and it is difficult to grasp the accurate meaning. When conducting research into wellbeing in edtech it is useful to develop a clear working definition. It is also a good starting point to decide how to go about measuring wellbeing in relation to the product and its intended impact. It is recommended to consider different measurement techniques and tools, such as standardised questionnaires, interviews and surveys, alongside robust strategies to validate and justify the use of the selected tool for product evaluation. It is also important to consider the context of the edtech, and acknowledge that the selected measurement technique or tool may require adaptation and iteration based on the evidence extracted from the context and targeted population. This may lead to a validation of such an instrument, scale or tool in research.

There are resources available to comprehend the measurement of wellbeing. Systematic reviews have been conducted in this area for adults and children. This allows edtech designers to understand the different domains of wellbeing and the diversity of its applications, depending on the research context of the edtech product.

Digital health tools and edtech research methodologies addressing wellbeing are nascent and in continuous development – but they should be based on robust evidence-based findings, informed protocols and frameworks from their design conception, including enlisting experts as co-designers. It is advisable to collect different sets of evidence with different methods of data collection. This will allow a broader understanding of wellbeing from the perspective of the user and target population. Further investigation on the digital context and its intersection with the naturalistic environment is also required in order to understand the factors and interactions that may influence wellbeing measurement in the context of edtech.

Given the complexities of measuring wellbeing and evaluating edtech with this construct, edtech should approach wellbeing that is context-relevant and informed by multi-disciplinary research.

It is encouraged to use a mixed-method approach in edtech research, with quantitative and qualitative data for the evaluation of wellbeing. It also is recommended to make use of proportional methods when resources are scarce or there are limitations to conduct a more robust research design. Despite some bias, such a research approach should spur the curiosity of edtech designers to promote further advances for this promising field.