RELIABILITY AND VALIDITY IN MEASURING EMOTIONS

The Assessdo social-emotional assessment Catch A Feeling platform tool captures quantitative competencies and dispositions that help youth and adults succeed in school, career, and life. Our measurement tools are designed based on 5 crucial domains (Self-Awareness, Self-Management, Decision Making, Social-Awareness, and Relationship Skills). The Assessdo Catch A Feeling platform tool uses industry standard psychometric tools to provide data that is reliable and valid according to test measures. Our assessment module’s validity measures SEL emotions for formative purposes using quantitative data through qualitative assessing. 

Assessdo uses a quantitative approach designed to convert emotions by encouraging youth and adults to describe their emotions in such a way that the system can instantaneously gauge without a loss of representation in the quality of data, a persons thoughts and emotions. For example, when an AI system processes qualitative data, there is a marginal percentage of accuracy; however when a person inputs data in relation to assigning emotions as numbers, they are self-reporting in a qualitative manner which gives the highest measure of quantitative accuracy.  

How the assessment and platform are validated and employ effective evidence-based tools and practices:

Our proposed computer adaptive assessment platform tools are a replication of a nationally normed study that is validated with the highest levels of reliability and provides a measure of growth overtime. The Assessdo assessment system provides SEL benchmark assessments indicating growth over time measures. Assessdo uses developed formulas that, from vast research, prove methods in capturing emotions. The results from our formative measurements provide deeper, more interpretive solutions to the five domains of SEL.  

Many of our Assessdo tools are aligned with the Geneva Appraisal Method which is one of the prominent methods in capturing emotion. The Geneva Appraisal Method also lends findings related to the social context of the emotional experience and the event, as well as questions on intensity, duration, and regulation of the emotional experience.  

RESEARCH STUDY    By: Shawon Dewan Shahriar

Some instruments contain questions that tap the appraisal criteria suggested by Scherer’s model.

One of our primary studies is “A Comparative Study on Evaluation of Methods in Capturing Emotion. What do we learn capturing emotion with different methods?” by Shawon Dewan Shahriar in which he states in his research, “ Picard mentioned term _ Affective Computing‘ in describing systems that use any of the broad spectrum of emotions and distinguishes between ‗inner experiences‘ and outward emotional expressions‘. In the framework for developing a user experience evaluation methodology integrates cognitive and affective/emotional components where the Cognitive components are instrumental and non-instrumental quality assessments of the human-technology-interaction by the user (Mahlke, 2002) and the Instrumental quality aspects are criteria like the usefulness and ease of use of a system (e.g. Davis, Bagozzi, & Warshaw, 1989).  

Several psychologists have suggested a different number of basic emotions ranging from 2 to 18 categories. But, considerable agreement was on the following six: anger, disgust, fear, happiness, sadness and surprise. Some scholars believe that these emotions have evolved in us as a way for people, regardless of communication differences, to predict what other people are thinking and feeling (Beck 2004). A study conducted among the people in New Guinea who have never seen Caucasians nor been exposed to photographs or television, to see if they could identify specific facial expressions. Researchers showed them pictures of people portraying seven different emotions which are known as core emotions – happiness, anger, sadness, disgust, surprise, fear, and contempt (Ekman & Friesen, 1971). Researchers found that the people of New Guinea could in fact point out the different emotions distinguishing between them. From this experiment researchers came to a decision that these specific emotions are universal and innate. They also varied the pictures of people ranging in age from infants to elders and found that the core emotions are always same, which further supported the discrete emotion hypothesis. The result was also same with the deaf and blind children. (Peter, Crane, 2008)

There are so many user experience evaluation methods. And for evaluating emotion in user experience we have so many different methods. Out of all this evaluation methods I have chosen 3E (Expressing Experiences and Emotion) and Geneva Appraisal Questionnaire (GAQ) as empirical methods to put in survey to capture the emotion. The Geneva Appraisal Questionnaire is the quantitative approach of data gathering while 3E is qualitative approach of data gathering. I have chosen Geneva Appraisal Questionnaire because it assesses emotion as much as possible through recall and verbal report of a participant‘s appraisal process in the case of an emotional episode and to do so the questionnaire designed as forced choice type with a single question with a request of verbal description. Because of these qualities this method is supposed to give the description of the event with details as well as the questionnaire based on the event is supposed to inquire the emotion and the mental and environmental context in a vivid way. It carries the description of the characteristics of the event with the causation of the event and the consequences of the event and also deals with the reaction with respect to the real or expected consequences, intensity and duration of the emotional experience with verbal description of the emotional experience which made me interested put this method into experiment to check. 

There are various instruments to measure emotion starting from simple pen and paper rating scales to high tech equipment‘s. The instrument relying on self-report utilizes rating scales, verbal protocols (verbal methods) or pictograms (nonverbal methods) are used by the user to depict his/her emotion. Two examples of self-report instrument using pictograms for nonverbal assessment of emotions are Emocard and Self-Assessment manikin (SAM). Self-Assessment Manikin represents visually three dimensions of emotion by three axes: pleasure-displeasure, degree of arousal and dominance submissiveness. Emocards Consists 16 cartoon faces (eight male and eight female faces) that depict eight emotional responses. The problem or drawback of these two methods is that they capture only emotions but not the context or cause of them. To clarify the cause or context researchers are required to do further assessment. On the contrast 3E allows the user to depict emotion with a different approach by drawing and as well as writing, so giving information of their feelings and reason behind them in a way of their preference. Participants I have taken total 22 participants both male and female to conduct my evaluation on emotion with Geneva Appraisal Questionnaire (GAQ), and 7 participant with 3E(Expressing Experiences and Emotion). 

Procedure With the experiment the Geneva Appraisal Questionnaire (GAQ) it took 10 to 15 minutes for the participants to answer all the questions. For this method I choose online survey and sent the questionnaire to people with all the guidelines and instruction mentioning in the mail and received the data and also for some participant I gave pen and paper to answer all the questions on the Geneva appraisal questionnaire. With the 3E method to evaluate emotion I let the participant to use Facebook and provided participant with a simple pictorial template for expressing emotions and experiences, in the form of sketched human body. The participants were able to draw a face to the human figure and so projecting her/his emotional state. The human figure was added with two cartoon like speech bubbles one of which was cloud like is used to depict the inner thought and the square like bubble for oral expressions. 

The data collection with Geneva Appraisal Questionnaire was easy to do and it took less time than the 3E method. With the Geneva Appraisal Questionnaire it was possible to collect data without being present in the spot in front of the participant. In contrast with 3E I had to be present in front of the participant and let them using Facebook. But concerning the privacy it was not possible to see really how they were using and sometime I felt all the emotional instances were not captured by the 3E process in their drawing by the users. There were more feelings going on inside the user which the user could not really depict and according to their words it was too abstract to explain or draw. And I felt that the user might get biased by the presence of the researcher in front of the user in evaluating emotion by 3E method. On the other hand the Geneva Appraisal Questionnaire (GAQ) allows the user to recall a memory of an emotional event of felt experience and along with that the questionnaire consisting issues of context and other inquiry based on the emotion. It was strait forward and easy to understand and reflect upon in answering; though some of the participant said that there were too many questions to answer. With 3E some of the participant was happy and spontaneous and according to them the stickman in the template for the experiment was funny and they liked to draw on it. 

Geneva Appraisal Method is one of the prominent method in capturing Emotion. There are a number of attractions to the dimensional approach like the idea of plotting emotions along dimensions respects a fundamental observation that emotions can vary very smoothly, both as they progress in time and from case to case. The Geneva Appraisal Questionnaire was developed by the members of Geneva Emotion Research Group on the basis of Klaus R. Scherer‘s component process model of emotion (CPM). The Component Process Model (CPM) (five components of emotion: cognitive, neurophysiological, motivational, expressive and subjective feeling) is Scherer’s major theory of emotions which regards emotions as the synchronization of many different cognitive and physiological components. The purpose of Geneva Appraisal Questionnaire is to assess, as much as is possible through recall and verbal report, the results of an individual’s appraisal process in the case of a specific emotional episode. To do so, the instrument contains questions that tap the appraisal criteria suggested by Scherer’s model.

The Geneva Appraisal Questionnaire deals with different appraisal goals like Novelty, Intrinsic Pleasantness, Goal/Need Significance, Coping potential, Compatibility with standards, and in addition the questionnaire contains questions on the timing and the social context of the emotional experience and the event, as well as questions on intensity, duration, and regulation of the emotional experience. Affectivity and appraisal systems – According to Roseman‘s (1984) description emotion and affectivity is a multifaceted phenomena consisting of: 1. Cognitive appraisal. Only the events which are judged or appraised to have Significance for our goals, concerns, values, needs, preferences or well-being elicit emotion. 2. Subjective feelings. The appraisal is accompanied by feelings that are good or bad, pleasant or unpleasant, and calm or aroused. 3. Physiological arousal. Emotions are accompanied by autonomic nervous system activity. 4. Expressive behaviors. Emotion is communicated through facial and bodily Expressions, postural and voice changes. 5. Action tendencies. Emotions carry behavioral intentions, and the readiness to act in certain ways.(Gabrys´-Barker 2011). Roseman (1984) emphasizes the significance of our value judgments as factors conducive to how we act in life and he points to the importance of our appraisal systems for effective functioning in life. According to Lazurus and Smith appraisals are generally seen as mostly affective and idiosyncratic in nature, based on one‘s own evaluation of values and factors that contribute to one‘s wellbeing‘ (Lazarus & Smith, 1993).

A full citation of his study is shown below. 

Department of informatics Master’s Programme in Human-Computer Interaction Master thesis 1-year level, 15 credits SPM 2011.03

https://www.diva-portal.org/smash/get/diva2:452381/FULLTEXT01.pdf