Publications

1 Social factors in human-agent teaming
de Melo, C., Files, B., Pollard, K., & Khooshabeh, P., A. Moallem (Eds.), Smart and Intelligent Systems, , 2021
Show abstract

Recent decades have seen impressive progress in the development of autonomous technology, such as robots, drones, self-driving cars, and personal assistants. These intelligent agents are able to engage with their surrounding environment in increasingly sophisticated ways. However, as this technology becomes pervasive in society, its success hinges on effective and efficient collaboration with humans. To accomplish this, agents need not only understand the functional aspects of the task, but also the broader social context. Here, we first review relevant psychological theory explaining why and when humans treat agents in a social manner and are socially influenced by them. Second, we summarize experimental evidence showing the importance of verbal (e.g., natural language conversation) and nonverbal (e.g., emotion expressions) communication for successful collaboration between humans and agents. Third, we review recent work showing how perceptions of social group membership with agents influence cooperation. Fourth, we cover research on key individual differences – e.g., anthropomorphic tendency – shaping social interaction with agents. Finally, we identify open challenges and opportunities in this emerging field.

2 Inferring intentions from emotion expressions in social decision making
Gratch, J., & de Melo, C., U. Hess, & S. Hareli (Eds.), The Social Nature of Emotion Expression, 141-160, 2019
Show abstract

In the last decade we have seen increasing experimental evidence that people make important inferences from emotion expressions about others intentions in situations of interdependent decision making. Reverse appraisal has been proposed as one mechanism whereby people retrieve, from emotion displays, information about how others are appraising the ongoing interaction (e.g., does my counterpart find the current outcome to be goal conducive? Does s/he blame me for it?); in turn, from these appraisal attributions, people make inferences about the others' goals (e.g., is my counterpart likely to cooperate?) that shape their decision making. Here we review experimental evidence and progress that has been done in understanding this inferential mechanism and its relationship to other mechanisms for the interpersonal effects of emotion (e.g., emotional contagion and social appraisal). We discuss theoretical implications for our understanding of the role of emotion expression on human decision making, but also practical implications for the growing industry of socially intelligent machines (e.g., personal digital assistants and social robots).

3 Emotion in games.
de Melo, C., Paiva, A., & Gratch, J., M. Angelides, H. Agius (Eds.), The Handbook of Digital Games, 575-592, 2014
Show abstract

Growing interest on the study of emotion in the behavioral sciences has led to the development of several psychological theories of human emotion. These theories, in turn, inspired computer scientists to propose computational models that synthesize, express, recognize and interpret emotion. This cross-disciplinary research on emotion introduces new possibilities for digital games. Complementing techniques from the arts for drama and storytelling, these models can be used to drive believable non-player characters that experience properly-motivated emotions and express them appropriately at the right time; these theories can also help interpret the emotions the human player is experiencing and suggest adequate reactions in the game. This chapter reviews relevant psychological theories of emotion as well as computational models of emotion and discusses implications for games. We give special emphasis to appraisal theories of emotion, undeniably one of the most influential theoretical perspectives within computational research. In appraisal theories, emotions arise from cognitive appraisal of events (e.g., is this event conducive to my goals? Who is responsible for this event? Can I cope with this event?). According to the pattern of appraisals that occur, different emotions are experienced and expressed. Appraisal theories can, therefore, be used to synthesize emotions in games, which are then expressed in different ways. Complementary, reverse appraisal has been recently proposed as a theory for the interpretation of emotion. Accordingly, people are argued to retrieve, from emotion displays, information about how others' are appraising the ongoing interaction, which then leads to inferences about the others' intentions. Reverse appraisal can, thus, be used to infer how human players, from their emotion displays, are appraising the game experience and, from this information, what their intentions in the game are. This information can then be used to adjust game parameters or have non-player characters react to the player's intentions and, thus, contribute to improve the player's overall experience.

4 Modeling gesticulation expression in virtual humans.
de Melo, C., & Paiva, A., N. Magnenat-Thalmann, L. Jain, & N. Ichalkaranje (Eds.), New Advances in Virtual Humans, 133-151, 2008
Show abstract

Gesticulation is the kind of unconscious, idiosyncratic and unconventional gestures humans do in conversation or narration. This chapter reviews efforts made to harness the expressiveness of gesticulation in virtual humans and proposes one such model. First, psycholinguistics research is overviewed so as to understand how gesticulation occurs in humans. Then, relevant computer graphics and computational psycholinguistics systems are reviewed. Finally, a model for virtual human gesticulation expression is presented which supports: (a) real-time gesticulation animation described as sequences of constraints on static (Portuguese Sign Language hand shapes, orientation palm axis, orientation angle and handedness) and dynamic features; (b) synchronization between gesticulation and synthesized speech; (c) automatic reproduction of annotations in GestuRA, a gesticulation transcription algorithm; (d) expression control through an abstract integrated synchronized language – Expression Markup Language (EML). Two studies, which were conducted to evaluate the model in a storytelling context, are also described.

5 Evolutionary expression of emotions in virtual humans using lights and pixels.
de Melo, C., & Paiva, A., J. Tao & T. Tan (Eds.), Affective Information Processing, 313-336, 2008
Show abstract

Artists express emotions through art. To accomplish this they rely on lines, shapes, textures, color, light, sounds, music, words and the body. The virtual humans field has been neglecting the kind of expression we see in the arts. In fact, researchers have tended to focus on gesture, face and voice for the expression of emotions. But why limit ourselves to the body? In this context, drawing on accumulated knowledge from the arts, this chapter describes an evolutionary model for the expression of emotions in virtual humans using lights, shadows, filters and composition. Lighting expression uses lighting techniques from the visual arts to convey emotions through the lights in the environment. Screen expression uses filters and composition to manipulate the virtual human's pixels themselves in a way akin to painting. Emotions are synthesized using the OCC model. To learn how to map affective states into lighting and screen expression, an evolutionary model which relies on genetic algorithms is used. The crossover and mutation operators generate alternatives for the expression of some affective state and a critic ensemble, composed of artificial and human critics, selects among the alternatives.