Tutorial@ACM MM’15

Tutorial on Emotional and Social Signals for Multimedia Research

Full Day Tutorial @ ACM Multimedia 2015


A challenge for human-centred multimedia is the analysis of human communicative behaviour in multimedia content when considering especially the spontaneous non-verbal signals that are generated by humans when interacting with each other. These signals require a different approach to multimedia computing where the methods developed need findings from other disciplines such as social and behavioural psychology, affective computing and social signal processing. This tutorial aims to address the gaps in understanding between these disciplines, providing core knowledge of each domain and to disseminate basic foundational concepts in emotional and social signal research in a very practical and interactive manner.

Tutorial Format:

In this tutorial, we will take a very active approach to analysing the role of emotional and social signals for multi-media research. The tutorial uses our recent IEEE Multimedia article: “Emotional and Social Signals in Multimedia: A Neglected Frontier?” as an initial basis for discussion. Attendees may find reading this article before attending the tutorial to be useful.

Aside lectring about the theories of emotional and social signals, we will use the following papers (with kind permission from all authors involved) as case studies to discuss how the themes covered address research questions in affective computing, social signal processing, and multimedia. We encourage attendees to already take a look at these papers (during the tutorial attendees will be split into groups and will discuss one paper from each category).

  1. Social Signal Processing Papers:
    1. Hayley Hung, Dinesh Babu Jayagopi, Chuohao Yeo, Gerald Friedland, Sileye O. Ba, Jean-Marc Odobez, Kannan Ramchandran, Nikki Mirghafori, Daniel Gatica-Perez: Using audio and video features to classify the most dominant person in a group meeting. ACM Multimedia 2007: 835-838
    2. Mohammadi, Gelareh, Antonio Origlia, Maurizio Filippone, and Alessandro Vinciarelli. “From speech to personality: mapping voice quality and intonation into personality differences.” In Proceedings of the 20th ACM international conference on Multimedia, pp. 789-792. ACM, 2012.
  2. Affective Computing Papers:
    1. Mihalis A. Nicolaou, Hatice Gunes, Maja Pantic: A multi-layer hybrid framework for dimensional emotion classification. ACM Multimedia 2011: 933-936.
    2. Yelin Kim and Emily Mower Provost. “Say Cheese vs. Smile: Reducing Speech-Related Variability for Facial Emotion Recognition.” Proceedings of the ACM International Conference on Multimedia. Florida, USA, November, 2014 .

The aim of these case studies are to gain insights into how to carry out research in each sub-discipline and ultimately to use the lessons learned to shape ones own research for even better research in a challenging intersection between the different domains.

We expect by the end of the day, each attendee will have good insights and a feasible plan of action to carry out research at the cross roads of these disciplines.

Speakers: Hayley Hung & Hatice Gunes

Hayley Hung is an Assistant Professor and Delft Technology Fellow in the Pattern Recognition and Bioinformatics group at TU Delft, The Netherlands, since 2013. Between 2010-2013, she held a Marie Curie Intra-European Fellowship at the Intelligent Systems Lab at the University of Amsterdam. Between 2007-2010, she was a post-doctoral researcher at Idiap Research Institute in Switzerland. She obtained her PhD in Computer Vision from Queen Mary University of London, UK in 2007 and her first degree from Imperial College, UK in Electrical and Electronic Engineering. Her research interests are in social computing, social signal processing, machine learning, and ubiquitous computing. She is local arrangements chair for ACM MM 2016, Workshop co-chair ACM ICMI 2015, area chair of the area on emotional and social signals at ACM MM (2014-2015), co-panel organiser for the panel on Emotional and Signals in Multimedia (ACM MM 2014), Doctoral Symposium co-chair ACM MM (2013). She has organized workshops on human behavior understanding (InterHUB ( AmI 2011), Measuring Behaviour in open spaces (MB 2012), HBU (ACM MM 2013). She is also a special issue guest editor for ACM Transactions on Interactive Intelligent Systems. She has received first prize in the IET Written Premium competition 2009, was nominated for outstanding paper at ICMI 2011, and was named outstanding reviewer at ICME 2014.

Hatice Gunes is a Senior Lecturer (Associate Professor) at Queen Mary University of London, leading the Affective and Human Computing Lab. Her research interests lie in the multidisciplinary areas of affective computing and social signal processing, focusing on automatic analysis of emotional and social behavior and human aesthetic canons, multimodal interaction, computer vision, machine learning, and human-computer and human-robot interactions. She published over 75 technical papers in these areas (Google scholar citations>1700, H-index=20) and was a recipient of awards for Outstanding Paper (IEEE FG’11), Quality Reviewer (IEEE ICME’11) and Best Demo (IEEE ACII’09). She serves as Associate Editor of IEEE Transactions on Affective Computing, on the Management Board of Association for the Advancement of Affective Computing, and the Steering Committee of IEEE Transactions on Affective Computing. She has also served as a Guest Editor of Special Issues in Int’l J. of Synthetic Emotions, Image and Vision Computing, and ACM Transactions on Interactive Intelligent Systems, and member of the Editorial Advisory Board for the Affective Computing and Interaction Book (IGI Global, 2011), cofounder and main organizer of the EmoSPACEWorkshops at IEEE FG’15, FG’13 and FG’11, workshop chair of MAPTRAITS’14, HBU’13 and AC4MobHCI’12, and area chair for ACM Multimedia’ 15, ACM Multimedia’14, IEEE ICME’13, ACM ICMI’13 and ACII’13. She has been involved as PI and Co-I in several projects funded by the Engineering and Physical Sciences Research Council UK (EPSRC) and the British Council.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s