STUDY OF MUSICAL INFLUENCE ON FACE USING THE LOCAL BINARY PATTERN (LBP) APPROACH

BARDEKAR A.A.¹*, DESHMUKH V.M.^2
1Department of CSE, P.R.M.I.T & R, College of Engineering & Technology, Badnera, Near Anjangaon Bari Road, Amravati, MS, India.
²Department of CSE, P.R.M.I.T & R, College of Engineering & Technology, Badnera, Near Anjangaon Bari Road, Amravati, MS, India.
* Corresponding Author : abardekar@rediff.com

Received : 21-02-2012     Accepted : 15-03-2012     Published : 19-03-2012
Volume : 3     Issue : 1       Pages : 64 - 67
J Signal Image Process 3.1 (2012):64-67

Different types of music evoke different feelings and emotions. Certain sounds produce joy, others grief and yet others affection and tenderness. According to Indian aesthetics, each poem or musical composition produces a certain rasa (emotion). Local Binary Patterns (LBP) have been well exploited for facial image analysis in the existing work, the LBP histograms are extracted from local facial regions, and used as a whole for the regional description. In this review paper we propose LBP Histogram (LBP) bins for the task of facial expression recognition while listening to Indian classical ragas. Our experiments illustrate that the selected LBP bins provide a compact and discriminative facial expression representation. The selected LBP bins will be used to obtain the best recognition performance rate on collected database. The local binary pattern (LBP) operator is defined as a gray-scale invariant texture measure, derived from a general definition of texture in a local neighborhood. Due to its discriminative power and computational simplicity, the LBP texture operator has become a popular approach in various applications, including visual inspection, image retrieval, remote sensing, biomedical image analysis, motion analysis, environment modeling, and outdoor scene analysis .

LBP, Facial Expression Analysis, Histogram, Classical raags, Emotional Face Expression.

According to Indian aesthetics, each poem or musical composition produces a certain rasa (emotion). Literally, rasa means juice, but in musical context it implies more than an aesthetic relish-a transcendental experience. Some consider rasa as sentiment, but it is something subtle, even more than an emotion or empathy. Different types of music evoke different feelings and emotions. Certain sounds produce joy, others grief and yet others affection and tenderness. Rasa is essentially emotional reaction and awareness of it. The feeling may be pleasant or sad, high or low, sublime or ludicrous, actual or imaginary, furious or peaceful. By and large, each raga is supposed to evoke a single emotion. For example, the notes of Khamaj raga are said to create a Happiness mood. Kafi raga is tranquilizing and pleasing and gives a feeling of peace. During the past few years, we have witnessed a development of a computationally simple yet very efficient texture operator called Local Binary Patterns (LBP) [3] . The LBP operator is defined as a grayscale invariant texture measure, derived from a general definition of texture in a local neighborhood. Through its extensions, the LBP operator has been made into a really powerful measure of image texture, showing excellent results in terms of accuracy and computational complexity in many empirical studies. The LBP operator can be seen as a unifying approach to the traditionally divergent statistical and structural models of texture analysis. Perhaps the most important property of the LBP operator in real world applications is its tolerance against illumination changes. Another equally important is its computational simplicity, which makes it possible to analyze images in challenging real-time settings. The LBP method has already been used in a large number of applications all over the world, including visual inspection, image retrieval, remote sensing, biomedical image analysis, face image analysis, motion analysis, environment modeling, and outdoor scene analysis. More recent developments showed that the LBP approach also provides outstanding results in representing and analyzing faces in both still images and video sequences.
The image matching methods can be divided into the following three categories: regional matching, feature matching and phase matching. Feature-based matching method can match the remarkable and easy to find the human face of the feature points or point sets, feature matching primitive contains a satisfactory statistical characteristics and the flexibility of programming algorithms [1] . Analyzing facial images is very useful in several applications, like tracking and identification of persons, human-machine interaction, video conferencing and content-based image retrieval. Facial image analysis may include face detection and facial feature extraction, face tracking and pose estimation, face and facial expression recognition, and face modeling and animation. All these tasks are challenging due to the fact that a face is a dynamic and non-rigid object which is difficult to handle. Its appearance varies due to changes in pose, expressions, illuminations and other factors such as age and make-up. Therefore, one needs a facial representation that is robust to these factors. Ideally, the representation should be discriminative, compact and easy to compute. In this context, the LBP based facial representation provided excellent results that outperformed many state-of-the-art methods in several face related tasks. The approach has also inspired several other research groups, including works on face recognition, facial expression recognition, gender recognition, face detection, face authentication and shape localization. The success of using LBP in facial image analysis proves simplicity and efficiency of LBP as a local texture operator, for face representation [3] .
The most important properties of LBP features are their tolerance against monotonic illumination changes and their computational simplicity. In the original LBP-based facial representation, as shown in “ [Fig-3] ”, face images are first equally divided into non-overlapping sub-regions to extract the LBP histograms within each sub-region, which are then concatenated into a single, spatially enhanced feature histogram.
Possible criticisms of this method are that dividing the face into a grid of sub-regions is somewhat arbitrary, as sub-regions are not necessary well aligned with facial features, and that the resulting facial representation suffers from fixed size and position of sub-regions. To address these, in, by shifting and scaling a sub-window over face images, many more sub-regions are obtained. “ [Fig-2] ” shows the selected sub-regions for each facial expression [7] . In most of the existing work, LBP histograms are extracted from local facial regions as the region-level description, where the n-bin histogram is utilized as a whole. However, not all bins in the LBP histogram are necessary to contain useful information for facial representation. It is helpful and interesting to have a closer look at the local LBP histogram at the bin level, to identify the discriminative LBP-Histogram (LBPH) bins for better facial representation. This paper determines the emotional expression state of a person listening to different raga, emotional state such as happiness, sadness, surprise, neutral, fear and disgust, regardless of the identity of the face. In an approach to facial expression recognition from static images using LBP histograms will be computed over non-overlapping blocks for face description. The Linear Programming (LP) technique will be used to classify seven facial expressions: anger, disgust, fear, happiness, sadness, surprise and neutral. We propose a Local Binary Pattern Histogram (LBP) bins for the task of facial expression recognition while listening to Indian classical ragas. Our experiments will illustrate that the selected LBP bins provide a compact and discriminative facial expression representation. The selected LBP bins will be used to obtain the best recognition performance rate on collected database. The local binary pattern (LBP) operator is defined as a gray-scale invariant texture measure, derived from a general definition of texture in a local neighborhood. Due to its discriminative power and computational simplicity, the LBP texture operator has become a popular approach in various applications, including visual inspection, image retrieval, remote sensing, biomedical image analysis, motion analysis, environment modeling, and outdoor scene analysis.

Emotions give meaning to our lives. No aspect of our mental life is more important to the quality and meaning of our existence than emotions. They make life worth living, or sometimes ending. The English word “emotion” is derived from the French word ‘emouvoir’ which means ‘move’. Great classical philosophers-Plato, Aristotle, Spinoza, Descartes conceived emotion as responses to certain sorts of events triggering bodily changes and typically motivating characteristics behavior [6] . Moving on to the 19th century, one of the important works on facial expression analysis that has a direct relationship to the modern day science of automatic facial expression recognition was the work done by Charles Darwin. In 1872, Darwin wrote a treatise that established the general principles of expression and the means of expressions in both humans and animals. He also grouped various kinds of expressions into similar categories. The categorization is as follows [5] , Low spirits, anxiety, grief, dejection, despair, Joy, high spirits, love, tender feelings, and devotion. Furthermore, Darwin also cataloged the facial deformations that occur for each of the above mentioned class of expressions. For example: “the contraction of the muscles round the eyes when in grief”, “the firm closure of the mouth when in reflection”, “the depression of the corners of the mouth when in low spirits”.
Traditionally facial expressions have been studied by clinical and social psychologists, medical practitioners, actors and artists. However in the last quarter of the 20th century, with the advances in the fields of robotics, computer graphics and computer vision, animators and computer scientists started showing interest in the study of facial expressions [5] . As a powerful means of texture description, LBP features have been widely exploited in many applications. For facial image analysis, LBP features have been extensively exploited recently. local LBP histograms as probability distributions and represents a generic face model by a collection of LBP histograms presented to extract multi-scale LBP histograms from each local regions, which has shown promising performance for face recognition. In Multi-scale Block LBP for face recognition, the computation is done based on average values of block sub-regions, instead of individual pixel. The volume LBP and LBP from three orthogonal planes for dynamic texture recognition shows promising performance on dynamic facial expression recognition by combining appearance and motion. The efficiency of the LBP approach in various face related tasks are discussed below [3] .
• Multiview Face Detection- Face detection determines whether there are faces present in an image (or video) and find the location, size and pose of each face. In an LBP based facial representation for frontal face detection. The approach uses a second degree polynomial kernel SVM for classification [3] . A real-time multi-view detector suitable for real world environments such as video surveillance, mobile devices and content based video retrieval. A real-time operation was thus achieved with as good detection accuracy as the original, which was a much slower approach.
• Eye Detection- A robust approach for eye detection using Haar-like features extracted from LBP images. Thus, the images are first filtered by LBP operator (LBP8, 1) and then Haar-like features are extracted and used with AdaBoost for building a cascade of classifiers.
• Face Recognition- To assess the performance of the LBP based facial representation in face recognition, a face recognition system using a nearest neighbor classifier and compared the LBP approach against well-known methods such as PCA and Bayesian intra/extra personal (BIC).
• Facial Expression Recognition- The goal of facial expression recognition is to determine the emotional state of the face, for example, happiness, sadness, surprise, neutral, anger, fear, and disgust, regardless of the identity of the face. In, an approach to facial expression recognition from static images was developed using LBP histograms computed over non-overlapping blocks for face description. The Linear Programming (LP) technique was adopted to classify seven facial expressions: anger, disgust, fear, happiness, sadness, surprise and neutral.
• Visual-Speech Recognition- Visual speech recognition aims at determining what a person is saying only from the lip movements. This plays an important role in speech recognition under noisy conditions or for listeners with hearing impairment. The Volume LBP approach is applied to the task of visual speech recognition. The movements of mouth regions are described using local binary patterns from XY, XT and YT planes, combining local features from pixel, block and volume levels. Raag literally means that which colors the mind. It is a collection of melodic gestures and a technique for developing them. Traditionally music is said to evoke seven basic Emotions sadness, romance, peace, strength, anger, dispassion, devotion. Each raag elicits a unique emotional state (rasa) consisting of one or more of these emotions [8] .

Facial Expression is one of the most powerful, nature, and immediate means for human beings to communicate their emotions and intentions. Due to its potential applications, facial expression recognition has attracted much attention over two decades. Though much progress has been made, recognizing facial expression with a high accuracy remains to be difficult due to the complexity and variety of facial expressions. With this approach we are performing an experimental study to find out while listening to classical raags whether emotions are generated and how they get reflected on face. For this purpose to extract facial expression we are using an LBP approach.

Local Binary Pattern (LBP) features have performed very well in various applications, including texture classification and segmentation, image retrieval and surface inspection [4] . LBP is a simple but very efficient texture operator which labels the pixels of an image by Thresholding the 3*3 neighborhood of each pixel with the value of the center pixel and considers the result as a binary number. [Fig-1] shows an example of LBP calculation. The value of the LBP code of a pixel (x_c, y_c) is given by:



Where gc corresponds to the gray value of the center pixel (X_c, Y_c), g_p refers to gray values of P equally spaced pixels on a circle of radius R, and s defines a Thresholding function as follows:



The calculation of the LBP codes can be easily done in single scan through the image. The 256-bin histogram of the labels computed over a region can be used as a texture descriptor. The original LPB operator has been extended to consider different neighborhood sizes [2] . For example, the operator LBP_4,1 uses only 4 neighbors while LBP_16,2 considers the 16 neighbors on a circle of radius 2. In general, the operator LBP_P,R refers to a neighborhood size of P equally spaced pixels on a circle of radius R that form a circularly symmetric neighbor set [3] . [Fig-2] shows some examples of neighborhood sets. LBP_P,R produces 2^P different output values, corresponding to the 2^P different binary patterns that can be formed by the P pixels in the neighbor set. It has been shown that certain bins contain more information than others. Therefore, it is possible to use only a subset of the 2^P local binary patterns to describe the textured images. Fundamental pattern (called also "uniform" patterns”) as those with a small number of bitwise transitions from 0 to 1 and vice versa. For example, 00000000 and 11111111 contain 0 transitions while 00000110 and 01111000 contain 2 transitions and so on. Accumulating the patterns which have more than 2 transitions into a single bin yields an LBP descriptor, denoted LBP ^u2_P,R, with less than 2^P bins. For example, the number of labels for a neighborhood of 8 pixels is 256 for standard LBP and 59 for LBP^u2_{8 1}. For the 16-neighborhood the numbers are 65536 and 243, respectively.

Music evokes different feelings and emotions. Certain sounds produce joy, others grief and yet others affection and tenderness. The LBP based facial Representation outperformed many state-of-the-art methods in several face related tasks. The approach includes work on emotional facial expression recognition while listening to Indian classical ragas.

[1] Dongcheng Shi, Fengguang Shi (2010) IEEE, A Face Analysis and Description Algorithm Based on Depth-Information, 978-1-42244-6943-7.  
» CrossRef   » Google Scholar   » PubMed   » DOAJ   » CAS   » Scopus  

[2] Kwang HoAn and Myung Jin chung (2009) IEEE, Cognitive Face Analysis System for Future Interactive TV, 0098 3063/09.  
» CrossRef   » Google Scholar   » PubMed   » DOAJ   » CAS   » Scopus  

[3] Abdenvour Hadid (2008) IEEE, The Local Binary Pattern Approach and its Application to Face Analysis, 978-1-4244-3322-3.  
» CrossRef   » Google Scholar   » PubMed   » DOAJ   » CAS   » Scopus  

[4] Jo chang-yeon (2008) Face Detection using LBP feature.  
» CrossRef   » Google Scholar   » PubMed   » DOAJ   » CAS   » Scopus  

[5] Vinay Kumar Bettadapura. Face Expression Recognition and Analysis: The State of the Art.  
» CrossRef   » Google Scholar   » PubMed   » DOAJ   » CAS   » Scopus  

[6] Wieczorkowska A., Datta A.K., Sengupta R., Dey N. and Mukherji B. Center for Development of Advanced Computing.  
» CrossRef   » Google Scholar   » PubMed   » DOAJ   » CAS   » Scopus  

[7] Shan C. and Gritti T. Learning Discriminative LBP-Histogram Bins for Facial Expression Recognition.  
» CrossRef   » Google Scholar   » PubMed   » DOAJ   » CAS   » Scopus  

[8] Chordia P. and Rae A., Understanding Emotion in Raag: An Empirical Study of Listener Responses.  
» CrossRef   » Google Scholar   » PubMed   » DOAJ   » CAS   » Scopus  

	Fig. 1- Example of an LBP calculation
	Fig. 2- Neighborhood set for different (P,R) P=4, R=1.0, P=8, R=1.0, P=12, R=2.5 P=16, R=4.0
	Fig. 3- Examples of an LBP based facial representation