Extraction of Lip Contour and Geometric Lip Features for Audio-Visual Phoneme Recognizer

Abstract

This paper presents a lip contour extraction method to extract geometric lip features for audio-visual phoneme recognizer (AVPR). In the first stage of the lip contour extraction technique, Viola-Jones algorithm is applied to each frame of the video to detect the face portion. We segmented the image into rectangular blocks and considered a fixed number of rows and columns to get the region which contains mouth portion. Region of interest (ROI) is calculated based on colour ratio thresholding and largest connected region detection. The elliptic shape guided fuzzy clustering is applied to the combination of green and weighted colour planes for lip segmentation. However, the boundary of the segmented lip is not smooth. Therefore, we propose a refinement method for smoothing the boundary of the segmented lip to get an accurate lip contour. The experimental results performed on GRID audio-visual database work very efficiently for lip contour extraction. The performance of the female speakers (segmented error (SE) = 7.3%, overlap (OL) = 92.7%) is better than male speakers (SE = 10.6%, OL = 87.8%). Each phoneme is modelled by using 10-states Hidden markov model (HMM) with 32 Gaussian mixtures. The phonemes are represented by symbols of international phonetic alphabet. This HMM based audio-visual phoneme recognizer can recognize 31 phonemes and the non speech event; silence (sil) and short pause (sp). By experimental analysis, we have shown the effectiveness of the geometrical lip features in phoneme recognition task under noisy condition.

Keywords: Elliptic shape guided fuzzy clustering, phoneme recognizer, polynomial fitting, region of interest

Cite this Article: Salam Nandakishor, Debadatta Pati. Extraction of Lip Contour and Geometric Lip Features for Audio-Visual Phoneme Recognizer. International Journal of Computer Science and Programming Language. 2020; 6(1): 25–33p.