Semantic Description of Humans in Images

Gaurav Sharma 1, 2
1 LEAR - Learning and recognition in vision
Inria Grenoble - Rhône-Alpes, LJK - Laboratoire Jean Kuntzmann, INPG - Institut National Polytechnique de Grenoble
2 Equipe Image - Laboratoire GREYC - UMR6072
GREYC - Groupe de Recherche en Informatique, Image, Automatique et Instrumentation de Caen
Abstract : In the present thesis we are interested in semantic description of humans in images. We propose to describe humans with the help of (i) semantic attributes e.g. male or female, wearing a tee-shirt, (ii) actions e.g. riding a horse, running and (iii) facial expressions e.g. smiling, angry. First, we propose a new image representation to better exploit the class specific spatial information. The standard representation \ie spatial pyramids, has two shortcomings. It assumes that the distribution of spatial information (i) is uniform and (ii) is same for all tasks. We address these shortcomings by learning the discriminative spatial information for a specific task. Further, we propose a model that adapts the spatial information for each image for a given task. This lends more flexibility to the model and allows for misalignments of discriminative regions e.g. the legs may be at different positions, in different images for running class. Finally, we propose a new descriptor for facial expression analysis. We work in the space of intensity differences of local pixel neighborhoods and propose to learn the quantization of the space and use higher order statistics of the difference vector to obtain more expressive descriptors. We introduce a challenging dataset of human attributes containing 9344 human images, sourced from the internet, with annotations for 27 semantic attributes based on sex, pose, age and appearance/clothing. We validate the proposed methods on our dataset of human attributes as well as on publicly available datasets of human actions, fine grained classification involving human actions and facial expressions. We also report results on related computer vision datasets, for scene recognition, object image classification and texture categorization, to highlight the generality of our contributions.
Complete list of metadatas

Cited literature [116 references]  Display  Hide  Download
Contributor : Thoth Team <>
Submitted on : Thursday, December 20, 2012 - 12:07:57 PM
Last modification on : Tuesday, February 5, 2019 - 12:12:44 PM
Long-term archiving on : Thursday, March 21, 2013 - 3:48:51 AM


  • HAL Id : tel-00767699, version 1


Gaurav Sharma. Semantic Description of Humans in Images. Computer Vision and Pattern Recognition [cs.CV]. Université de Caen, 2012. English. ⟨tel-00767699⟩



Record views


Files downloads