Skip to Main content Skip to Navigation

Content based images retrieval based on implicit gaze annotations

Abstract : One daunting challenge of Content Based Image Retrieval systems is the requirement of annotated databases. To limit the burden of annotation, this thesis proposes a system of image annotation based on gaze data. The purpose is to classify a small set of images according to a target category (binary classification) in order to classify a set of unseen images. First, we have designed a protocol based on visual preference paradigm in order to collect gaze data from different groups of participants during a category identification task. Among the gaze features known to be informative about the intentions of the participants, we have determined a Gaze-Based Intention Estimator (GBIE), computable in real-time; independent from both the participant and the target category. This implicit annotation is better than random annotation but is inherently uncertain. In a second part, the images annotated by the GBIE from the participants’ gaze data are used to classify a bigger set of images with an algorithm that handles label uncertainty: P-SM combining classification and regression SVM. We have determined among different strategies a criterion of relevance in order to discriminate the most reliable labels, involved in the classification part, from the most uncertain labels, involved in the regression part. The average accuracy of P-SVM is evaluated in different contexts and can compete with the performances of standard classification algorithm trained with true-class labels. These evaluations were first conducted on a standard benchmark for comparing with state-of-the-art results and later conducted on food image dataset.
Complete list of metadatas

Cited literature [103 references]  Display  Hide  Download
Contributor : Abes Star :  Contact
Submitted on : Tuesday, March 6, 2018 - 2:47:10 PM
Last modification on : Monday, October 12, 2020 - 10:30:40 AM
Long-term archiving on: : Thursday, June 7, 2018 - 1:57:01 PM


Version validated by the jury (STAR)


  • HAL Id : tel-01724391, version 1



Stéphanie Lopez. Content based images retrieval based on implicit gaze annotations. Databases [cs.DB]. Université Côte d'Azur, 2017. English. ⟨NNT : 2017AZUR4115⟩. ⟨tel-01724391⟩



Record views


Files downloads