Skip to Main content Skip to Navigation

Joint analysis of eye movements and EEGs using coupled hidden Markov

Brice Olivier 1, 2 
2 MISTIS - Modelling and Inference of Complex and Structured Stochastic Systems
Inria Grenoble - Rhône-Alpes, Grenoble INP - Institut polytechnique de Grenoble - Grenoble Institute of Technology, LJK - Laboratoire Jean Kuntzmann
Abstract : This PhD thesis consists in jointly analyzing eye-tracking signals and multi-channel electroencephalograms (EEGs) acquired concomitantly on participants doing an information collection reading task in order to take a binary decision - is the text related to some topic or not ? Textual information search is not a homogeneous process in time - neither on a cognitive point of view, nor in terms of eye-movement. On the contrary, this process involves several steps or phases, such as normal reading, scanning, careful reading - in terms of oculometry - and creation and rejection of hypotheses, confirmation and decision - in cognitive terms.In a first contribution, we discuss an analysis method based on hidden semi-Markov chains on the eye-tracking signals in order to highlight four interpretable phases in terms of information acquisition strategy: normal reading, fast reading, careful reading, and decision making.In a second contribution, we link these phases with characteristic changes of both EEGs signals and textual information. By using a wavelet representation of EEGs, this analysis reveals variance and correlation changes of the inter-channels coefficients, according to the phases and the bandwidth. And by using word embedding methods, we link the evolution of semantic similarity to the topic throughout the text with strategy changes.In a third contribution, we present a new model where EEGs are directly integrated as output variables in order to reduce the state uncertainty. This novel approach also takes into consideration the asynchronous and heterogeneous aspects of the data.
Document type :
Complete list of metadata

Cited literature [222 references]  Display  Hide  Download
Contributor : ABES STAR :  Contact
Submitted on : Thursday, October 10, 2019 - 7:07:07 PM
Last modification on : Friday, March 25, 2022 - 9:44:22 AM


Version validated by the jury (STAR)


  • HAL Id : tel-02311373, version 1



Brice Olivier. Joint analysis of eye movements and EEGs using coupled hidden Markov. Signal and Image Processing. Université Grenoble Alpes, 2019. English. ⟨NNT : 2019GREAM030⟩. ⟨tel-02311373⟩



Record views


Files downloads