Joint analysis of eye movements and EEGs using coupled hidden Markov

Brice Olivier 1, 2
2 MISTIS - Modelling and Inference of Complex and Structured Stochastic Systems
Inria Grenoble - Rhône-Alpes, LJK - Laboratoire Jean Kuntzmann, INPG - Institut National Polytechnique de Grenoble
Abstract : This PhD thesis consists in jointly analyzing eye-tracking signals and multi-channel electroencephalograms (EEGs) acquired concomitantly on participants doing an information collection reading task in order to take a binary decision - is the text related to some topic or not ? Textual information search is not a homogeneous process in time - neither on a cognitive point of view, nor in terms of eye-movement. On the contrary, this process involves several steps or phases, such as normal reading, scanning, careful reading - in terms of oculometry - and creation and rejection of hypotheses, confirmation and decision - in cognitive terms.In a first contribution, we discuss an analysis method based on hidden semi-Markov chains on the eye-tracking signals in order to highlight four interpretable phases in terms of information acquisition strategy: normal reading, fast reading, careful reading, and decision making.In a second contribution, we link these phases with characteristic changes of both EEGs signals and textual information. By using a wavelet representation of EEGs, this analysis reveals variance and correlation changes of the inter-channels coefficients, according to the phases and the bandwidth. And by using word embedding methods, we link the evolution of semantic similarity to the topic throughout the text with strategy changes.In a third contribution, we present a new model where EEGs are directly integrated as output variables in order to reduce the state uncertainty. This novel approach also takes into consideration the asynchronous and heterogeneous aspects of the data.
Document type :
Theses
Complete list of metadatas

Cited literature [222 references]  Display  Hide  Download

https://tel.archives-ouvertes.fr/tel-02311373
Contributor : Abes Star <>
Submitted on : Thursday, October 10, 2019 - 7:07:07 PM
Last modification on : Thursday, October 17, 2019 - 11:20:04 AM

File

OLIVIER_2019_archivage.pdf
Version validated by the jury (STAR)

Identifiers

  • HAL Id : tel-02311373, version 1

Collections

Citation

Brice Olivier. Joint analysis of eye movements and EEGs using coupled hidden Markov. Signal and Image Processing. Université Grenoble Alpes, 2019. English. ⟨NNT : 2019GREAM030⟩. ⟨tel-02311373⟩

Share

Metrics

Record views

135

Files downloads

27