Skip to Main content Skip to Navigation

Contributions to large-scale data processing systems

Abstract : This thesis covers the topic of large-scale data processing systems,and more precisely three complementary approaches: the design of asystem to perform prediction about computer failures through theanalysis of monitoring data; the routing of data in a real-time systemlooking at correlations between message fields to favor locality; andfinally a novel framework to design data transformations usingdirected graphs of blocks.Through the lenses of the Smart Support Center project, we design ascalable architecture, to store time series reported by monitoringengines, which constantly check the health of computer systems. We usethis data to perform predictions, and detect potential problems beforethey arise.We then dive in routing algorithms for stream processing systems, anddevelop a layer to route messages more efficiently, by avoiding hopsbetween machines. For that purpose, we identify in real-time thecorrelations which appear in the fields of these messages, such ashashtags and their geolocation, for example in the case of tweets. Weuse these correlations to create routing tables which favor theco-location of actors handling these messages.Finally, we present λ-blocks, a novel programming framework to computedata processing jobs without writing code, but rather by creatinggraphs of blocks of code. The framework is fast, and comes withbatteries included: block libraries, plugins, and APIs to extendit. It is also able to manipulate computation graphs, foroptimization, analyzis, verification, or any other purposes.
Document type :
Complete list of metadata

Cited literature [119 references]  Display  Hide  Download
Contributor : Abes Star :  Contact
Submitted on : Wednesday, October 10, 2018 - 8:59:05 AM
Last modification on : Thursday, November 19, 2020 - 1:02:02 PM
Long-term archiving on: : Friday, January 11, 2019 - 1:04:12 PM


Version validated by the jury (STAR)


  • HAL Id : tel-01891825, version 1



Matthieu Caneill. Contributions to large-scale data processing systems. Other [cs.OH]. Université Grenoble Alpes, 2018. English. ⟨NNT : 2018GREAM006⟩. ⟨tel-01891825⟩



Record views


Files downloads