, Amélioration des Performances, de la Durabilité et de la Cohérence dans les Systèmes de Stockage pour Streaming

, Elles présentent des exigences similaires : un modèle de données basé sur les flux, une faible latence et un haut débit, et dans de nombreux cas, elles nécessitent une cohérence forte de données. Cependant, les solutions existantes, telles qu'Apache Kafka, ne sont pas optimisées en termes de disponibilité et de cohérence. Dans cette contribution, nous présentons la conception et la mise en oeuvre de mécanismes de durabilité dans KerA, un système de stockage pour streaming à latence faible. KerA fournit un accès aux données a granularité fine, permettant un débit élevé tout en respectant la charge des clients. En outre, Kera permet un service de données continu, même en cas de défaillance, sans sacrifier la cohérence de données, Alors que Les nouvelles applications generant de larges volumes de données, les analyses et les simulations scientifiques, présentent des besoins sans précédent en termes de performances de stockage, 2018.

S. ?-yacine-taleb, G. Ibrahim, T. Antoniu, and . Cortes, Characterizing Performance and Energy-efficiency of The RAMCloud Storage System, Proceedings of the The 37th IEEE International Conference on Distributed Computing Systems (ICDCS'17), 2017.

, Publications dans des Workshops Internationaux ? Yacine Taleb. Optimizing Fault-Tolerance in In-memory Storage Systems, in the EuroSys 2018 Doctoral Workshop (EuroDW'18), 2018.

S. ?-yacine-taleb, G. Ibrahim, T. Antoniu, and . Cortes, An Empirical Evaluation of How The Network Impacts The Performance and Energy Efficiency in RAMCloud, Workshop on the Integration of Extreme Scale Computing and Big Data Management and Analytics in conjunction with IEEE/ACM CCGRID'17, 2017.

?. Y. Taleb, S. Ibrahim, G. Antoniu, and T. Cortes, Understanding how the network impacts performance and energy-efficiency in the RAMCloud storage system, Big Data Analytics : Challenges and Opportunities, held in conjuction with ACM/IEEE SC'16, 2016.

H. Amur, J. Cipar, and V. Gupta, Robust and flexible power-proportional storage, ACM Symposium on Cloud Computing (SoCC '10), pp.217-228, 2010.

. Architecture-of-giants, Data Stacks at Facebook, Netflix, Airbnb, and Pinterest

B. Atikoglu, Y. Xu, and E. Frachtenberg, Workload Analysis of a Large-scale Key-value Store, Proceedings of the 12th ACM SIGMETRICS/PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems. SIGMETRICS '12, pp.53-64, 2012.

A. Luiz and . Barroso, The Price of Performance, Queue 3.7 (Sept. 2005), pp.48-53

L. Barroso, M. Marty, and D. Patterson, Attack of the Killer Microseconds, Commun. ACM, vol.60, pp.48-54, 2017.

E. Brewer, Pushing the CAP: Strategies for Consistency and Availability, Computer 45, vol.2, pp.23-29, 2012.

N. Bronson, Z. Amsden, and G. Cabrera, TAO: Facebook's Distributed Data Store for the Social Graph, Presented as part of the 2013 USENIX Annual Technical Conference (USENIX ATC 13), 2013.

K. Choi, R. Soma, and M. Pedram, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (TCAD), vol.24, pp.18-28, 2005.

A. Cidon, S. Rumble, and R. Stutsman, Copysets: Reducing the Frequency of Data Loss in Cloud Storage, Presented as part of the 2013 USENIX Annual Technical Conference (USENIX ATC 13), pp.978-979, 2013.

B. F. Cooper, A. Silberstein, and E. Tam, Benchmarking Cloud Serving Systems with YCSB, Proceedings of the 1st ACM Symposium on Cloud Computing. SoCC '10, pp.978-979, 2010.

A. Couchedb and . Apache, , pp.2018-2024, 2018.

, CPU-The New Bottleneck? 2016

, The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software, 2009.

, Why Haven't CPU Clock Speeds Increased in the Last Few Years?, 2014.

E. M. Curry, Message-Oriented Middleware, Middleware for Communications, pp.1-28, 2004.

P. Lake and P. Crowther, Concise Guide to Databases, Topics in Computer Science. 2013

J. Dean and S. Ghemawat, MapReduce: Simplified Data Processing on Large Clusters, Communications of the ACM (CACM), vol.51, pp.107-113, 2008.

G. Decandia, D. Hastorun, and M. Jampani, Dynamo: Amazon's Highly Available Key-value Store, Proceedings of Twenty-first ACM SIGOPS Symposium on Operating Systems Principles. SOSP '07, pp.205-220, 2007.

A. Dragojevi´cdragojevi´c, D. Narayanan, and O. Hodson, FaRM: Fast Remote Memory, Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation. NSDI'14, pp.978-979, 2014.

A. Dragojevi´cdragojevi´c, D. Narayanan, and E. B. Nightingale, No Compromises: Distributed Transactions with Consistency, Availability, and Performance, Proceedings of the 25th Symposium on Operating Systems Principles. SOSP '15, pp.54-70, 2015.

, Data Center Efficiency Assessment, pp.2018-2024, 2014.

, 400 Gb/s Ethernet Task Force, IEEE, vol.802, 2018.

B. Fan, D. G. Andersen, and M. Kaminsky, MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing, Presented as part of the 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI 13, pp.978-979, 2013.

X. Fan, W. Weber, and L. Barroso, Power Provisioning for a Warehouse-sized Computer, Proceedings of the 34th Annual International Symposium on Computer Architecture. ISCA '07, pp.13-23, 2007.

R. Patgiri and A. Ahmed, Big Data: The V's of the Game Changer Paradigm, IEEE International Conference on High Performance Computing and Communications (HPCC '16, pp.17-24, 2016.

S. Guo, R. Dhamankar, and L. Stewart, DistributedLog: A High Performance Replicated Log Service, 2017 IEEE 33rd International Conference on Data Engineering (ICDE), pp.1183-1194, 2017.

H. Sap, , pp.2018-2024, 2018.

, SAP HANA sales fly but there's more to the in-memory story, 2013.

, IB Specification, vol.1, 2015.

I. K. Trends and . Perkins, , 2018.

W. Jiang, J. Liu, and H. Jin, High performance MPI-2 one-sided communication over InfiniBand, IEEE International Symposium on Cluster Computing and the Grid, pp.531-538, 2004.

R. Johnson, I. Pandis, and R. Stoica, Scalability of write-ahead logging on multicore and multisocket hardware, The VLDB Journal, vol.21, issue.2, pp.239-263, 2012.

P. Flavio, I. Junqueira, B. Kelly, and . Reed, Durability with BookKeeper". In: SIGOPS Oper. Syst. Rev, vol.47, pp.9-15, 2013.

, Kafka cluster healing and workload balancing

, Replicas Falling Behind, 2018.

, Exactly-once Semantics are Possible: Here's How Kafka Does it, 2018.

A. Kalia, M. Kaminsky, and D. G. Andersen, FaSST: Fast, Scalable and Simple Distributed Transactions with Two-sided (RDMA) Datagram RPCs, Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation. OSDI'16, pp.978-979, 2016.

A. Kalia, M. Kaminsky, and D. G. Andersen, Using RDMA Efficiently for Key-value Services, SIGCOMM Comput, vol.44, pp.295-306, 2014.

R. Kallman, H. Kimura, and J. Natkins, H-store: A High-performance, Distributed Main Memory Transaction Processing System, Proc. VLDB Endow, vol.1, pp.1496-1499, 2008.

T. Rini, M. Kaushik, and . Bhandarkar, GreenHDFS: Towards an energy-conserving, storage-efficient, hybrid Hadoop compute cluster, USENIX International Conference on Power Aware Computing and Systems (HotPower '10), pp.1-9, 2010.

J. Kreps, N. Narkhede, and J. Rao, Kafka: A distributed messaging system for log processing, Proceedings of 6th International Workshop on Networking Meets Databases (NetDB), 2011.

A. Lakshman and P. Malik, Cassandra: A Decentralized Structured Storage System, SIGOPS Oper. Syst. Rev, vol.44, 2010.

C. Lee, J. Seo, A. Park, and . Kejriwal, Implementing Linearizability at Large Scale and Low Latency, Proceedings of the 25th Symposium on Operating Systems Principles. SOSP '15, pp.71-86, 2015.

B. Li, Z. Ruan, and W. Xiao, KV-Direct: High-Performance In-Memory Key-Value Store with Programmable NIC, Proceedings of the 26th Symposium on Operating Systems Principles. SOSP '17, pp.137-152, 2017.

H. Li, A. Ghodsi, and M. Zaharia, Tachyon: Reliable, Memory Speed Storage for Cluster Computing Frameworks, Proceedings of the ACM Symposium on Cloud Computing. SOCC '14, vol.6, pp.1-6, 2014.

H. Lim, D. Han, and D. G. Andersen, MICA: A Holistic Approach to Fast In-memory Key-value Storage, Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation. NSDI'14, pp.978-979, 2014.

A. Malik, A. Malik, and K. Hiekkanen, Impact of privacy, trust and user activity on intentions to share Facebook photos, Journal of Information, Communication and Ethics in Society, vol.14, pp.364-382, 2016.

. Ovidiu-cristian, A. Marcu, G. Costan, and . Antoniu, KerA: Scalable Data Ingestion for Stream Processing, ICDCS 2018-38th IEEE International Conference on Distributed Computing Systems, pp.1-6, 2018.

. Memcached, , pp.2018-2024, 2018.

, Memory Prices from 1957 to, pp.2018-2024, 2017.

C. Mitchell, Y. Geng, and J. Li, Using One-Sided RDMA Reads to Build a Fast, CPU-Efficient Key-Value Store, Presented as part of the 2013 USENIX Annual Technical Conference (USENIX ATC 13), pp.103-114, 2013.

R. Mittal, . Vinh-the, N. Lam, and . Dukkipati, TIMELY: RTT-based Congestion Control for the Datacenter, Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication. SIGCOMM '15, pp.537-550, 2015.

C. Mohan, D. Haderle, and B. Lindsay, ARIES: A Transaction Recovery Method Supporting Fine-granularity Locking and Partial Rollbacks Using Write-ahead Logging, In: ACM Trans. Database Syst, vol.17, pp.94-162, 1992.
DOI : 10.1145/128765.128770
URL : http://www.sai.msu.su/~megera/postgres/gist/papers/concurrency/p94-mohan.pdf

. Mongodb, , pp.2018-2024, 2018.

J. Nieplocha and B. Carpenter, ARMCI: A portable remote memory copy library for distributed array libraries and compiler run-time systems". In: Parallel and Distributed Processing: 11th IPPS/SPDP'99 Workshops Held in Conjunction with the 13th International Parallel Processing Symposium and 10th Symposium on Parallel and Distributed Processing, pp.533-546, 1999.

R. Nishtala, H. Fugal, and S. Grimm, Scaling Memcache at Facebook, Presented as part of the 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI 13), pp.385-398, 2013.

, Database Management System Choices Overview, vol.2, 2008.

J. Ousterhout, A. Gopalan, and A. Gupta, The RAMCloud Storage System, ACM Trans. Comput. Syst, vol.33, issue.3, 2015.

A. Papagiannis, G. Saloustros, and P. González-férez, Tucana: Design and Implementation of a Fast and Efficient Scale-up Key-value Store, Proceedings of the 2016 USENIX Conference on Usenix Annual Technical Conference. USENIX ATC '16, pp.537-550, 2016.

M. Poke and T. Hoefler, DARE: High-Performance State Machine Replication on RDMA Networks, Proceedings of the 24th International Symposium on High-Performance Parallel and Distributed Computing. HPDC '15, pp.107-118, 2015.

A. C. Vijayan-prabhakaran, R. H. Arpaci-dusseau, and . Arpaci-dusseau, Analysis and Evolution of Journaling File Systems, Proceedings of the Annual Conference on USENIX Annual Technical Conference. ATEC '05, pp.8-8, 2005.

. Pravega, , pp.2018-2024, 2018.

D. Pritchett, BASE: An Acid Alternative, vol.6, pp.48-55, 2008.

A. Pulsar and . Apache, , pp.2018-2024, 2018.

T. Rabl, S. Gómez-villamor, and M. Sadoghi, Solving Big Data Challenges for Enterprise Application Performance Management, Proc. VLDB Endow, vol.5, pp.2150-8097, 2012.

. Redis, , pp.2018-2024, 2018.

R. Robert and E. Eric, Introducing CloudLab: Scientific Infrastructure for Advancing Cloud Architectures and Applications, pp.1045-9219

, RDMA over Converged Ethernet, pp.2018-2024, 2018.

M. Rosenblum and J. K. Ousterhout, The Design and Implementation of a Log-structured File System, ACM Trans. Comput. Syst, vol.10, issue.1, pp.26-52, 1992.

S. M. Rumble, D. F. Mazières, and J. K. Ousterhout, Memory and Object Management in RAMCloud, 2014.

, Big Data-What Is It? SAS, 2013.

P. Schulz, M. Matthe, and H. Klessig, Latency Critical IoT Applications in 5G: Perspective on the Design of Radio Interface and Network Architecture, IEEE Communications Magazine, vol.55, pp.70-78, 2017.

K. Shvachko, H. Kuang, and S. Radia, The Hadoop Distributed File System, Proceedings of the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST). MSST '10, pp.1-10, 2010.

, The Square Kilometer Array Project, 2018.

, SteamCharts: An ongoing analysis of Steam concurrent players. Streaml, 2018.

M. Stonebraker, U. Çetintemel, and S. Zdonik, The 8 Requirements of Real-time Stream Processing, SIGMOD Rec, vol.34, pp.42-47, 2005.

M. Stonebraker, What is Big Data, Communications of the ACM, 2012.

. Messaging, , 2017.

M. Su, M. Zhang, and K. Chen, RFP: When RPC is Faster Than Server-Bypass with RDMA, Proceedings of the Twelfth European Conference on Computer Systems. EuroSys '17, pp.1-15, 2017.

, The BDEC 'Pathways to Convergence

, The Environmental Toll of a Netflix Binge

, The OpenCompute Project

E. Thereska, A. Donnelly, and D. Narayanan, Sierra: Practical power-proportionality for data center storage, ACM European Conference on Computer Systems (EuroSys '11, pp.169-182, 2011.

N. Aniruddha, N. Udipi, N. Muralimanohar, and . Chatterjee, Rethinking DRAM Design and Organization for Energy-constrained Multi-cores, Proceedings of the 37th Annual International Symposium on Computer Architecture. ISCA '10, pp.978-979, 2010.

Y. Wang, L. Zhang, and J. Tan, HydraDB: A Resilient RDMA-driven Key-value Middleware for In-memory Cluster Computing, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. SC '15, vol.22, 2015.

, When Streams Fail: Implementing a Resilient Apache Kafka Cluster at Goldman Sachs

D. Wong and M. Annavaram, KnightShift: Scaling the Energy Proportionality Wall Through Server-Level Heterogeneity, Proceedings of the 2012 45th Annual IEEE/ACM International Symposium on Microarchitecture. MICRO-45, pp.119-130, 2012.

T. Ylönen, Concurrent Shadow Paging: A New Direction for Database Research, 1992.

D. Zachary, . Stephens, Y. Skylar, F. Lee, and . Faghri, Big Data: Astronomical or genomical?, In: PLOS Biology, vol.13, pp.1-11, 2015.

M. Zaharia, M. Chowdhury, and T. Das, Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing, Presented as part of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12), pp.978-931971, 2012.

H. Zhang, B. M. Tudor, and G. Chen, Efficient In-memory Data Management: An Analysis, Proc. VLDB Endow, vol.7, pp.833-836, 2014.

H. Zhang, M. Dong, and H. Chen, Efficient and Available In-memory KV-Store with Hybrid Erasure Coding and Replication, 14th USENIX Conference on File and Storage Technologies (FAST 16), pp.978-979, 2016.

Y. Zhu, H. Eran, and D. Firestone, Congestion Control for Large-Scale RDMA Deployments, Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication. SIGCOMM '15, pp.523-536, 2015.

R. ?-yacine-taleb, G. Stutsman, T. Antoniu, and . Cortes, Tailwind: Fast and Atomic RDMA-based Replication, Proceedings of the 2018 USENIX Annual Technical Conference, (USENIX ATC'18), 2018.

S. ?-yacine-taleb, G. Ibrahim, T. Antoniu, and . Cortes, Characterizing Performance and Energy-efficiency of The RAMCloud Storage System, Proceedings of the The 37th IEEE International Conference on Distributed Computing Systems (ICDCS'17), 2017.

, Workshops and Demos at International Conferences ? Yacine Taleb. Optimizing Fault-Tolerance in In-memory Storage Systems, the EuroSys 2018 Doctoral Workshop (EuroDW'18), 2018.

S. ?-yacine-taleb, G. Ibrahim, T. Antoniu, and . Cortes, An Empirical Evaluation of How The Network Impacts The Performance and Energy Efficiency in RAMCloud, Workshop on the Integration of Extreme Scale Computing and Big Data Management and Analytics in conjunction with IEEE, 2017.

?. Y. Taleb, S. Ibrahim, G. Antoniu, and T. Cortes, Understanding how the network impacts performance and energy-efficiency in the RAMCloud storage system, Big Data Analytics: Challenges and Opportunities, held in conjuction with ACM/IEEE SC'16, 2016.