In Search of an Understandable Consensus Algorithm
Diego Ongaro and John Ousterhout • USENIX ATC 2014
The Raft consensus algorithm - a more understandable alternative to Paxos
consensusdistributed-systemsraft
Paper Shelf
A curated collection of research papers I find fascinating and insightful. Each paper has shaped my understanding of various technical domains.
Inspired by Arpit Bhayani's paper collection
Distributed Systems
Papers about distributed systems, consensus algorithms, and scalable architectures
MapReduce: simplified data processing on large clusters
Jeffrey Dean and Sanjay Ghemawat • OSDI 2004
The Google File System
Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung • SOSP 2003
Foundational paper describing GFS - Google's distributed file system
distributed-systemsstoragegoogle
Dynamo: Amazon's Highly Available Key-value Store
Giuseppe DeCandia, et al. • SOSP 2007
Introduces the concept of eventual consistency and influenced many NoSQL databases
distributed-systemsdatabasesamazon
ZooKeeper: Wait-free coordination for Internet-scale systems
Patrick Hunt, Mahadev Konar, Flavio P. Junqueira, and Benjamin Reed • USENIX ATC 2010
Describes the coordination service used by many distributed systems
distributed-systemscoordination
Spanner: Google's Globally-Distributed Database
James C. Corbett, et al. • OSDI 2012
First globally-distributed database with external consistency guarantees
distributed-systemsdatabasesgoogle
CRAQ: Chain Replication with Apportioned Queries
Jeff Terrace and Michael J. Freedman • USENIX ATC 2009
Extends chain replication with apportioned queries for improved read throughput while maintaining strong consistency
distributed-systemsreplicationconsistency
Finding a needle in Haystack: Facebook's photo storage
Doug Beaver, Sanjeev Kumar, Harry C. Li, Jason Sobel, Peter Vajgel • OSDI 2010
Describes Facebook's efficient photo storage system that handles billions of uploads
distributed-systemsstoragefacebook
Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing
Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin, Scott Shenker, Ion Stoica • NSDI 2012
Introduces RDDs (Resilient Distributed Datasets) - the core abstraction in Apache Spark that enables efficient fault-tolerant distributed computation
distributed-systemsbig-dataapache-sparkfault-tolerance
Discretized Streams: Fault-Tolerant Streaming Computation at Scale
Matei Zaharia, Tathagata Das, Haoyuan Li, Timothy Hunter, Scott Shenker, Ion Stoica • SOSP 2013
Introduces Spark Streaming's micro-batch architecture for scalable stream processing
distributed-systemsstream-processingapache-sparkfault-tolerance
TensorFlow: A System for Large-Scale Machine Learning
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, et al. • OSDI 2016
Describes the architecture and implementation of TensorFlow, Google's distributed system for training and serving machine learning models
distributed-systemsmachine-learninggoogletensorflow
Ray: A Distributed Framework for Emerging AI Applications
Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, et al. • OSDI 2018
Introduces Ray, a distributed system designed for emerging AI applications requiring flexible distributed computation
distributed-systemsmachine-learningrayreinforcement-learning
A comprehensive study of Convergent and Commutative Replicated Data Types
Marc Shapiro, Nuno Preguiça, Carlos Baquero, Marek Zawirski • INRIA TR 2011
Introduces CRDTs (Conflict-free Replicated Data Types) that guarantee eventual consistency in distributed systems without coordination
distributed-systemscrdtconsistencyreplication
Serverless Computing
Papers about Serverless computing aka Function-As-S-Service(FAAS)
On-demand Container Loading in AWS Lambda
Marc Brooker, Mike Danilov, Chris Greenwood, and Phil Piwonka • USENIX ATC 2023
serverlessaws-lambda
Firecracker: Lightweight Virtualization for Serverless Applications
Alexandru Agache, Marc Brooker, Andreea Florescu, et al. • NSDI 2020
Describes AWS's innovative microVM technology that powers Lambda and Fargate, focusing on fast startup times and security isolation
serverlessvirtualizationawsmicrovm