Durability with BookKeeper

Abstract

Practical systems must often guarantee that changes to the system state are durable. Examples of such systems are databases, file systems, and messaging middleware with guaranteed delivery. One common way of implementing durability while keeping performance high is to use a log to persist updates to the system state. Such systems use the log to reconstruct the system state in the event of a crash. When implementing such a log, if the log is only stored locally, the system state is permanently lost when the server writing the log experiences a permanent hardware failure. BookKeeper is a system that exposes a log abstraction for building high performance, highly available distributed systems. BookKeeper transparently implements replication for high availability and striping for high performance. A Book- Keeper deployment comprises storage servers called bookies, which are designed to serve a large number of concurrent ledgers. BookKeeper is currently an open-source project and is in production use at Yahoo!