Affiliation:
1. Brandeis University, USA
2. Boston University, USA
3. TileDB, Inc., USA
Abstract
Data-intensive applications have fueled the evolution of
log-structured merge (LSM)
based key-value engines that employ the
out-of-place
paradigm to support high ingestion rates with low read/write interference. These benefits, however, come at the cost of
treating deletes as second-class citizens
. A delete operation inserts a
tombstone
that invalidates older instances of the deleted key. State-of-the-art LSM-engines do not provide guarantees as to how fast a tombstone will propagate to
persist the deletion
. Further, LSM-engines only support deletion on the sort key. To delete on another attribute (e.g., timestamp), the entire tree is read and re-written, leading to undesired latency spikes and increasing the overall operational cost of a database. Efficient and persistent deletion is key to support: (i) streaming systems operating on a window of data, (ii) privacy with latency guarantees on data deletion, and (iii)
en masse
cloud deployment of data systems.
Further, we document that LSM-based key-value engines perform suboptimally in the presence of deletes in a workload. Tombstone-driven logical deletes, by design, are unable to purge the deleted entries in a timely manner, and retaining the invalidated entries perpetually affects the overall performance of LSM-engines in terms of space amplification, write amplification, and read performance. Moreover, the potentially unbounded latency for persistent deletes brings in critical privacy concerns in light of the data privacy protection regulations, such as the
right to be forgotten
in EU’s GDPR, the
right to delete
in California’s CCPA and CPRA, and
deletion right
in Virginia’s VCDPA. Toward this, we introduce the delete design space for LSM-trees and highlight the performance implications of the different classes of delete operations.
To address these challenges, in this article, we build a new key-value storage engine,
Lethe
+
, that uses a very small amount of additional metadata, a set of new delete-aware compaction policies, and a new physical data layout that weaves the sort and the delete key order. We show that
Lethe
+
supports any user-defined threshold for the delete persistence latency offering
higher read throughput
(1.17× -1.4×) and
lower space amplification
(2.1× -9.8×), with a modest increase in write amplification (between 4% and 25%) that can be further amortized to less than 1%. In addition,
Lethe
+
supports efficient range deletes on a
secondary delete key
by dropping entire data pages without sacrificing read performance or employing a costly full tree merge.
Publisher
Association for Computing Machinery (ACM)
Reference93 articles.
1. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/EC;Official Journal of the European Union (Legislative Acts),2016
2. Assembly Bill No. 375 Chapter 55 2018 California Consumer Privacy Act
3. 2020. The California Privacy Rights Act of 2020. https://thecpra.org/. (2020).
4. 2021. Virginia Consumer Data Protection Act. https://www.sullcrom.com/files/upload/SC-Publication-Virginia-Second-State-Enact-Privacy-Legislation.pdf. (2021).
5. The dataflow model