- Feature Name: Virtual sstables - Status: in-progress - Start Date: 2022-10-27 - Authors: Arjun Nair - RFC PR: https://github.com/cockroachdb/pebble/pull/2116 - Pebble Issues: https://github.com/cockroachdb/pebble/issues/1683 ** Design Draft** # Summary The RFC outlines the design to enable virtualizing of physical sstables in Pebble. A virtual sstable has no associated physical data on disk, and is instead backed by an existing physical sstable. Each physical sstable may be shared by one, or more than one virtual sstable. Initially, the design will be used to lower the read-amp and the write-amp caused by certain ingestions. Sometimes, ingestions are unable to place incoming files, which have no data overlap with other files in the lsm, lower in the lsm because of file boundary overlap with files in the lsm. In this case, we are forced to place files higher in the lsm, sometimes in L0, which can cause higher read-amp and unnecessary write-amp as the file is moved lower down the lsm. See https://github.com/cockroachdb/cockroach/issues/80589 for the problem occurring in practice. Eventually, the design will also be used for the disaggregated storage masking use-case: https://github.com/cockroachdb/cockroach/pull/70419/files. This document describes the design of virtual sstables in Pebble with enough detail to aid the implementation and code review. # Design ### Ingestion When an sstable is ingested into Pebble, we try to place it in the lowest level without any data overlap, or any file boundary overlap. We can make use of virtual sstables in the cases where we're forced to place the ingested sstable at a higher level due to file boundary overlap, but no data overlap. ``` s2 ingest: [i-j-------n] s1 L6: [e---g-----------------p---r] a b c d e f g h i j k l m n o p q r s t u v w x y z ``` Consider the sstable s1 in L6 and the ingesting sstable s2. It is clear that the file boundaries of s1 and s2 overlap, but there is no data overlap as shown in the diagram. Currently, we will be forced to ingest the sstable s2 into a level higher than L6. With virtual sstables, we can split the existing sstable s1 into two sstables s3 and s4 as shown in the following diagram. ``` s3 s2 s4 L6: [e---g]-[i-j-------n]-[p---r] a b c d e f g h i j k l m n o p q r s t u v w x y z ``` The sstable s1 will be deleted from the lsm. If s1 was a physical sstable, then we will keep the file on disk as long as we need to so that it can back the virtual sstables. There are cases where the ingesting sstables have no data overlap with existing sstables, but we can't make use of virtual sstables. Consider: ``` s2 ingest: [f-----i-j-------n] s1 L6: [e---g-----------------p---r] a b c d e f g h i j k l m n o p q r s t u v w x y z ``` We cannot use virtual sstables in the above scenario for two reasons: 1. We don't have a quick method of detecting no data overlap. 2. We will be forced to split the sstable in L6 into more than two virtual sstables, but we want to avoid many small virtual sstables in the lsm. Note that in Cockroach, the easier-to-solve case happens very regularly when an sstable spans a range boundary (which pebble has no knowledge of), and we ingest a snapshot of a range in between the two already-present ranges. slide in between two existing sstables is more likely to happen. It occurs when we ingest a snapshot of a range in between two already present ranges. `ingestFindTargetLevel` changes: - The `ingestFindTargetLevel` function is used to determine the target level of the file which is being ingested. Currently, this function returns an `int` which is the target level for the ingesting file. Two additional return parameters, `[]manifest.NewFileEntry` and `*manifest.DeletedFileEntry`, will be added to the function. - If `ingestFindTargetLevel` decides to split an existing sstable into virtual sstables, then it will return new and deleted entries. Otherwise, it will only return the target level of the ingesting file. - Within the `ingestFindTargetLevel` function, the `overlapWithIterator` function is used to quickly detect data overlap. In the case with file boundary overlap, but no data overlap, in the lowest possible level, we will split the existing sstable into virtual sstables and generate the `NewFileEntry`s and the `DeletedFileEntry`. The `FilemetaData` section describes how the various fields in the `FilemetaData` will be computed for the newly created virtual sstables. - Note that we will not split physical sstables into virtual sstables in L0 for the use case described in this RFC. The benefit of doing so would be to reduce the number of L0 sublevels, but the cost would be additional implementation complexity(see the `FilemetaData` section). We also want to avoid too many virtual sstables in the lsm as they can lead to space amp(see `Compaction` section). However, in the future, for the disaggregated storage masking case, we would need to support ingestion and use of virtual sstables in L0. - Note that we may need an upper bound on the number of times an sstable is split into smaller virtual sstables. We can further reduce the risk of many small sstables: 1. For CockroachDB's snapshot ingestion, there is one large sst (up to 512MB) and many tiny ones. We can choose the apply this splitting logic only for the large sst. It is ok for the tiny ssts to be ingested into L0. 2. Split only if the ingested sst is at least half the size of the sst being split. So if we have a smaller ingested sst, we will pick a higher level to split at (where the ssts are smaller). The lifetime of virtual ssts at a higher level is smaller, so there is lower risk of littering the LSM with long-lived small virtual ssts. 3. For disaggregated storage implementation, we can avoid masking for tiny sstables being ingested and instead write a range delete like we currently do. Precise details on the masking use case are out of the scope of this RFC. `ingestApply` changes: - The new and deleted file entries returned by the `ingestFindTargetLevel` function will be added to the version edit in `ingestApply`. - We will appropriately update the `levelMetrics` based on the new information returned by `ingestFindTargetLevel`. ### `FilemetaData` changes Each virtual sstables will have a unique file metadata value associated with it. The metadata may be borrowed from the backing physical sstable, or it may be unique to the virtual sstable. This rfc lists out the fields in the `FileMetadata` struct with information on how each field will be populated. `Atomic.AllowedSeeks`: Field is used for read triggered compactions, and we can populate this field for each virtual sstable since virtual sstables can be picked for compactions. `Atomic.statsValid`: We can set this to true(`1`) when the virtual sstable is created. On virtual sstable creation we will estimate the table stats of the virtual sstable based on the table stats of the physical sstable. We can also set this to `0` and let the table stats job asynchronously compute the stats. `refs`: The will be turned into a pointer which will be shared by the virtual/physical sstables. See the deletion section of the RFC to learn how the `refs` count will be used. `FileNum`: We could give each virtual sstable its own file number or share the file number between all the virtual sstables. In the former case, the virtual sstables will be distinguished by the file number, and will have an additional metadata field to indicate the file number of the parent sstable. In the latter case, we can use a few of the most significant bits of the 64 bit file number to distinguish the virtual sstables. The benefit of using a single file number for each virtual sstable, is that we don't need to use additional space to store the file number of the backing physical sstable. It might make sense to give each virtual sstable its own file number. Virtual sstables are picked for compactions, and compactions and compaction picking expect a unique file number for each of the files which it is compacting. For example, read compactions will use the file number of the file to determine if a file picked for compaction has already been compacted, the version edit will expect a different file number for each virtual sstable, etc. There are direct references to the `FilemetaData.FileNum` throughout Pebble. For example, the file number is accessed when the the `DB.Checkpoint` function is called. This function iterates through the files in each level of the lsm, constructs the filepath using the file number, and reads the file from disk. In such cases, it is important to exclude virtual sstables. `Size`: We compute this using linear interpolation on the number of blocks in the parent sstable and the number of blocks in the newly created virtual sstable. `SmallestSeqNum/LargestSeqNum`: These fields depend on the parent sstable, but we would need to perform a scan of the physical sstable to compute these accurately for the virtual sstable upon creation. Instead, we could convert these fields into lower and upper bounds of the sequence numbers in a file. These fields are used for l0 sublevels, pebble tooling, delete compaction hints, and a lot of plumbing. We don't need to worry about the L0 sublevels use case because we won't have virtual sstables in L0 for the use case in this RFC. For the rest of the use cases we can use lower bound for the smallest seq number, and an upper bound for the largest seq number work. TODO(bananabrick): Add more detail for any delete compaction hint changes if necessary. `Smallest/Largest`: These, along with the smallest/largest ranges for the range and point keys can be computed upon virtual sstable creation. Precisely, these can be computed when we try and detect data overlap in the `overlapWithIterator` function during ingestion. `Stats`: `TableStats` will either be computed upon virtual sstable creation using linear interpolation on the block counts of the virtual/physical sstables or asynchronously using the file bounds of the virtual sstable. `PhysicalState`: We can add an additional struct with state associated with physical ssts which have been virtualized. ``` type PhysicalState struct { // Total refs across all virtual ssts * versions. That is, if the same virtual // sst is present in multiple versions, it may have multiple refs, if the // btree node is not the same. totalRefs int32 // Number of virtual ssts in the latest version that refer to this physical // SST. Will be 1 if there is only a physical sst, or there is only 1 virtual // sst referencing this physical sst. // INVARIANT: refsInLatestVersion <= totalRefs // refsInLatestVersion == 0 is a zombie sstable. refsInLatestVersion int32 fileSize uint64 // If sst is not virtualized and in latest version // virtualSizeSumInLatestVersion == fileSize. If // virtualSizeSumInLatestVersion > 0 and // virtualSizeSumInLatestVersion/fileSize is very small, the corresponding // virtual sst(s) should be candidates for compaction. These candidates can be // tracked via btree annotations. Incrementlly updated in // BulkVersionEdit.Apply, when updating refsInLatestVersion. virtualSizeSumInLatestVersion uint64 } ``` The `Deletion` section and the `Compactions` section describe why we need to store the `PhysicalState`. ### Deletion of physical and virtual sstables We want to ensure that the physical sstable is only deleted from disk when no version references it, and when there are no virtual sstables which are backed by the physical sstable. Since `FilemetaData.refs` is a pointer which is shared by the physical and virtual sstables, the physical sstable won't be deleted when it is removed from the latest version as the `FilemetaData.refs` will have been increased when the virtual sstable is added to a version. Therefore, we only need to ensure that the physical sstable is eventually deleted when there are no versions which reference it. Sstables are deleted from disk by the `DB.doDeleteObsoleteFiles` function which looks for files to delete in the the `DB.mu.versions.obsoleteTables` slice. So we need to ensure that any physical sstable which was virtualized is added to the obsolete tables list iff `FilemetaData.refs` is 0. Sstable are added to the obsolete file list when a `Version` is unrefed and when `DB.scanObsoleteFiles` is called when Pebble is opened. When a `Version` is unrefed, sstables referenced by it are only added to the obsolete table list if the `FilemetaData.refs` hits 0 for the sstable. With virtual sstables, we can have a case where the last version which directly references a physical sstable is unrefed, but the physical sstable is not added to the obsolete table list because its `FilemetaData.refs` count is not 0 because of indirect references through virtual sstables. Since the last Version which directly references the physical sstable is deleted, the physical sstable will never get added to the obsolete table list. Since virtual sstables keep track of their parent physical sstable, we can just add the physical sstable to the obsolete table list when the last virtual sstable which references it is deleted. `DB.scanObsoleteFiles` will delete any file which isn't referenced by the `VersionSet.versions` list. So, it's possible that a physical sstable associated with a virtual sstable will be deleted. This problem can be fixed by a small tweak in the `d.mu.versions.addLiveFileNums` to treat the parent sstable of a virtual sstable as a live file. Deleted files still referenced by older versions are considered zombie sstables. We can extend the definition of zombie sstables to be any sstable which is not directly, or indirectly through virtual sstables, referenced by the latest version. See the `PhysicalState` subsection of the `FilemetaData` section where we describe how the references in the latest version will be tracked. ### Reading from virtual sstables Since virtual sstables do not exist on disk, we will have to redirect reads to the physical sstable which backs the virtual sstable. All reads to the physical files go through the table cache which opens the file on disk and creates a `Reader` for the reads. The table cache currently creates a `FileNum` -> `Reader` mapping for the physical sstables. Most of the functions in table cache API take the file metadata of the file as a parameter. Examples include `newIters`, `newRangeKeyIter`, `withReader`, etc. Each of these functions then calls a subsequent function on the sstable `Reader`. In the `Reader` API, some functions only really need to be called on physical sstables, whereas some functions need to be called on both physical and virtual sstables. For example, the `Reader.EstimateDiskUsage` usage function, or the `Reader.Layout` function only need to be called on physical sstables, whereas, some function like, `Reader.NewIter`, and `Reader.NewCompactionIter` need to work with virtual sstables. We could either have an abstraction over the physical sstable `Reader` per virtual sstable, or update the `Reader` API to accept file bounds of the sstable. In the latter case, we would create one `Reader` on the physical sstable for all of the virtual sstables, and update the `Reader` API to accept the file bounds of the sstable. Changes required to share a `Reader` on the physical sstable among the virtual sstable: - If the file metadata of the virtual sstable is passed into the table cache, on a table cache miss, the table cache will load the Reader for the physical sstable. This step can be performed in the `tableCacheValue.load` function. On a table cache hit, the file number of the parent sstable will be used to fetch the appropriate sstable `Reader`. - The `Reader` api will be updated to support reads from virtual sstables. For example, the `NewCompactionIter` function will take additional `lower,upper []byte` parameters. Updates to iterators: - `Reader.NewIter` already has `lower,upper []byte` parameters so this requires no change. - Add `lower,upper` fields to the `Reader.NewCompactionIter`. The function initializes single level and two level iterators, and we can pass in the `lower,upper` values to those. TODO(bananabrick): Make sure that the value of `bytesIterated` in the compaction iterator is still accurate. - `Reader.NewRawRangeKeyIter/NewRawRangeDelIter`: We need to add `lower/upper` fields to the functions. Both iterators make use of a `fragmentBlockIter`. We could filter keys above the `fragmentBlockIter` or add filtering within the `fragmentBlockIter`. To add filtering within the `fragmentBlockIter` we will initialize it with two additional `lower/upper []byte` fields. - We would need to update the `SetBounds` logic for the sstable iterators to never set bounds for the iterators outside the virtual sstable bounds. This could lead to keys outside the virtual sstable bounds, but inside the physical sstable bounds, to be surfaced. TODO(bananabrick): Add a section about sstable properties, if necessary. ### Compactions Virtual sstables can be picked for compactions. If the `FilemetaData` and the iterator stack changes work, then compaction shouldn't require much, if any, additional work. Virtual sstables which are picked for compactions may cause space amplification. For example, if we have two virtual sstables `a` and `b` in L5, backed by a physical sstable `c`, and the sstable `a` is picked for a compaction. We will write some additional data into L6, but we won't delete sstable `c` because sstable `b` still refers to it. In the worst case, sstable `b` will never be picked for compaction and will never be compacted into and we'll have permanent space amplification. We should try prioritize compaction of sstable `b` to prevent such a scenario. See the `PhysicalState` subsection in the `FilemetaData` section to see how we'll store compaction picking metrics to reduce virtual sstable space-amp. ### `VersionEdit` decode/encode Any additional fields added to the `FilemetaData` need to be supported in the version edit `decode/encode` functions.