Repository navigation
Conversation
Signed-off-by: somfornot <318146256+somfornot@users.noreply.github.com>
somfornot
force-pushed
the
small-object/bucket-engine
branch
from
October 21, 2026 06:05
d85e3c6 to
ad9d75c
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Small objects can waste disk space and index memory when each object is stored as an individually aligned entry. This adds an independent, CacheLib BigHash-inspired engine selected through
SmallEngineConfig.The engine packs records into fixed-size buckets, locates buckets directly from key hashes, and uses an 8-byte Bloom filter per bucket instead of a per-object address index. Buckets evict records in FIFO order. Worker shards serialize mutations, bound queued work, and invalidate affected buckets on overflow so dropped replacements or deletes cannot expose stale values.
The change includes checksummed bucket encoding, compression, recovery modes, cancellation-safe clearing, shutdown error reporting, metrics, public exports, documentation, and a runnable restart/delete example. BlockEngine remains independent; automatic size-based routing is outside this change.
Tradeoffs: mutations read and rewrite a whole bucket, pending mutations temporarily hide that bucket, and overflow can evict neighboring entries. Recovery scans the device and requires stable capacity, bucket size, and key hashing. This is a disposable cache without fsync/WAL durability; performance has not been benchmarked.
Validation:
foyer-storage,foyer, andfoyer-commonwithstrict_assertions,test_utils: 91 passed, 1 ignored.serde,tracing,strict_assertions,test_utils: 12 passed, including direct I/O, io_uring, recovery, overflow, corruption, concurrency, cancellation, and injected write failure.cargo run --example small: passed.cargo fmt --all --check: passed; Clippy reports the existing unknownchunks_exact_to_as_chunkslint warning.cargo x --fastsuite and performance benchmarks have not been run.This is a single-layer stack targeting
main, with no prerequisite PRs.