Reimplement the data backend of the Storage component so it can become scalable horizontally #178

Activity

1. matthyx

   added this to Roadmapon Dec 10, 2024
2. matthyx

   moved this to Feature in Kubescapingon Jan 7, 2025
3. Oshratn

   added

   backlog

   core platform enhancements

   on May 13, 2025
4. yugal07 commented on May 5, 2026

   

   yugal07

   on May 5, 2026

   Contributor

   More actions

   Why the current backend can't scale out

   StorageImpl (pkg/registry/file/storage.go) holds three pieces of node-local state. All three break when running more than one replica:

   Piece	Where	Why it breaks N>1
   Metadata	sqlitemigration.Pool over a local SQLite file (sqlite.go)	File-locked, single-writer; replicas can't share
   Payloads	afero.Fs rooted at a local dir	Each replica has its own disk
   Watch fan-out	WatchDispatcher (watch.go), in-process channels	Replica A never sees replica B's writes — silently breaks every controller that watches these CRDs

   The sqlite+afero design exists because etcd couldn't handle the payload sizes (large CVE manifests, application profiles, network neighborhoods). Any replacement must preserve that property . "move back to etcd" is not an option.

   Proposed plan , two phases

   Phase 1: extract a Backend interface (refactor only, no behavior change)

   Introduce an interface that captures the three responsibilities above:

   type Backend interface {
       Metadata() MetadataStore  // CRUD on metadata rows
       Payloads() PayloadStore   // read/write opaque object payloads
       Watch()    WatchBus       // publish/subscribe across replicas
   }

   The existing sqlite + afero + in-process dispatcher becomes LocalBackend and remains the default. StorageImpl stops talking to *sqlitemigration.Pool and afero.Fs directly and goes through Backend.

   This phase is mergeable on its own, ships zero user-visible change, and is the prerequisite for any remote implementation. It's also the right place to land test coverage that pins current behavior before swapping internals.

   Phase 2: add a RemoteBackend for HA installs (separate PR, behind config)

   Selected via Helm values / config flag; LocalBackend stays the default for single-node installs. Candidate stacks for maintainers to weigh in on:

   • Postgres + S3-compatible object store + Postgres LISTEN/NOTIFY (or NATS) for watch fan-out. Most flexible, three concerns cleanly split, two new infra deps for the operator.
   • CockroachDB / FoundationDB for metadata + same object store for payloads. One distributed DB instead of two services, heavier single dep.
   • Postgres for both metadata and payloads (bytea / large objects) , fewer moving parts, but payload size pushes Postgres harder than it likes.

   My weak preference is option 1 (Postgres + S3 + NATS) , it matches how each subsystem is independently scaled in practice and lets payload storage be swapped for any S3-compatible service the user already runs.

   Few Clarifications before implementation

   1. Is a pluggable backend acceptable, or do you want a single replacement with no LocalBackend left behind?
   2. Which Phase 2 stack is acceptable as a hard dependency for HA Kubescape installs?
   3. Migration path from existing on-disk sqlite+afero data to a remote backend , required for v1, or a follow-up?
   4. Watch semantics under multi-replica: is at-least-once delivery acceptable, or do we need exactly-once?

   If this is open for contribution can I take on this? @matthyx
5. matthyx commented on May 6, 2026

   

   matthyx

   on May 6, 2026

   ContributorAuthor

   More actions

   @yugal07 I would not touch this... the whole reason we switched from etcd to SQLite + files was to reduce memory usage.

   If you start using Postgres, S3, or these distributed datastores, we could even just deploy a new etcd and rely on the existing implementation inside sample-apiserver.

   I think this will remain here until a hyperscaler asks specifically for a scalable storage because their clusters are too big for us...
6. yugal07 commented on May 6, 2026

   

   yugal07

   on May 6, 2026

   Contributor

   More actions

   Ahh I see. No issues, Thanks