Spent a few days designing sidecar mutation files. Tombstones for deletes, append logs for inserts, compaction to merge them back. The whole write-ahead-log pattern that every database uses. Seemed right.
Then I flipped the whole thing. Consolidate into worker-aligned shards, mutate in place, keep the source files clean and immutable. No mutation log, no compaction debt, no garbage collection. Source files are the source of truth. Shards are derived and disposable.
The hard part is modified-file epoch updates -- telling the cluster that a shard changed without triggering full redistribution. Haven't built it yet. But the design feels right this time. The sidecar approach was adding a database inside a database and I don't want to be in the database business.