Architecture decisions, performance dives, and the gotchas behind shipping a distributed vector database.
8 days after the 48.8 QPS post: same collection, same harness, 1,620 QPS through the public API. Including the run where I benchmarked my own rate limiter.
Profiling said 93.7% of worker CPU was the scalar cosine distance. SIMD'd it. Zero QPS gain -- and vamana was quietly 64% faster than the index I was tuning.
Every vector-DB benchmark is one dot after a human hand-tuned an index. Ours refuses to sit still, so I measured the curve instead.
The first honest 1M benchmark came back at 48.8 QPS. Spent a day proving it wasn't a bug. It wasn't -- it was a mutex I wrote and forgot.
The first genuine 1M end-to-end run found two P1 bugs in 24 hours. Both invisible at 100k, by construction.
A PCA index reported great recall on its training sample and got promoted. The serving-recall probe demoted it in 36 seconds.
Our internal recall numbers were quietly self-flattering. Rebuilt benchmarking on VectorDBBench so a skeptic can reproduce them.
Wired the full format matrix into CI, made the suite 8x faster and un-hangable, and stopped accidentally load-testing Wikipedia.
Rebuilt billing: one system of record, the free $5 as a real invoice discount, and a credit model simple enough to audit by hand.
A CSV wider than the detection buffer, a ten-row upload that never became queryable, and metadata that wanted to be a list.
Spent a week building full collection export. Parquet out, no limits, no asterisks.
Shipped langchain-vectorpanda 0.1.0. The interesting part was everything it refuses to do -- and a default that was quietly lying about scores.
Quickstart used random vectors. New users saw noise-floor scores and assumed it was broken. Replaced with 5,000 Wikipedia plots and a bundled ONNX encoder.
express.json() caps bodies at 100KB out of the box. SDK inline upserts walked right into it.
Released the SDK, watched the first new-customer flow blow up on a TLS error. SAN didn't include api.vectorpanda.com.
Tagged 0.4.3, shipped a wheel that reported __version__='0.4.2'. Two sources of truth, one release script.
PCA queries returned a different top-k than brute-force cosine. The 32D projection was throwing away most of the signal.
Schema confirmation was a manual click for every upload. If there's exactly one valid interpretation, just take it.
Same string hashed in two processes returned two different u128s. Spent a confused afternoon staring at the wire protocol before noticing.
Inline upserts shouldn't pay the artifact-pipeline tax. WAL plus a per-worker overlay merged at query time.
MongoDB-style filters on query API, wired through the whole stack.
Threw out sidecar mutation files, went with in-place shard mutation.
Warm tier using mmap over NFS, faster than expected.
Real E2E tests on .150, one collection per format, exact top-k validation.
Added explicit status field instead of inferring readiness from internal state.
Delete broken in 4 places at once, had to implement two-phase deletion.
Duplicate keys under parallel JSONL load, two bugs that looked like one.
Generated keys colliding in multi-file collections.
Binary vector readers weren't rewinding between reads, garbage dimensions.
Was ignoring the explicit ID tensor in safetensors files.
Convention for CSV dimension columns, plus Feather zstd.
Added six vector formats in an afternoon, every one broke ID handling.
Binary metadata index, under 5ms for 500 lookups.
Returning metadata in query results, first two approaches too slow.
Consumer site is the only public entry point, everything else is internal.
First commit of the Python SDK, points at the gateway.
Self-service signup and API keys, no more manual account creation.
Free tier on warm storage, renamed cold to paused, 512D normalization for metering.
Threw out the old test harness. It was testing a system that doesn't exist anymore.
Standardized on systemd for Rust, PM2 for Node. One sync script per service.
SSE events, worker distribution panel, experiments UI. Real-time epoch promotion.
Wrote real docs, contributor guide, one sync script per service.
Artifact file signature got mangled. Silent failure. Took too long to find.
Node site decodes Rust bincode from the artifact server now.
Schema detection UI and system status page on the consumer site.
Gutted speculative code paths from the coordinator.
Workers that take too long loading files now get timed out by the coordinator.
Planted an auto_tier flag in the coordinator schema for later.
Set up PM2 for the consumer site immediately.
Put pricing on the site even though the model isn't final.
Collection detail page groups files by source, shows status, errors, artifacts.
Stood up the consumer site with Express, OAuth, and a dashboard that shows real numbers.
SQLite for auth and billing, plus discovery-based collections.
Coordinator rejects workers running stale builds.
Workers report RAM so the coordinator stops overloading them.
Mutex around query dispatch, intentionally single-threaded.
Detecting worker restarts with boot UUIDs.
Epoch-based data versioning for the cluster.
End-to-end query path working, no error handling.
Purged misleading names from the prototype era.
First coordinator commit, just WebSocket connections and liveness.
Renamed the project twice in 3 days.
Blocking fanotify calls inside tokio, classic mistake.
Replaced JSON with bincode for internal events.
Startup recovery scan for when the watcher restarts.
Got FAN_DELETE working, made deletes first-class.
Named the project, built the directory watcher.
First commit, no name, just a filesystem watcher in Rust.