Five different data stores, each answering a question the others answer badly: distributed SQL for transactional business data, wide-column for high-write telemetry, an in-memory store for ephemeral state, object storage for files, and a vector database for semantic search. Picking correctly is most of data engineering, and it is nearly always done by inertia instead.
| Measure | Value | Source | Measured |
|---|---|---|---|
| Distinct data stores in production | 5 (YugabyteDB, ScyllaDB, Redis/Valkey, MinIO, Qdrant) | kubectl get statefulsets,deployments --all-namespaces | |
| Distributed SQL topology | yb-master and yb-tserver as separate stateful sets | kubectl get statefulsets -n kamo | |
| SQL migration files under version control | 90 | find . -name '*.sql' across all service repositories | |
| Production database engine migrations completed | 1 (CockroachDB to YugabyteDB, live data) | service configuration history; port 26257 retired in favour of 5433 |
What building it taught us
The default answer to where data goes is whatever the last project used. That is how a business ends up running analytical aggregations against the database taking its customer orders, or keeping thirty gigabytes of uploaded files in table rows. We chose each store against the workload, and we migrated when we got it wrong — the platform previously ran on a different distributed SQL engine and moved off it, which is a genuinely hard thing to do with production data and worth more as experience than any amount of architecture discussion.
Distributed SQL is not a drop-in replacement for a single-node database, and the ways it differs are the ways that bite. A transaction sees one snapshot for its whole life even in read-committed mode, so the second read of a row inside one transaction does not observe a write another transaction committed in between — code that was correct on a single node is quietly wrong here. Read-and-write conflicts surface as retryable errors on transactions that never wrote anything. These are not exotic edge cases; they are Tuesday, and they are why we are careful about what we promise on data work.
Object storage carries every uploaded file, with deduplication by content, and that has its own lesson: a failed upload can leave a row that a later retry deduplicates against, producing a file the system believes exists and cannot serve. We found that in production. It is now something we check for in any system we build that deduplicates anything.
Built with
- YugabyteDB
- ScyllaDB
- Redis
- Valkey
- MinIO
- Qdrant
- PostgreSQL wire protocol
- Hibernate
What this qualifies us for
Tell us what you are trying to do
A short conversation is usually enough to tell whether we are the right people. If we are not, we will say so and point you somewhere better.