Data engineering and migration

Moving data between systems without losing any of it

Every migration is a negotiation with data that is worse than anyone remembers. Duplicate records nobody reconciled, dates in three formats, a free-text field carrying meaning that was supposed to be a dropdown, and a key that was unique until 2019. The technical part of moving data is rarely the hard part. The hard part is deciding what the data means, and getting someone to own that decision before the cutover rather than during it.

What the work involves

  • Profiling the source data honestly before committing to a plan, including the parts nobody wants to look at
  • Schema design for the queries that will actually run, rather than a transliteration of the old structure
  • Migrations under version control and rehearsed against a copy of production, repeatedly
  • Reconciliation that proves row counts and totals match, rather than assuming
  • A rollback plan that has been tested, not merely written
  • Cutover sequencing designed around the business, not around the engineers' convenience

Why we are credible at this

We migrated our own production platform between distributed SQL engines with live data. We also know from experience that in a distributed database a transaction sees a single snapshot for its whole life even in read-committed mode, which quietly invalidates code that was correct on a single node.

What moves the price

We quote after discovery and the number is fixed before anything starts. These are the things that move it, so you can see roughly where your project sits before you talk to us.

  • How clean the source data is — this dominates everything else, usually by a wide margin
  • Whether the old and new systems must run in parallel, and for how long
  • Acceptable downtime at cutover, since near-zero downtime is a different and more expensive design
  • Whether historical data must be preserved exactly or can be summarised

Proof we have done this

A distributed data layer, chosen deliberatelyDistributed SQL, wide-column, cache, object storage and vectors — each for a reasonThe cluster we run our own company onA production Kubernetes platform, operated by the people who deploy to it

Questions we get asked

Other App Development work

Custom web applicationsThe system your business actually runs on, built properlyAPI and integration developmentMaking two systems agree, including the one with no documentationMobile applicationsCross-platform where it makes sense, native where it does notAI and retrieval systemsAssistants that read your documents, and know what they are not allowed to read
All App Development

Tell us what you are trying to do

A short conversation is usually enough to tell whether we are the right people. If we are not, we will say so and point you somewhere better.

Start a conversationI already use KamoCRMHow we quote