Why your lakehouse migration keeps slipping
Every stalled migration we have been called into was blamed on the technology. The Databricks setup was fine. Snowflake was fine. The pipelines mostly ran. What was actually stuck was a decision nobody wanted to make, and the engineering was just waiting behind it.
Migrations do not slip on code. They slip on unmade decisions. Here are the ones that do it, roughly in the order they tend to bite.
What "done" means
If done means the new platform runs, you will finish quickly. If it means every legacy report matches to the penny and the old system is switched off, that is a far larger project wearing the same name. Most teams never say which one they mean, so the finish line quietly keeps moving. Decide it on day one and write it down where everyone can see it.
Who owns the cutover
"The team" cannot flip the switch. One person has to be able to say go, accept the risk, and live with the result. When that person is not named, everyone waits for someone else to feel sure, and sure never quite arrives. Name them early, before the pressure is on.
How long you run both systems
Running the old and the new in parallel feels safe, so teams do it, and then they never stop. Two systems cost twice as much and mean twice the work every time something changes. Set the date you turn the old one off before you start, not in the anxious meeting three months later when nobody wants to be the one who says it is time.
The reconciliation bar
The new numbers will not match the old ones exactly, because the old ones were quietly wrong in places nobody documented. Someone has to decide how close is close enough, and that has to be agreed before you are staring at a variance report with a room full of nervous people. "Within a defined tolerance, signed off by finance" is a decision. "It has to match" is a wish.
A lakehouse migration is mostly an agreement problem wearing an engineering costume.
Scope creep, one source at a time
My favorite line, because it always sounds so reasonable: "while we're in there, can we add these three sources?" Each one resets the clock, and there is always one more. Freeze the scope for the cutover. The extra sources are a second project, and that is fine. Just do not let them hide inside the first one.
What we actually do
Before we touch a pipeline, I get these decisions on the table and written down. What done means. Who owns the cutover. When the old system goes dark. What tolerance counts as reconciled, and who signs it off. What is in scope and what waits. Once those are settled, the engineering is the straightforward part, and it stops slipping, because there is nothing left for it to wait on.
One migration we joined had been stuck for months on a single report that had to tie out exactly. The moment we agreed the tolerance and named the person who would sign off, it shipped in two weeks. Nothing technical had changed. The decision had just finally been made.
So if your migration keeps sliding a month at a time, do not go looking for a faster cluster. Look for the decision everyone is quietly avoiding. Sort the agreements, and the slipping stops.
Migration stuck and not sure why? Get in touch.