Migrating data between servers without losing anything

Why this matters

Every server is eventually replaced, and its data has to move. A migration that loses a folder, strips permissions or silently skips locked files may not be noticed for weeks, by which time the old server may be gone. A migration that copies everything but makes every file readable by everyone is arguably worse, because it looks like success.

This lesson covers moving data between servers so that everything arrives, arrives intact and arrives with the same protections, and so there is always a way back until that has been proved.

The lesson

Planning a migration: inventory, cutover and a way back

A migration is planned before a single file is copied, starting with an inventory of everything that is moving and everything that depends on it: the data itself, the shares and their permissions, the applications and scripts that refer to paths on the old server, scheduled tasks, service accounts, and the names people and systems use to reach it.

Large migrations are usually done in two phases to keep downtime short:

1. Pre-seed: copy the bulk of the data while the old server is still in use. This can take days, and nobody needs to stop working.
2. Cutover: in a scheduled window, stop changes on the old server (for example by making shares read-only), run a final delta sync that copies only what changed since the pre-seed, then point users and systems at the new server, typically by changing DNS records or a namespace rather than asking everyone to use a new name.

The way back is the most important part of the plan. Keep the old server intact, and read-only, until the new one has been verified and has run in production for an agreed period. Never delete or wipe the source as part of the migration itself. If something is missing, the source is where it is.

Copy tools -- robocopy and rsync -- and what their flags preserve

Ordinary drag-and-drop copying is not suitable for migrations. It stops or skips on errors, often loses permissions, and cannot resume. Two tools are the standard.

Robocopy on Windows:

• /E copies subdirectories, including empty ones.
• /COPYALL copies data, attributes, timestamps, security permissions, owner and auditing information; /DCOPY:T preserves directory timestamps.
• /MIR mirrors the source to the destination, which means it also deletes anything at the destination that is not in the source. It is ideal for the delta sync and catastrophic if the source and destination are typed the wrong way round.
• /L lists what would happen without copying anything: a dry run.
• /R and /W set retries and the wait between them. The defaults are very high, so a single locked file can stall a copy for a long time; set sensible values explicitly.
• /MT copies with multiple threads, /Z makes copies restartable, and /LOG writes a log that is essential for verification.

Rsync on Linux:

• -a (archive) copies recursively and preserves permissions, timestamps, symbolic links, and owner and group (owner needs root).
• -A and -X add ACLs and extended attributes, and -H preserves hard links.
• --delete makes the destination a mirror, with the same danger as robocopy's /MIR.
• -n (--dry-run) shows what would happen first.
• A trailing slash on the source matters: dir/ copies the contents of dir, while dir copies the directory itself. Getting it wrong puts everything one level too deep.

Both tools copy only what has changed on later runs, which is what makes the pre-seed and delta sync approach fast.

Keeping permissions, ownership and timestamps intact

Files carry more than their contents: permissions saying who can access them, an owner, and timestamps recording when they were created and changed.

A copy that loses permissions is a security problem. Files copied without their access control lists inherit whatever permissions the destination folder has, which can suddenly expose confidential data to everyone, or lock out the people who need it. A copy that loses timestamps breaks retention policies, backups that rely on modification times, and users' ability to find their recent work.

Two further traps appear when moving between systems:

• On Windows, permissions are stored against security identifiers, not names. Within one domain they carry over; migrating between domains needs the identifiers mapped, or files end up owned by unknown accounts.
• On Linux, ownership is stored as numeric user and group IDs. If the same username has a different ID on the new server, files end up owned by the wrong account, or an unknown one.

After copying, check permissions on a sample of files and folders, especially sensitive ones, rather than assuming the flags worked.

Physical-to-virtual and virtual-to-virtual migrations

Sometimes the whole server moves, not just its data.

A physical-to-virtual (P2V) migration converts a physical server into a virtual machine, using tools such as VMware's converter or Microsoft's Disk2vhd. The conversion copies the entire system, but the virtual machine has different hardware, so storage and network drivers change, the network adapter gets a new MAC address, vendor hardware management agents become useless and should be removed, and the operating system may need reactivating.

A virtual-to-virtual (V2V) migration moves a virtual machine between hypervisors, such as from VMware to Hyper-V, converting its disk format along the way, or simply between hosts running the same hypervisor, which live migration can do without downtime.

Test a converted server in isolation before cutting over. A converted copy switched on while the original is still running has the same name and the same IP address, and the resulting conflicts can disrupt the working server.

Proving the copy is complete with counts and hashes before cutover

A migration is not finished until it has been verified.

• Compare counts and sizes. The number of files and folders, and the total size, should match between source and destination. Robocopy's log summary reports counts, copied files, skipped files and failures; skipped and failed entries need explaining, not ignoring.
• Compare hashes. Counts prove everything arrived, not that it arrived intact. File hashes, from Get-FileHash in PowerShell or sha256sum on Linux, prove contents are identical, at least for a representative sample and for the most important files. A second rsync pass with --checksum compares contents rather than sizes and dates.
• Test the application, not just the files: open documents, run the application against the new location, and check that permissions behave as expected.

Only when all three agree is it time to cut over, and even then the source stays in place, read-only, until the new server has proved itself in production.