Excloud legacy-VM live migration

How a running legacy VM's local root disk becomes a bootable network volume behind a brand-new VM — with the workload running the whole time except a sub-minute frozen window. Every claim on this page is pinned to a file and line so you can open the code and point at it.

Section 1TL;DR

An operator in the admin org runs exc compute migrate --source-vm-id N --new-name X. That single command drives everything:

1. computeapi creates a destination volume (source root size, rounded up to GiB) and a disposable receiver VM (t1a.micro) in the customer's own org and subnet, locked down by a temporary security group that admits only the source VM's IP on TCP 7443.
2. The CLI installs the pinned, SHA-256-verified excloud-migrate binary on the source VM over instance-connect exec. The agent streams the live root disk to the receiver in 4 MiB BLAKE3-hashed chunks over fingerprint-pinned TLS 1.3, then keeps reconciling drift in hash-compare passes.
3. On seal, the source freezes its filesystems (a detached watchdog process guarantees an emergency thaw within ~30 s if anything dies), runs one final verified pass that both sides hash into a 32-byte manifest, thaws, and uninstalls itself.
4. The receiver then sanitizes the cloned disk in a private mount namespace — new machine identity, no SSH host keys, no stale netplan — and reports done via its outbound heartbeat. Nothing ever connects into the receiver except the source.
5. Launch: computeapi detaches the destination volume, marks it root-eligible, and creates the replacement VM from it — same image record, mapped instance type (n1.* → m1a.*, t1.* → t1a.*), same security groups, deliberately new private IP + MAC and no public IPv4.

Section 2System map & repo responsibilities

Responsibilities per repo

The compatibility surface between computeapi and any given agent binary is deliberately tiny: the two imported Go packages (security for HMAC tokens + pinned TLS, control for the Status struct), the source→receiver /v1/* protocol, and the heartbeat contract. computeapi pins the exact binary via MIGRATION_AGENT_URL + MIGRATION_AGENT_SHA256, so an agent upgrade is always an explicit operator action (excloud-migrate/README.md).

Section 3The hot path — what actually runs, in order

This is the exact code path of a successful migration with default flags (--auto-cutover=true). Actor and file:line for every step.

if source.Manifest.RootVolumeId != 0 || source.Manifest.RootFSSize <= 0 {
    return migrationSource{}, exclouderrors.IntErrBadRequest.
        WithMessagef("source VM must use a legacy local root disk")
}

for {
    if size, e := device.Size(r.Disk.Path); e != nil || size != r.Disk.Size { // resize guard }
    status, e := r.Client.Status(ctx)
    if e != nil { time.Sleep(time.Second); continue }
    if status.Aborted { return r.Client.Thawed(ctx) }
    if status.Sealing { return r.seal(ctx, rep, pass) }
    if _, e = rep.Reconcile(ctx, pass); e != nil { return e }
    pass++
}

createReq := &types.ComputeCreatePostRequest{
    Name: r.NewName, OrgId: source.OrgId, ZoneId: int(source.ZoneId),
    SubnetId: source.SubnetId, ImageId: int32(source.Manifest.ImageId),
    InstanceType: r.TargetInstanceType, SSHPubKey: r.SSHPubKey,
    SecurityGroupIds: r.SourceSecurityGroupIDs,
    AllocatePublicIPv4: false,
    RootVolume: types.VolumeCreatePostRequest{ VolumeID: r.Clone.DestinationVolID },
}

The same thing as a sequence diagram

§3b — Seal & freeze, the riskiest 30 seconds

Freezing a production VM's filesystems is the one step that can hurt the customer, so it is wrapped in three independent safety layers:

1. A detached watchdog process. Before freezing, the source re-execs itself with EXCLOUD_MIGRATE_WATCHDOG=<config> and releases it. The main process touches a heartbeat file every timeout/3; the watchdog every second checks (a) that heartbeat file and (b) that the receiver is still reachable. If either is stale beyond 30 s it emergency-thaws every mount and re-enables swap, then notifies the receiver.
2. A deferred thaw in the seal function itself — any error path (or success) thaws, writes the freeze.done file that dismisses the watchdog, and notifies /v1/thawed.
3. Restart recovery. The systemd unit restarts a crashed agent; on startup it asks the receiver for status, and if the migration is already sealed/aborted it only emergency-thaws and reports — it will never re-freeze or write again (fix 415f0be).

func watchdogExpired(now, lastHeartbeat, lastReceiverContact time.Time, timeout time.Duration) bool {
    return now.Sub(lastHeartbeat) > timeout || now.Sub(lastReceiverContact) > timeout
}

The freeze itself: swapoff for swap on the same disk, syncfs on every mount, then FIFREEZE ioctl deepest-mount-first. Filesystems that don't support the ioctl (the vfat EFI partition) are tolerated — syncfs already flushed them — and kept out of the thaw set. internal/source/freeze_linux.go:27-66

The final pass is the integrity centerpiece. Under freeze, the source hashes every chunk in order and streams compare batches marked final. The receiver enforces contiguity from offset 0, feeds every (offset, length, hash) into a running BLAKE3 manifest, requires any repaired chunk to arrive with exactly the hash the compare promised (a frozen disk cannot legitimately change), and verifies each repair by reading it back. seal-complete then must present the same pass number and manifest the receiver computed itself:

if proof.Pass != r.verifiedPass || subtle.ConstantTimeCompare(proof.Manifest[:], r.verifiedManifest[:]) != 1 {
    return fmt.Errorf("seal proof does not match the latest verified final pass")
}
r.status.Sealed = true

Section 4State machine

There is no state column. State is derived on every read from the migration record + the latest heartbeat, in one small pure function — so there is exactly one place state logic can be wrong:

func migrationState(c repository.VMClone) string {
    if c.NewVMID > 0 { return "LAUNCHED" }
    if c.Sealed && c.AgentStatus.Sanitized && c.AgentStatus.DestinationClosed { return "READY" }
    if c.DesiredAction == "ABORT" { return "ABORTING" }
    if c.DesiredAction == "SEAL" { return "SEALING" }
    if c.Sealed { return "SEALING" }
    return "REPLICATING"
}

Section 5computeapi endpoint reference

All seven live under /compute/internal/migration/, registered in router/router.go:85-99, hidden from OpenAPI. Six require a bearer token whose org is the admin org (1576) via requireMigrationAdmin; the heartbeat is the exception — no platform auth (DisableSecurity), HMAC instead.

Section 6Receiver /v1 protocol (source → receiver, TLS 1.3 pinned)

Served by the receiver on :7443, reachable only from the source IP (security group) and only with the shared HMAC token (constant-time compared). Every non-status handler clears/sets last_error, which flows to the CLI via heartbeat → status. internal/receiver/receiver.go:118-147

Section 7Agent internals — excloud-migrate

One static Go binary (~3.4k lines incl. tests), CGO-disabled, distributed only as a compiled artifact from repo.excloud.in. Mode is os.Args[1]: source, receiver, or preflight; a fourth hidden personality — the watchdog — is selected by the EXCLOUD_MIGRATE_WATCHDOG env var so it survives even if argv parsing changes. cmd/excloud-migrate/main_linux.go:39-71

Source mode

Replication engine (internal/source/replicate.go): one sequential reader (disk-friendly), 4 concurrent HTTP uploaders for the initial pass. Reconcile passes are memory-bounded on purpose — hashes for 256 chunks (1 GiB of disk) are sent, and only mismatched chunks are re-read and retransmitted, re-hashed at send time because the live disk may have moved on:

// Do not retain every 4 MiB block while waiting for the mismatch
// bitmap. Re-read only mismatches and hash the retransmitted bytes
// again because the live source may have changed since the scan.
b := make([]byte, items[i].Length)
if e = readExactAt(r.Source, b, items[i].Offset); e != nil { … }
h := protocol.ChunkHeader{Offset: …, Hash: blake3.Sum256(b)}
if final && !bytes.Equal(h.Hash[:], items[i].Hash[:]) {
    return [32]byte{}, 0, fmt.Errorf("source chunk at offset %d changed while frozen", h.Offset)
}

Crash resume: on start the runner asks the receiver's status. Sealed/aborted → emergency-thaw + report only. Otherwise replicationResume skips the initial pass iff the receiver already counted one, and continues at completedPasses + 1. internal/source/source.go:32-54,81-86

Receiver mode

Startup discovers its own root disk, enumerates all mounts, and finds the destination as the exactly-one unmounted, partitionless disk of the expected size — then locks it exclusively (flock). The HTTP server (TLS 1.3, the prepare-minted cert) is wrapped so every handler is token-authed. All progress lives in an atomically-saved checkpoint, so a receiver reboot resumes exactly where it was, including re-triggering sanitation if it died between thaw and sanitize (internal/receiver/receiver.go:72-84), and re-closing the destination fd if it had already closed it (receiver.go:61-66). Throughput is deliberately reset per process — a persisted byte counter over fresh uptime would fake a spike (receiver.go:56-59).

Sanitation — making the clone a different machine

Two subtleties worth knowing cold: the whole routine runs on a locked OS thread because Linux mount namespaces are thread-local, and Go migrates goroutines between threads — the tainted thread is discarded when the goroutine exits (fix 4c35bb4, the newest agent commit). And netplan removal has a why-comment: the source's 50-cloud-init.yaml embeds its MAC/IP; the clone's first cloud-init run regenerates it from the clone's own metadata before network-online.target. internal/receiver/sanitize.go:21-24

Device layer — why O_DIRECT and exact sizes

Root disk discovery resolves the mount's major:minor through sysfs to the whole disk, and refuses anything layered — md, dm, nbd, rbd, loop, zram, or any device with slaves. Simple topologies only, by design. internal/device/discover_linux.go:88-131

O_DIRECT: reading a mounted filesystem's underlying block device through the page cache is not coherent with writes made through the filesystem. The final pass must see exactly the blocks that FIFREEZE + syncfs made durable, so all source reads bypass the cache, 4096-aligned via a manually aligned buffer. internal/device/direct_linux.go:16-31

Destination matching: excloud EBS NVMe namespaces present 3 MiB beyond the API size for storage metadata, so the receiver accepts a disk that is exactly the expected size or exactly expected+3 MiB — nothing else, so an unrelated bigger disk can never be clobbered. internal/device/destination_linux.go:14-26

Section 8Security model

Section 9Edge cases — the questions your boss will ask

Each is phrased as the question, with the mechanism and the code that answers it. Green = handled gracefully; red = fails safe with a documented recovery.

Section 10Abort — the rollback path

Section 11Recent fixes — what already went wrong once

The migration feature landed 2026-07-17 (c5f20a3 feat: migrate!) and hardened fast. These are the commits your boss may have seen fly by, decoded:

Section 12CLI — exc compute migrate

Added in e131837 (gen-cli-sdk). Registered under compute in cli.json; talks to computeapi with the operator's admin-org token from ~/.exc/config (org 1576) or EXCLOUD_ACCESS_TOKEN; base URL https://compute.excloud.dev unless EXCLOUD_COMPUTE_BASE_URL overrides. internal/pkg/compute/migrate/command.go

The consequential UX choice is the warning before consent: "writes made on the source after the final seal will not exist on the replacement." Cutover is a data-divergence point and the CLI makes the operator own it. migrate/command.go:232-243

Section 13Ops — config, releases, testing

computeapi environment

Fixed constants: staging image MIGRATION_STAGING_IMAGE_ID = 10, receiver instance type t1a.micro with a 16 GiB root, admin org 1576.

Agent release flow

Tag vX.Y.Z in excloud-migrate → Gitea Actions builds the static binary + SHA256SUMS → run ubuntu-repo deploy with that version → binary appears at repo.excloud.in/bin/excloud-migrate-X.Y.Z → update the two pins in computeapi and deploy. Agent, CLI, and computeapi releases are fully independent. README.md

End-to-end verification harness

scripts/manual-migration-source-workload turns any test VM into a migration guinea pig: it snapshots identity (machine-id, boot-id, hostname, network, block layout, SSH host key fingerprints), drops a 16 MiB random file with a recorded checksum, and starts a systemd writer appending a fsynced sequence log every second. After migrating, you diff the before-files on the clone: checksum must match, identity files must differ, and the sequence log shows exactly where the final seal cut the timeline. scripts/manual-migration-source-workload

Section 14Code health review — an honest read

Overall: this is a small, disciplined codebase — ~1.7k lines of orchestration, ~2.6k of agent (excl. tests), and the complexity lives where the risk lives (freeze safety, verification, sanitation), which is exactly the right KISS trade. The findings below are ranked; none are fires.

Section 15File map — find anything in 10 seconds

Generated 2026-07-18 from computeapi@8756171, excloud-migrate@4c35bb4, gen-cli-sdk@e131837 — every line reference checked against those commits. If the code moved, git log -S the symbol.