Getting Started

PodMotion migrates live Kubernetes pods between nodes within a single cluster using CRIU checkpoint/restore. This guide takes you from a fresh cluster to a verified migration driven entirely by the PodMigration custom resource and kubectl apply.

Prerequisites

Before you begin, confirm every node and the cluster meet the baselines from the project README:

• Kubernetes 1.27+ and a matching kubectl (Supported floor is 1.27+, per ADR-0104. Kubernetes 1.25 and 1.26 are not supported.)
• Go 1.24.6+ and Docker 17.03+ (only to build images)
• Linux kernel 5.15+ on every node (Ubuntu 22.04 HWE floor); the authoritative current end-to-end proof (Sprint 33/34) ran on kernel 6.17 (amd64). The earlier arm64/kind PoC ran on kernel 6.8.
• PostCopy mode additionally requires CONFIG_USERFAULTFD=y on the destination kernel
• CRIU 4.x installed on every node (4.2 used for iterative pre-copy)
• bpfman DaemonSet running on every node before the PodMotion agent starts (see bpfman below)

kindnet (host-routed, no VXLAN overlay) is the CNI of the current authoritative end-to-end proof (Sprint 33/34). The earlier arm64/kind Flannel VXLAN PoC is historical only — not the current proven substrate. TCP source-IP preservation (spec.sourcePodIP) is Calico/Cilium-only and is a roadmap item.

bpfman

bpfman is an eBPF program manager required by the PodMotion node agent to load TC egress programs. The node agent uses bpfman to attach its TC egress hook to each pod's network interface — without bpfman, the eBPF relay that enables TCP sequence translation cannot load.

bpfman must be running on every node as a DaemonSet before the PodMotion agent starts. Install it from the upstream project: bpfman.io — Installation.

Install

Install the CRDs and deploy the manager and node-agent DaemonSet:

# Install the 5 CRDs (podmigrations, migrationpolicies,
# nodemigrationcapacities, migrationcheckpoints, migrationwebhookconfigs)
make install

# Deploy the controller-manager
make deploy IMG=<your-registry>/podmotion:<tag>

git clone https://github.com/podmotion-io/podmotion.git
cd podmotion

# Build the operator + agent binaries (linux/amd64) and the kind node image
make build-linux
make kind-image KIND_NODE_IMG=podmotion-node:dev
# edit images/kind/kind-config.yaml to reference podmotion-node:dev, then:
make kind-create KIND_CLUSTER=podmotion-dev

# Build and load the operator + agent images
docker build -f images/operator/Dockerfile -t podmotion-operator:dev .
docker build -f images/agent/Dockerfile    -t podmotion-agent:dev .
make kind-load IMG_OPERATOR=podmotion-operator:dev IMG_AGENT=podmotion-agent:dev \
  KIND_CLUSTER=podmotion-dev

# CRDs must be applied separately (the Helm chart's crds/ directory is empty)
KUBECONFIG=/tmp/podmotion-dev.kubeconfig kubectl apply -f config/crd/bases/

# Install via Helm (the authoritative deploy path — there is no flat install manifest)
KUBECONFIG=/tmp/podmotion-dev.kubeconfig \
helm upgrade --install podmotion charts/podmotion/ \
  --namespace podmotion-system --create-namespace \
  --set operator.image.repository=podmotion-operator --set operator.image.tag=dev \
  --set operator.image.pullPolicy=Never \
  --set agent.image.repository=podmotion-agent --set agent.image.tag=dev \
  --set agent.image.pullPolicy=Never \
  --set operator.replicas=1 --set webhook.certManager.enabled=false \
  --set scheduler.enabled=false --set installCRDs=false \
  --set checkpoint.transferMode="" --set checkpoint.registry.url=""

helm install podmotion ./charts/podmotion -n podmotion-system --create-namespace

helm install podmotion ./charts/podmotion -n podmotion-system --create-namespace \
  --set webhook.certManager.enabled=false

A successful install gives you the controller-manager Deployment, the node-agent DaemonSet, the admission webhooks, and the five migration.podmotion.io CRDs.

Your First Migration

A minimal PodMigration requires podName and podNamespace:

apiVersion: migration.podmotion.io/v1alpha1
kind: PodMigration
metadata:
  name: migrate-my-app
  namespace: default
spec:
  podName: my-app-pod          # required
  podNamespace: default        # required
  targetNodeName: worker-2     # optional; scheduler picks if empty
  mode: PreCopy                # PreCopy (default) | PostCopy | Cold
  # tcpPreservationMode defaults to None (no TCP continuity).
  # Opt in with Strict or BestEffort — see below.

Apply it:

kubectl apply -f migrate-my-app.yaml

The CRD has shortName pm and category podmotion. List migrations with kubectl get podmigration or kubectl get pm. Note: podmotion is the category label — kubectl get podmotion resolves by category, not by resource name; use podmigration or pm for reliable listing.

TCP connection continuity is opt-in

To preserve TCP connections, set the mode explicitly:

The full TCP-first pipeline runs only for Strict and BestEffort.

Verify It Worked

Watch the migration status:

kubectl get podmigration migrate-my-app -o yaml

Track status.phase. The full MigrationPhase enum has 22 values — the list below is a partial selection of the most observable phases. See api/v1alpha1/podmigration_types.go for the complete enum.

Common phases: Pending, Validating, Checkpointing, PreCopyMemory, SocketInventory, ZeroWindowArm, OverlayHandoff, Transferring, Restoring, RestoreSocketReattach, DisengageHold, TCPVerifying, ServiceVerifying, CutoverComplete, Complete, DryRunComplete, Failed, RollingBack.

Inspect status.conditions. The canonical condition types are:

• TCPVerified — aggregate TCP gate (only populated for Strict/BestEffort)
• TCPSequenceContinuityVerified — the condition that reports seq_delta=0 (zero TCP sequence delta), True when Strict mode holds a connection byte-exact across a migration. First proven at Sprint 46 for a single held connection; the broader multi-flow guarantee is still ahead. (Separately, the BestEffort/kindnet end-to-end proof above verifies IP reachability after restore, not TCP-connection survival.)
• TrafficVerified — destination is serving traffic; this is the gate that authorizes source-pod deletion (ADR-0012). Source pod is never deleted before TrafficVerified=True.

See the CRD Reference for the authoritative condition list.

For a dry run that produces a DryRunEstimate without moving the pod, set spec.dryRun: true; the migration ends in phase DryRunComplete.

Security

PodMotion is a privileged operator. Read this before installing on any cluster you care about.

• Privileged node-agent DaemonSet (charts/podmotion/templates/daemonset.yaml): a node-root surface area required for live CRIU checkpoint-restore and eBPF relay. The main agent container itself runs privileged: false with drop: [ALL], adding back only SYS_PTRACE, SYS_ADMIN, CHECKPOINT_RESTORE, DAC_READ_SEARCH, SYS_RESOURCE, NET_ADMIN, NET_RAW, KILL (plus BPF/PERFMON on kernels ≥5.8), with allowPrivilegeEscalation: true and a read-write mount of the containerd Unix socket. The netns-broker sidecar is the one genuinely privileged: true, long-lived container; two short-lived init containers (install-apparmor-profile, init-checkpoint-tmpfs) also run privileged: true. A CRIU-scoped seccomp profile ships in the chart but is opt-in (disabled by default). The DaemonSet tolerates all taints (operator: Exists), so it runs on every node including control-plane nodes. ADR-0148 (Accepted, unimplemented) replaces the catch-all tolerations with a dedicated migration-node pool (nodeSelector podmotion.io/migration-node=true + matching NoSchedule taint/toleration) to bound the blast radius off control-plane nodes. It hostPath-mounts the host CRIU binary (/usr/sbin/criu, read-only). See the Node Agent Architecture page for the full privilege model. ADR-0148 (Accepted, unimplemented) defines the post-1.0 hardening roadmap: further capability reduction, read-only containerd socket, seccomp on by default, and user namespace isolation.
• Checkpoints are plaintext. Checkpoints are written to tmpfs at /run/podmotion/checkpoints/<ref>/ (mode 0700, wiped on node reboot). Checkpoint encryption is NOT implemented. ADR-0029 (which recorded encryption as on-by-default) was never wired into the CRIU writer and is Superseded by ADR-0160 (Draft, the authoritative forward plan: none/aes-gcm/aws-kms/gcp-kms/azure-kms/vault providers). See Security Architecture. CRIU checkpoints are full memory dumps and may contain secrets, tokens, and in-flight data. Protect the checkpoint store accordingly.
• Transport security is opt-in. The agent gRPC endpoint (:9090) defaults to plaintext when TLS certificates are not configured. Enable mTLS before deploying to any non-isolated environment — see the security warning above and Security Architecture for configuration steps.
• The controller-manager itself is hardened by contrast (runAsNonRoot, seccomp RuntimeDefault, read-only root filesystem, all capabilities dropped).

See SECURITY.md in the repository for the full security policy and disclosure process.