Conditions and reasons
The condition types and reason strings the controller writes to status.conditions.
They are API contract: stable identifiers that kubectl wait, GitOps health checks and alerts can match on without reading messages.
Every condition is a metav1.Condition and carries observedGeneration, so a stale condition is detectable after a spec change.
status.phase summarizes conditions and worker progress for the printer column.
Its Pending value records the controller's first observation, before conditions exist; API-server creation does not initialize status.
Conditions stay authoritative for every outcome after that.
kubectl wait --for=condition=Complete migration/shop --timeout=1h
Phases
status.phase is the printer column: one word for what the migration is doing now.
After the first Pending observation, it reflects conditions and what the worker is doing.
Automation should wait on conditions, not on phase strings.
| Phase | The operator is | Next |
|---|---|---|
Pending |
Persisted its first observation, before validation or provisioning | Validating, Failed, or Suspended |
Validating |
Materializing the spec and running the preflight Job | Cloning, Completed (a passed dry run), or Failed |
Cloning |
Running the worker: schema, then table data | Finalizing, Streaming, Completed, or Failed |
Finalizing |
Past the data copy, finishing indexes, constraints and vacuum; on a clone with clone.ownerAfterRestore, also handing the restored objects over once the worker has exited |
Streaming, Verifying, Completed, or Failed |
Streaming |
Applying changes from the replication slot (live migrations) | CutoverPending, or Failed |
CutoverPending |
Caught up and waiting for approval (cutover.mode: Manual) |
CuttingOver |
CuttingOver |
Setting the end position, draining, proving the drain | Verifying, Completed, or Failed |
Verifying |
Running the requested pgcopydb compare checks |
Completed, or Failed |
Completed |
Finished, or a spec.dryRun preflight passed; terminal |
|
Failed |
Finished badly; terminal | |
Suspended |
Holding, because spec.suspend is true |
whatever it was doing |
Inside Cloning and Finalizing
pgcopydb does not run its steps in sequence. Once the schema is in place it starts the table copy, index builds, constraint creation, large-object copy and vacuum all at once. It then prints nothing until they finish, so the operator reports the copy and its tail as separate phases.
| Sub-state | Phase | What is running | How it looks |
|---|---|---|---|
| Catalog and schema | Cloning |
Source catalog queries, pg_dump, pre-data pg_restore |
Seconds. The target barely grows |
| Table data | Cloning |
tableJobs COPY workers, plus index, constraint and vacuum workers on whatever has already finished |
The bulk of the run. The target grows steadily |
| The tail | Finalizing |
Index builds, constraints, and vacuum on the tables that finished last | The target stops growing while work continues |
| Post-data | Finalizing |
Post-data pg_restore: foreign keys and the rest |
Seconds |
A table's vacuum cannot start until that table's own copy finishes, and the largest table finishes last.
So a clone routinely ends with one VACUUM ANALYZE still running while every other worker is idle.
The VACUUM tail has the measurement.
[!important] During
Finalizingthe target database stops growing, so anything derived from its size reads as finished while real work continues. This is whyPgcopydbMigrationCloneStalledmatchesCloningalone: a long tail is not a stall. Performance tuning explains how to trade the vacuum away for the time.
The phase comes from a pg_stat_activity query on the target that touches no pgcopydb catalog.
If a client reads that catalog while the copy writes it, the workers die.
The query counts copy workers and other workers separately.
Zero on both counts is the unknown answer, not the tail, so the phase stays where the last answered sample left it.
Condition types
Each type is named for what True means.
Eight are normal-true: True is the desired state.
Failed is abnormal-true: True means the migration ended in failure.
| Type | Polarity | True means |
|---|---|---|
Validated |
normal-true | The spec materializes cleanly and the preflight passed. |
CloneCompleted |
normal-true | The base copy finished. |
Streaming |
normal-true | Logical replication is applying changes (live migrations only). |
CaughtUp |
normal-true | Replication lag has been at or below spec.follow.maxCatchupLag on two consecutive samples. |
CutoverCompleted |
normal-true | The drain is proven: a clean data compare, or origin progress exactly at the cutover LSN on the rare cutover where the two coincide. With clone.ownerAfterRestore set, the handover has finished first. |
OwnershipApplied |
normal-true | The restored objects belong to clone.ownerAfterRestore. Only present when that field is set. |
Verified |
normal-true | The requested pgcopydb compare checks found source and target matching. |
Complete |
normal-true | The migration finished. Terminal and absorbing. |
Failed |
abnormal-true | The migration failed for good. Terminal and absorbing. |
Reasons
Every reason the controller sets, spelled exactly as it appears on the wire.
| Condition | Status | Reason | Appears when |
|---|---|---|---|
Validated |
True |
SpecValid |
The connections and clone options materialize cleanly and the preflight passed; refreshed on every reconcile of an active Migration. |
Validated |
Unknown |
PreflightRunning |
The preflight Job is running; when its pod cannot start, the message carries the kubelet reason verbatim (misnamed Secret, unbound PVC, unschedulable). |
Validated |
False |
InvalidSpec |
The spec cannot be rendered into a worker Job. The Migration fails terminally with the same reason. |
Validated |
False |
PreflightFailed |
Connectivity, selected extension availability or ownership, a target clone privilege, an all-databases superuser or database-listing probe, or a follow prerequisite failed. The message names the failed check and recovery action; a superuserSecretRef hint applies only to grant remediation. Terminal. |
CloneCompleted |
False |
CloneRunning |
A worker attempt is running the base copy. |
CloneCompleted |
False |
CopyingData |
The probe has seen this attempt's copy workers connected to the target; it replaces CloneRunning for the rest of the attempt, and the phase cannot reach Finalizing before it is set. |
CloneCompleted |
False |
CloneFailed |
The final attempt failed; the message carries the Job failure and the last pgcopydb error line. |
CloneCompleted |
False |
TablesEmptyOnTarget |
Live migration: the worker logged the base copy finished, but the pass's own sample found tables holding rows on the source and none on the target, which a --resume after killed attempts can produce. The message names them. The marker is not trusted, the stream is reported but not driven, and the reason stands until a sample finds a row in each of them. |
CloneCompleted |
False |
CloneIncomplete |
Clone-only migration: the worker exited 0, but pgcopydb's own catalog, read by a Job that mounts the same work dir once the worker is gone, counts tables not done. The Migration fails with the same reason. |
CloneCompleted |
True |
CloneSucceeded |
Clone-only migration: the worker Job finished, and pgcopydb's catalog, where it could be read, counted every table done. |
CloneCompleted |
True |
BaseCopyDone |
Live migration: the worker logged the base copy finished, the pass's own sample found no table holding rows on the source and none on the target, and change replay took over. |
Streaming |
True |
Replaying |
The worker's apply process is replaying changes to the target. |
CaughtUp |
True |
LagBelowThreshold |
Two consecutive samples measured the replication lag at or below spec.follow.maxCatchupLag. |
CaughtUp |
False |
Lagging |
Lag is above the threshold, or no replication sample is available yet. |
CaughtUp |
False |
ConfirmingCatchUp |
One sample measured the lag at or below spec.follow.maxCatchupLag, and the operator needs a second consecutive one before it sets CaughtUp True. A single sample can land while the worker still confirms its raw receive position, where the lag reads near zero whatever the apply backlog is. A sample above the threshold returns the condition to Lagging and the count starts over. |
CutoverCompleted |
True |
DrainVerified |
The verify Job proved the drain: the target's origin progress sits exactly on the cutover LSN, or pgcopydb compare data found every migrated table matching. Any other reading is decided by content, which is the path nearly every cutover takes, because publication-filtered WAL and unapplied commits measure alike. Changes are applied and sequences are synced. |
CutoverCompleted |
False |
DrainIncomplete |
Drain verification did not show the target holding every change: pgcopydb compare data found a difference or gave no verdict. The Migration fails with the same reason. |
OwnershipApplied |
Unknown |
OwnershipRunning |
The <name>-reown Job is running: after the worker exited on a clone, after the drain is proven on a live migration. No worker restarts while it exists. |
OwnershipApplied |
Unknown |
OwnershipSuspended |
spec.suspend deleted a running handover Job. It re-runs on resume and picks up the statements that are left; each ALTER commits on its own. |
OwnershipApplied |
True |
OwnershipApplied |
Every schema, relation, routine and type the migration role owned in the target database now belongs to clone.ownerAfterRestore. |
OwnershipApplied |
False |
OwnershipFailed |
The handover Job failed after its retries, and the data is on the target. The Migration fails with the same reason. |
Verified |
Unknown |
VerificationRunning |
A pgcopydb compare Job is running. |
Verified |
True |
ComparePassed |
Every requested compare found source and target matching. |
Verified |
False |
SchemaMismatch |
pgcopydb compare schema reported differences. |
Verified |
False |
DataMismatch |
pgcopydb compare data reported differences while the schema matched (or was not checked). |
Complete |
True |
MigrationSucceeded |
The migration finished; on live migrations, set after cleanup and verification. |
Complete |
True |
DryRunSucceeded |
A spec.dryRun Migration's preflight passed. No worker Job was created and no data moved. |
Failed |
True |
InvalidSpec |
Spec validation failed; retrying cannot help (source and target are immutable). |
Failed |
True |
PreflightFailed |
The preflight failed before any data moved. |
Failed |
True |
BackoffLimitExceeded |
The retry budget is exhausted (backoffLimit + 1 attempts). |
Failed |
True |
PermissionDenied |
An attempt hit a permission error retries cannot fix (best-effort log-tail classification; a miss keeps normal retries); the message carries the matched log line, and the remaining retry budget stays unspent. |
Failed |
True |
CloneIncomplete |
A clone-only worker exited 0 while pgcopydb's catalog counted tables not done. Do not use the target as a complete copy. |
Failed |
True |
DrainIncomplete |
Cutover drain verification refuted completeness. |
Failed |
True |
OwnershipFailed |
The ownership handover failed; the data is on the target. |
Two of those terminal reasons need a recovery step rather than a new Migration.
On DrainIncomplete, do not switch applications to the target; Phase Failed with reason DrainIncomplete covers what the verify Job logs say and how to recover.
The replication slot is kept, so the data stays recoverable.
On OwnershipFailed, finish the handover by hand with the statements in the Job log, following Ownership handover failures; on a live migration the slot is kept until the Migration is deleted.
The condition message carries the Job's last lines: the missing role, or the exact GRANT.
A mismatch on Verified does not fail the Migration, because the transfer itself finished.
Complete is set either way; see Verification.
Event reasons
Events record each transition.
Reasons are stable; messages are not.
Terminal failures also emit a Warning event whose reason equals the Failed condition reason above.
| Reason | Type | Appears when |
|---|---|---|
AttemptStarted |
Normal | A worker attempt's Job was created. |
AttemptFailed |
Warning | An attempt failed; the next one resumes from the work-dir catalogs. |
TablesEmptyOnTarget |
Warning | The worker logged the base copy finished while a sample showed tables holding rows on the source and none on the target; the message names them. Once per refusal, see the condition reason of the same name. |
WorkerZombie |
Warning | The pgcopydb supervisor died but a child process kept the worker pod alive (upstream pgcopydb 0.18 defect); the operator removed the pod so the normal retry could resume. |
PreflightStarted |
Normal | The preflight Job was created. |
PreflightPassed |
Normal | Every preflight check passed; the message counts checks and applied grants, or in a dry run the grants that would be applied. |
PreflightRemediated |
Normal | The preflight applied missing grants through superuserSecretRef; one event per tier (clone, follow), each message listing that tier's exact statements. |
PreflightWouldRemediate |
Normal | A dry run found grants superuserSecretRef would apply, and applied none; one event per tier, each message listing that tier's statements. |
DryRunSucceeded |
Normal | A dry run's preflight passed; the Migration completed without a worker Job. |
CutoverStarted |
Normal | The cutover LSN is set; the stream is frozen and draining. |
CutoverRetry |
Warning | Setting the cutover LSN failed transiently; retried on the next pass. |
CleanupStarted |
Normal | The cleanup Job (slot, publication, origin) was created. |
CleanupFailed |
Warning | Cleanup exhausted its retries; the named slot may still hold WAL on the source and needs manual removal. |
OwnershipStarted |
Normal | The <name>-reown handover Job was created. |
OwnershipApplied |
Normal | The handover finished; the restored objects belong to clone.ownerAfterRestore. |
Suspended |
Normal | spec.suspend deleted the worker, and a running handover Job with it; the work volume is kept. |
SlotRetained |
Warning | A suspended live migration keeps its replication slot, which holds WAL on the source. |
VerificationStarted |
Normal | A compare Job was created. |
Verified |
Normal | Compare found source and target matching. |
VerificationMismatch |
Warning | Compare reported differences; details are in the compare Job logs. |
Completed |
Normal | The migration finished. |