From: meskill (@meskill) <noreply+meskill@github.com>
To: pgdogdev/pgdog <noreply+pgdogdev-pgdog@github.com>
Subject: [pgdogdev/pgdog] PR #2058: feat(resharding): add initial setup for resumable resharding tasks
Date: Wed, 18 Nov 2026 13:08:42 +0000
Message-ID: <gh-pgdogdev-pgdog-2058@github.com> (raw)
Add initial setup to be able to resume tasks after cancellation or errors.
### What's implemented
- store the resharding run info in the `pgdog.resharding` table - the command, configuration, slot names. Everything that is required to resume the previous runs and to be able to validate configuration changes.
- use advisory locks on the resharding table to validate that only single instance is executing the resharding
- add new option `cleanup_on_cancel` on COPY_DATA/REPLICATION commands that control if we remove the slots/origins after task is stopped. By default it's true, we may change it in the future and/or remove the option.
- add new option `abort` to the same commands to be able to run the cleanup of the previously stopped task.
- add `SHOW RESHARDING` command to show table of currently running reshardings we can fetch from config
### Flow
- run command `COPY_DATA source destination publication WITH (cleanup_on_cancel false)`
- the task could be stopped with `STOP_TASK task_id`
- the state would be preserved, but there is no resume yet. It should be visible through `SHOW RESHARDING`
- the cleanup could be done with the mirror command `COPY_DATA source destination publication WITH (abort)`
### Next steps
Implement actual resume with the resume support for copy tables and replication.
inline patch 1: .github/workflows/ci.yml, Cargo.lock, docs/RESHARDING.md and 49 more (+5593/-220 lines) download inline-1.patchdiff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 454e6241c..648d183b9 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -130,6 +130,16 @@ jobs:
script: integration/resharding/cli.sh,
needs_docker: true,
}
+ - {+ name: resharding-resumable,+ script: integration/resharding/resumable.sh,+ needs_docker: true,+ }+ - {+ name: resharding-cutover,+ script: integration/resharding/cutover.sh,+ needs_docker: true,+ }
- {
name: server,
script: integration/server/run.sh,
diff --git a/Cargo.lock b/Cargo.lockindex f337ec7cf..b29847524 100644--- a/Cargo.lock+++ b/Cargo.lock@@ -3435,6 +3435,7 @@ dependencies = [
"serde",
"serde_json",
"serde_with",
+ "uuid",
]
[[package]]
diff --git a/docs/RESHARDING.md b/docs/RESHARDING.mdindex f0f9a97d9..cbc4b3009 100644--- a/docs/RESHARDING.md+++ b/docs/RESHARDING.md@@ -342,12 +342,24 @@ returned by any task surfaces via `table?` and aborts the manager's loop. Remain
completion or abort via their own `AbortSignal`, but their results are ignored once the channel
is dropped.
-On a failed or aborted migration, `ReshardTask::run` ([`api/resharding.rs`](../pgdog/src/api/resharding.rs))-obtains a guard via `Orchestrator::publication_guard()` and calls `PublicationGuard::cleanup()`, which-locks the publisher and has `Publisher::cleanup()` drop the permanent WAL slot via-`DROP_REPLICATION_SLOT "name" WAIT`. On success the slot is kept so reverse replication can roll-back. If the process crashes before this runs, the slot survives and keeps accumulating WAL on the-source — drop it manually before retrying.+On a failed or stopped migration, `ReshardTask::run` ([`api/resharding.rs`](../pgdog/src/api/resharding.rs))+calls `ReshardingState::cancel_run()`. With `cleanup_on_cancel true`, it drops the permanent slots that+the run created via `DROP_REPLICATION_SLOT "name" WAIT` and all origins of the run, then deletes the+row. Slots that existed before the run stay. With `cleanup_on_cancel false`, the run pauses and keeps+them. At cutover the forward slots are dropped and reverse slots are created, so reverse replication+can roll back. Each resharding needs its own slot name: two runs with the same explicit slot name on+one source share the slots.++Each run also writes a row to `pgdog.resharding` on destination shard 0 when it starts, with stage+`initializing`. While the stage is `initializing`, a stop or an error drops the slots that the run+created and deletes the row, also with `cleanup_on_cancel false`.+The cutover deletes the row and releases the lock, so the reverse phase is not resumable: a stop or+an error during reverse replication drops the reverse slots and origins. If the process crashes+before the cutover, the slot survives and keeps accumulating WAL on the source, and the row stays. A new run with the+same source, destination and publication then fails with `ReshardingExists`. Run the same command+`WITH (abort)`, or `data-sync --abort`, to drop the slots and origins of the run and its row, then+retry. A run with `cleanup_on_cancel false` keeps its slots, origins and row on failure too. See+[RESUMABLE_RESHARDING.md](./RESUMABLE_RESHARDING.md).
### Temporary replication slots
diff --git a/docs/RESUMABLE_RESHARDING.md b/docs/RESUMABLE_RESHARDING.mdnew file mode 100644index 000000000..d67c355b8--- /dev/null+++ b/docs/RESUMABLE_RESHARDING.md@@ -0,0 +1,670 @@+# Resumable resharding — design++This document describes the design for stoppable and resumable resharding. Stage 1 is implemented. The later stages are planned. For the current pipeline see [RESHARDING.md](./RESHARDING.md).++Part 1 gives the plan. Part 2 gives the design that all stages share. Part 3 gives one section per implementation stage. Each stage describes the approach and the result, not the code changes.++"State" in this document means the data that pgdog saves for resume.++|Stage|Status|+|---|---|+|1. State foundation and pause|Implemented.|+|2. Replication resume|Not implemented. Parts of the resume logic exist in the shared run state, but no command starts a resume. `WITH (resume)` and `data-sync --resume` do not exist yet.|+|3. Copy and table sync resume|Planned.|+|4. Schema phase resume|Planned.|+|5. Cutover recovery|Planned.|+|6. Operations|Planned.|++# Part 1. Plan++## Goal++Today a run lives only in pgdog memory. A stop or an error drops the slots and origins. A crash loses all progress.++The goal: a run that can stop, resume and survive a crash, in every phase.++## Selected inputs++|Topic|Selected input|+|---|---|+|State store|The Postgres table `pgdog.resharding` on destination shard 0 (SM1). The control plane Raft (SM2) was rejected. See "Storage mode".|+|Stored data|Only facts that Postgres does not store. Slots, origins and replication progress come from the catalogs.|+|Stop mode|The option `cleanup_on_cancel`, default `true`, keeps today's behavior. With `cleanup_on_cancel false`, the run is resumable. A resumable run never drops its slots on stop, error or crash. `WITH (abort)` cleans up.|+|Resume trigger|Manual command only. There is no startup scan. `SHOW RESHARDING` reports unfinished runs, and a new resharding logs the other runs on its destination.|+|Copy unit|The table part: one source publication table from one source shard. Each part commits with 2PC on all destination shards, so a part is all or nothing. pgdog never truncates destination tables.|+|Copy mark|A marker row on each destination shard, written inside the copy transaction (shard marker). See Stage 3.|+|2PC owner|The resumable copy runs `PREPARE TRANSACTION` and `COMMIT PREPARED` itself. The existing 2PC `Manager` and its WAL are not used for resumable runs. The resume rules are the only recovery.|++**Note: 2PC is the only reliable way to copy a part. Without 2PC, a failure between shard commits leaves the part on some shards only, and a destination crash shortly after a commit can lose the part with no error. With 2PC, a part is all or nothing and durable on every shard.**++## Stages++Each stage is usable alone. A phase that is not resumable yet refuses resume. The operator can abort.++```mermaid+flowchart LR+ S1[Stage 1: State foundation and pause] --> S2[Stage 2: Replication resume]+ S2 --> S3[Stage 3: Copy and table sync resume]+ S3 --> S4[Stage 4: Schema phase resume]+ S2 --> S5[Stage 5: Cutover recovery]+ S4 --> S6[Stage 6: Operations]+ S5 --> S6+```++|Phase|S1|S2|S3|S4|S5|+|---|---|---|---|---|---|+|Pre-data schema|no|no|no|yes|yes|+|Slot creation|no|yes|yes|yes|yes|+|Copy|no|no|yes|yes|yes|+|Post-data schema|no|no|no|yes|yes|+|Table sync|no|no|yes|yes|yes|+|Forward replication|no|REPLICATE only|yes|yes|yes|+|Cutover|no|no|no|no|yes|+|Reverse replication|no|no|no|no|no|++Integration tests are new scripts in the resharding integration suite. pgbench writes during each scenario.++# Part 2. Shared design++## Pipeline today++Phase order: pre-data schema, slot creation, copy, post-data schema, table sync, replication, cutover schema, validation, cutover, reverse replication. Stage flags (skip schema sync, replicate only, sync only, auto cutover) remove phases.++Facts that drive the design:++- The slot name is a user value or a random name. The random name is not stored.+- One permanent slot per source shard exists before the first table copy. It holds WAL for every later table snapshot.+- One origin exists per (slot, destination shard). Its name comes from the slot name and the shard. Today origins are dropped and created again, which loses apply progress.+- Each table part is copied from its own snapshot. Its copy LSN is the first skip watermark for replication.+- Sharded tables replay with a plain insert. Without origins and watermarks, a replay gives duplicate key errors.+- Every stop or error runs a cleanup that drops slots and origins.+- The cutover drops the forward slots before the config swap. The swap is in memory only.+- Each part copy runs in one transaction per destination shard. A failure before the commit rolls back every shard. Without 2PC, a failure between shard commits leaves the part on some shards only.+- Copy connections use `synchronous_commit = off`. A destination crash shortly after a copy commit can lose the copy with no error.+- When the cluster enables 2PC, the copy commits through the 2PC `Manager`. The `Manager` keeps its own WAL and runs its own recovery at startup. Its transaction names carry the instance id, not the run.++## Stored data++Rule: store a fact only when Postgres does not store it and resume needs it.++|Stored|Why resume needs it|Stage|+|---|---|---|+|Source and destination names, publication|Run key. One unfinished run per key.|1|+|Command with its options, format version|Rebuild the run. Refuse a newer format.|1|+|Slot name|Find the slots and origins. A generated name is lost on crash.|1|+|Stage: `initializing`, `copy_tables` or `replication`|Tells which slots and origins the row describes, and whether the copy is finished.|1|+|Config snapshot: shard counts and sharding rules of both clusters|Detect a config change before resume|1|+|Copy progress per table part: copy LSN and completion|Skip complete parts. Resolve prepared transactions. Give watermarks to table sync and replication.|3|+|Cutover step: `none`, `started`, `swap_done`|Recover a crash inside cutover.|5|++|Not stored|Source of truth|+|---|---|+|Slots, origins, replication progress|`pg_replication_slots`, `pg_replication_origin`, confirmed flush, origin progress|+|Synced LSN per table|Not needed. After table sync, slots and origins cover every table.|+|Run status|Row exists: unfinished. Lock held: running. Lock free: paused. Completed and aborted runs delete the row. A cutover also deletes the row.|+|Owner and heartbeat|The advisory lock. All writes use its connection. The lock does not fence a lost owner, see Ownership.|+|Endpoints|Stage 5 finds the swap state from the location of the state table.|+|Publication list, shard counts|Resume compares them with the copy progress and the slot and origin counts.|+|Validation progress|Validation can run again safely.|+|Attempts, row counts, errors|Logs|++## Storage mode++Selected: SM1, a table in the `pgdog` schema on destination shard 0. It works in open source, enterprise and the CLI.++### Rejected: control plane Raft (SM2)++SM2 keeps the state in the Raft state machine of the enterprise control plane. A variant (SM3) uses SM1 in open source and SM2 in enterprise behind one interface.++SM2 needs Raft enabled in the control plane. Raft is off by default, and without it nothing is durable. Control plane features are stubs in open source, like `pgdog.nextval`. So SM2 makes resumable resharding enterprise only.++|Point|SM1|SM2|+|---|---|---|+|Editions|Open source, enterprise, CLI|Enterprise with Raft enabled|+|Writes to user databases|One state table on destination shard 0|None for the state. Shard markers stay on the destination.|+|Ownership|Advisory lock. Freed when the owner session ends. After a host or network failure, TCP keepalive ends the session in about 2 minutes.|Lease. Takeover after the lease expires.|+|Availability|Depends only on the destination|Also depends on the control plane quorum|+|Swap detection|Location of the state table|Needs stored endpoints|+|Copy commit with a state mark|One local row write between prepare and commit|One Raft commit between prepare and commit. Blocks while the control plane is down.|+|Copy commit with a shard marker|No state write|No state write|++Both modes rely on 2PC for copy durability. Neither mode detects a part lost by a destination failover, unless Stage 3 selects the shard marker.++Why SM2 was rejected:++- Effort. The control plane Raft stores only sequences and maintenance mode today, with unconditional writes. SM2 needs new events, new Raft tables, a storage format upgrade, compare-and-set, a lease, fenced writes, a new control plane API and a pgdog client that renews the lease. SM1 needs one table and an advisory lock.+- Upgrade order. All control plane nodes must upgrade before pgdog uses the new state.+- Maintenance. Resharding and replication are open source. SM1 is still needed there, so SM2 means two backends with different ownership rules, two test sets, and changes in two codebases that must stay consistent.+- Availability. A control plane outage or a lost quorum pauses every resumable run, also when the databases are healthy.+- Takeover. After an owner crash, takeover waits one lease period. SM1 detects the crash at once.+- Fencing. The lease protects only the Raft state. A stale owner can still change Postgres until it stops itself.+- Maturity. Raft in the control plane is marked "Experimental".+- Copy commit. With a state mark, each part commit waits for a Raft write between prepare and commit. A control plane outage then leaves prepared transactions that hold locks on the destination.+- Config. The control plane does not store the pgdog config today. Config persistence after cutover (P4) is also new work.++Where SM2 would be better, and why it matters little:++|SM2 benefit|Why it matters little|+|---|---|+|No state table in user databases: no DDL, no dump exclusion.|pgdog already installs a `pgdog` schema on the shards. Shard markers stay on the destination in both modes.|+|A record of the run survives the loss of the destination.|No restore is possible. The copied data, origins and reverse slots are on the destination. The run must start again in both modes.|+|Maintenance mode already persists in Raft. After a crash, resume knows the real traffic state.|This works today without SM2. Resume can read it with no new storage.|+|One shared view of the run for all pgdog instances and the UI.|pgdog already pushes task updates to the control plane.|+|A lease survives short network breaks. The advisory lock ends on a short break.|The run pauses and can resume. No data is at risk.|++SM2 can be revisited if resumable resharding becomes an enterprise-only feature, or if the control plane gets a general state API for other features.++## State table (SM1)++```mermaid+erDiagram+ RESHARDING ||--o{ PART_MARKER : "one per destination shard"+ RESHARDING {+ uuid id PK+ text source_db UK+ text destination_db UK+ text publication UK+ jsonb command+ text slot_name+ text stage+ text cutover_step "Stage 5"+ jsonb config+ text instance_id+ uuid run_id+ bigint task_id+ int format_version+ timestamptz started_at+ timestamptz resumed_at "null until a resume"+ }+ PART_MARKER {+ uuid resharding_id PK+ text table_schema PK+ text table_name PK+ int source_shard PK+ text copy_lsn+ }+```++Stage 3 records which table parts are copied in table `PART_MARKER`, one on each destination shard. Each shard inserts its marker inside its own copy transaction, so the marker commits or rolls back with the copied rows. It has no FK, because it lives on other databases than `RESHARDING`. It refers to the run by `resharding_id`. A marker from an earlier run with the same key has another id, so a new run never reads it as its own.++The primary key of `RESHARDING` is `id`, a UUID v7 that pgdog generates when it creates the row. A resume keeps the id. The key (source, destination, publication) is `UNIQUE`: lookup and the advisory lock use it, and it allows one unfinished run per key. Stage updates, resume and delete find the row by `id` only. A stale owner thus cannot change the row of a newer run with the same key. The owner columns (`instance_id`, `run_id`, `task_id`) name the root task of the attempt that wrote the row.++`RESHARDING` is the table `pgdog.resharding`. It has one row per unfinished resharding, on destination shard 0. The row describes the resharding and every object it keeps.++Stage values: `initializing`, `copy_tables`, `replication`. The row is inserted when the run starts, with `initializing`. The copy path writes `copy_tables` after all forward slots exist. Replication writes `replication` after all forward slots and origins exist. While the stage is `initializing`, a stop or an error drops the slots that the run created and all its origins, and deletes the row, also with `cleanup_on_cancel false`. Cutover deletes the row, so the reverse phase has no row and is never resumed. A resume refuses a row in `initializing`, so a resume never creates a slot: a missing slot is an error.++All slots use the stored name: one forward slot per source shard. Reverse slots use the same name, but no row describes them. Each slot has one origin on each shard of the other side. Each table part is its own redo unit.++The state store holds only `RESHARDING`.++The state store creates its tables at run start with idempotent DDL. The existing `pgdog` schema setup does not run for every resharding run, so it is not enough.++### Format changes++Two layers change separately: the table shape and the meaning of a row.++Table shape. The DDL file runs at every state store open, under a transaction advisory lock. Changes are appended to it as idempotent statements. The destination role needs CREATE on the database, also when the schema and the table exist.++- Add columns only, and always with a default. Older pgdog versions name their columns, so they ignore new columns, and their inserts still work.+- Never drop, rename or retype a column. Stop using it instead.+- The primary key never changes. A key change needs a new table and a format version bump.++Row meaning. Each row stores the format version of the pgdog that wrote it.++|Change|Version bump|Rule|+|---|---|---|+|New option in the command or a new field in the config snapshot JSON|No|The field has a serde default. Older versions ignore a new command option. A new config field makes an older version report the config as changed, see the config snapshot rules.|+|New column with a default|No|See the table shape rules.|+|New stage value or cutover step, or a field with a new meaning|Yes|An older pgdog refuses the row. It does not resume or abort a resharding it cannot read.|+|A newer pgdog reads an older row|No|The row parser fills the defaults of the newer format.|++`SHOW RESHARDING` and the check at resharding start read the key columns first. A row that this version cannot read still shows its key, with the reason. It does not hide the other rows.++## Ownership++The owner holds a session advisory lock on the run key (source, destination, publication). It uses a dedicated state connection. Postgres frees the lock when the owner session ends. After a host or network failure, TCP keepalive on the state session ends it in about 2 minutes.++All state writes use this connection. The watchdog runs a heartbeat every 5 s. It stops the run at once when the query fails, and after 30 s with no reply. A slow reply under 30 s only logs a warning. The lock gives no fencing: in that time the old owner can still apply changes, while another instance can take the free lock, for example to abort. A lease survives short network breaks, but it needs owner and expiry columns.++## Resumable run state++No new layer wraps the tasks. The resumable state lives in the shared run state that every resharding task already receives. Each task reads its own part of it.++```mermaid+flowchart LR+ OP[Operator] --> CMD[Start command with cleanup_on_cancel false or WITH resume]+ CMD --> V[Resume validator]+ CMD --> RS[Shared resharding state: clusters, slots, origins, state store, lock]+ RS --- ST[(State table)]+ RS --> PH[Phase tasks]+ PH --> S[(Source: slots)]+ PH --> D[(Destination: tables, origins, reverse slots)]+```++|Job|Where it happens|+|---|---|+|Take the lock, check the key, insert the row|The first step of the reshard or replication task takes the lock and refuses a key with a row. Then it inserts the row with stage `initializing`. The start command replies with the task id first, so a duplicate key fails the task with `ReshardingExists`, visible in `SHOW TASKS`. The task puts the resumable state into the shared run state.|+|Hold the lock and the state connection|The shared run state. The lock lives until the run ends or pauses, not until the last copy of the state is dropped.|+|Skip finished phases|The resharding pipeline reads the stored stage. A row at `copy_tables` cannot resume until copy progress exists (Stage 3).|+|Skip done parts, adopt slots, keep origins|Each phase task reads its part of the resumable state.|+|Stop, error or lock loss → pause|The existing cleanup paths check the resumable state (Stage 1).|+|Validation|The first step of the task with `WITH (resume)`, before any phase runs. The validator also checks the config snapshot. One error per failed check.|+|"Resumed" flag in the UI|The task definition carries the flag. Subtasks show it too.|++Standalone COPY_DATA and REPLICATE runs get the same behavior, because they receive the same shared run state.++```mermaid+stateDiagram-v2+ [*] --> running: start with cleanup_on_cancel false+ running --> paused: STOP_TASK, error, crash, lock lost+ paused --> running: start command WITH (resume)+ running --> [*]: done or WITH (abort), row deleted+ paused --> [*]: WITH (abort), row deleted+```++```mermaid+flowchart TD+ A[Start or resume] --> B{Row exists?}+ B -- no --> B1[Fresh run: row insert, pre-data, slots] --> C+ B -- yes --> G{Stage}+ C[copy_tables: copy, post-data, table sync] --> G1+ G -- initializing --> XI[Refused: abort only]+ G -- copy_tables --> X[Refused until Stage 3]+ G -- replication --> G1[S2: forward replication] --> H[Cutover in one go]+ H --> R[Reverse replication, not recorded]+```++For each side effect, the state records what resume needs before the effect, or the effect proves its own progress. Resume completes or undoes each half-done effect.++## Commands++|Command|Behavior|Stage|+|---|---|---|+|RESHARD, COPY_DATA, REPLICATE `WITH (cleanup_on_cancel false)`. CLI `data-sync --cleanup-on-cancel false`.|Starts a resumable run. The reply does not change. The task fails if the key (source, destination, publication) has an unfinished resharding. Each other publication on the same pair has its own resharding.|1|+|STOP_TASK|Pauses. Slots and origins stay. Traffic resumes if it was stopped.|1|+|RESHARD, COPY_DATA, REPLICATE `WITH (abort)`. CLI `data-sync --abort`.|The key is (source, destination, publication). The command kind must match the stored row: RESHARD and COPY_DATA are one kind, REPLICATE is another. The options of the command are not compared. The slot must match when it is given. The config snapshot is not checked. Abort only cleans up a paused or crashed run before the cutover. After the cutover, the run has no row, and abort fails. If the run holds the lock, abort refuses with an error and the run continues. Stop the run first with `STOP_TASK` on the instance that owns it. The abort runs as a registered task, `abort <command> <source> -> <destination>` in `SHOW TASKS`. It drops every slot with the run's name, all its origins and the row. Destination data stays. Admin returns the task id. The CLI waits for the task.|1|+|`SHOW TASKS`|Existing command. No change.|1|+|`SHOW RESHARDING`|New command. It lists every unfinished resharding from the state tables. Columns: `source`, `destination`, `command`, `publication`, `stage`, `state`, `instance`, `task_id`, `replication_slot`, `started_at`, `resumed_at`. Stage 6 adds slot WAL status and retained bytes.|1|+|RESHARD, COPY_DATA, REPLICATE `WITH (resume)`. CLI `data-sync --resume`.|Not implemented yet. Continues the unfinished resharding with the same key (source, destination, publication). The command and its arguments must match the stored row, or it refuses. It refuses on a critical config mismatch. Takes the lock, validates, reconciles, continues. No new command.|2|+|CUTOVER|No change.|5|++Abort and resume search shard 0 of the destination. A cutover deletes the row, so no row is on the source side after the config swap. Stage 5 adds the search on the source side.++## Known limits++- Sharding changes during a pause are detected through the config snapshot. DDL changes are not detected.+- Runs with the default `cleanup_on_cancel true` cannot resume. After a crash, their row stays, and abort drops their slots and origins.+- Each resharding needs its own slot name. Two reshardings with the same explicit slot name on one source share the slots: one run can consume changes the other needs, and abort or cleanup of one drops the slots of the other. Their origins also share names on a destination server, so they share apply progress. Generated slot names never collide.+- `data-sync --abort` always aborts a RESHARD row. Abort a REPLICATE run with the admin command `REPLICATE ... WITH (abort)`.+- Abort builds the slot and origin names from the current shard counts. If the shard count changed during a pause, drop the remaining slots and origins by hand.+- A hard crash inside cutover, after the row delete, can leave slots with no row. Drop them by hand.+- A failed forward slot or origin drop in cutover does not stop the cutover. The remaining forward slots keep source WAL until the operator drops them by hand.+- The reverse phase after a cutover is not resumable. A stop or an error in the reverse phase drops the reverse slots and origins, also with `cleanup_on_cancel false`. To replicate in the reverse direction again, start REPLICATE.++# Part 3. Stages++## Stage 1. State foundation and pause++Goal: a resumable run never leaks slots or origins. The operator can see and abort it. No resume yet.++In this stage, "resumable run" means a run with `cleanup_on_cancel false`. "Cancel" covers every way a run stops before it ends: `STOP_TASK`, an error, a shutdown, a crash and a lost state connection.++### Approach++1. Parse `cleanup_on_cancel`. The default is `true`. The flag decides only what happens on stop or error. Every run is recorded.+2. Open a dedicated state connection to destination shard 0. Create `pgdog.resharding` with idempotent DDL. Take the session advisory lock on the key (source, destination, publication). Fail if the key has a row.+3. Probe the slots before the run creates them. If all slots exist, the run reuses them, and a live run keeps them on stop. If only some slots exist, refuse.+4. Capture the config snapshot: the shard counts and sharding rules of both clusters.+5. Insert the row right after the key check: command with its options, publication, slot name, stage `initializing`, config snapshot, owner task (instance id, run id, task id), format version, start time. While the stage is `initializing`, a stop or error always drops the slots that the run created and all its origins, deletes the row and releases the lock, also with `cleanup_on_cancel false`. After a crash in `initializing`, the row stays, and abort drops every slot with the run's name.+6. Write `copy_tables` when the copy path has created all forward slots. Write `replication` when replication has all forward slots and origins. A write never moves the stored stage back.+7. Every 5 s the owner runs a query on its state connection (heartbeat, lock watchdog). If the query fails, or gives no reply in 30 s, the connection and the lock count as lost, and the run stops.+8. On stop, error, shutdown or lock loss, with `cleanup_on_cancel false`: stop tasks, keep slots and origins, release the lock. With `true`: drop the slots that the run created and all its origins, then delete the row and release the lock. If the state connection is broken, the delete opens a new connection and takes the lock again. If the drop fails, or another instance holds the lock, the row stays, so abort can finish the cleanup. After a crash, Postgres frees the lock and the row stays.+9. Cutover ends the run, with no resume point inside it: drop the forward slots and origins, delete the row and release the lock, then create the reverse slots. A failed forward drop does not stop the cutover. pgdog logs a warning, and the operator drops the remaining forward slots and origins by hand. A crash inside cutover is not resumable. The reverse phase has no row and no lock, so abort and resume after the cutover fail with `NoResharding`. A stop or an error during reverse replication drops the reverse slots and origins, also with `cleanup_on_cancel false`.+10. At the end of a run with no cutover, delete the row.++Before the cutover, every automatic cleanup path skips runs with `cleanup_on_cancel false`. For them, only abort and the normal run end drop objects.++Abort runs as the registered task `AbortReshardingTask`. Admin and CLI use the same task. `SHOW TASKS` lists it, and `STOP_TASK` can stop it. The command finds the row by its key (source, destination, publication). Before the task starts, the command checks the row: the command kind, the slot when it is given, and the lock. A mismatch, or a lock that a running run holds, is an error at once. Abort never stops a run, on this instance or on another one. The task takes the lock, so a run that starts again between the check and the task makes the task fail with the same error. Then the task drops every slot with the run's name, all its origins and the row. This includes slots that existed before the run: after abort, nothing can be continued. Each phase skips missing objects, so a failed abort can run again. Abort does not check the config snapshot.++The config snapshot holds the shard counts and the `sharded_tables`, `sharded_mappings`, `omnisharded_tables` and `sharded_schemas` entries of both databases, as the serde JSON of the config structs. The compare reads the stored entries with the current config structs, and a field that a stored entry lacks reads as its default. So an upgrade that adds a config field does not report a change. Config fields are never removed or renamed between versions. A downgrade is different: `sharded_tables` and `sharded_mappings` refuse unknown fields, so an older pgdog reports a newer field as a change and refuses the resume. This is intended, because the older pgdog cannot apply that field. Resume is supported only between close pgdog versions. Every field is critical. Shard hosts are not stored: a failover or a DNS change keeps the same databases. The stored slots and origins prove that the databases are the same. Stage 1 stores the snapshot only. Stage 2 compares it with the live config and refuses a resume on a mismatch.++`SHOW RESHARDING` reads the state table on shard 0 of each configured database at query time. A held lock in `pg_locks` means `running`, no lock means `paused`. The scan never takes the lock. `instance` is the instance that started or last resumed the run. `task_id` is its root task while the run is running, so `STOP_TASK` on that instance stops it. It is null while the run is paused, because task ids are valid only in the process that created them. When a resharding starts, it checks its own key (an existing row refuses a fresh start; a resume checks the command, slot and config). It then reads the rest of its destination's state table: a running row is logged as in progress, a paused row gets a warning as an unfinished resharding.++Exclude the state table from the schema dump. A later run can use this destination as its source.++### Interface changes++- RESHARD, COPY_DATA and REPLICATE accept the option `cleanup_on_cancel <bool>`, default `true`. `WITH (cleanup_on_cancel false)` starts a resumable run. A bare `cleanup_on_cancel` means `true`. Accepted values: `true`, `false`, `on`, `off`.+- RESHARD and REPLICATE now accept `WITH (...)` options, as COPY_DATA does. The replies do not change.+- RESHARD, COPY_DATA and REPLICATE accept `WITH (abort)`. It aborts the unfinished resharding with the key (source, destination, publication) of the command, before the cutover. The reply is one column `task_id`, the abort task. `abort` together with `cleanup_on_cancel` is refused. COPY_DATA refuses `abort` with `tables_only`.+- CLI `data-sync --cleanup-on-cancel <true|false>` (default `true`) and `data-sync --abort`. `--abort` conflicts with `--cleanup-on-cancel`, `--replicate-only`, `--tables-only`, `--sync-only` and `--skip-schema-sync`.+- `SHOW TASKS` does not change for reshard and replication tasks. The abort task shows as `abort <command> <source> -> <destination>`.+- New `SHOW RESHARDING`. Columns: `source`, `destination`, `command`, `publication`, `stage`, `state`, `instance`, `task_id`, `replication_slot`, `started_at`, `resumed_at`. `task_id` is null while the run is paused. `resumed_at` is null until the first resume. A row that this version cannot read shows its key and `unsupported: <error>` in `state`. A state table that cannot be read shows `error: <error>` in `state`, the database names that share it in `destination`, and null in the other columns.+- No startup scan. `SHOW RESHARDING` lists each unfinished resharding. A new resharding logs each other unfinished resharding on its destination.+- Control plane: a new field `cleanup_on_cancel` on the reshard and replication definitions. A missing field reads as `true`. A new definition kind `abort_resharding`. The abort task reports the generic task status. All changes are additive.+- New table `pgdog.resharding` on destination shard 0. `pg_dump` in schema sync excludes it.+- The state connection uses `application_name` `pgdog_resharding:<instance_id>`. It holds the advisory lock `(hashtext('pgdog.resharding'), hashtext('<source>/<destination>/<publication>'))`. The first half is the namespace of all resharding locks.+- Before the cutover, STOP_TASK on a resumable run pauses it and keeps its slots and origins. STOP_TASK on a run with `cleanup_on_cancel true` drops its slots and origins as today, then deletes its row. After the cutover, STOP_TASK drops the reverse slots and origins for every run.+- Every run, also with the default `cleanup_on_cancel true`, writes a row and takes the lock. A second run with the same key is refused while the first one runs or has a row.++### Tests++|#|Scenario|Expected result|+|---|---|---|+|1|COPY_DATA `WITH (cleanup_on_cancel false)`. STOP_TASK during replication.|Paused in `SHOW RESHARDING`, also after a restart. Forward slots and origins still exist.|+|2|`WITH (abort)` after 1.|Row gone. All slots with the run's name and all its origins gone, also slots that existed before the run.|+|3|REPLICATE with the default `cleanup_on_cancel true`. STOP_TASK.|A row while it runs. On stop, today's cleanup, then the row is deleted.|++Unit tests on a real Postgres: idempotent DDL, lock conflict with the owner name, write failure after the lock connection is lost, writes with an old row id, writes after a failed relock. Unit tests for the config validator. An integration test proves that the cutover deletes the row and releases the lock, that abort after the cutover fails, and that a stop during reverse replication drops the reverse slots and origins.++Result: a stopped run shows paused with its slots, and abort drops every slot and origin of the run.++## Stage 2. Replication resume++Goal: a resumable REPLICATE run survives stop and crash. It continues from slot confirmed flush with no duplicate apply.++### Approach++What exists when a start command with `WITH (resume)` runs:++|Source|Data|+|---|---|+|State table|Command with its options, publication, slot name, stage, config snapshot, format version|+|Source catalog, per source shard|The slot, its confirmed flush position and its WAL status|+|Destination catalog, per destination shard|The origin and its progress: the last source commit applied on that shard|+|Config|The source and destination clusters and their shard counts|++Nothing from the old process survives: no task tree, no in-memory state, no open streams.++How resume rebuilds the run:++1. Find the run row and take the lock. If another instance holds the lock, refuse. Write the new owner and `resumed_at`.+2. Validate: known format, config snapshot with no critical mismatch, matching shard counts, every slot present with its WAL.+3. Build the shared run state from the stored data: same publication, slot name and command options. A resumed run is always resumable: a stop or an error keeps its slots, origins and row, whatever `cleanup_on_cancel` is. Only abort and the normal run end drop them.+4. Adopt the slots by name. The row exists only when all slots exist, so a missing slot is an error and is never created. Create missing origins and never reset the progress of an existing origin.+5. Load the publication tables. Set each table's watermark from its stored copy LSN (Stage 3). A REPLICATE run has no copy, so its watermarks start at zero.+6. Start a new task with the "resumed" flag, at the stored stage.++How replication continues without duplicates:++- Each stream starts from its slot. Postgres starts at the slot's confirmed flush position. pgdog moves that position only after every destination shard has made the change durable, so no change after it is lost.+- Postgres can send again the changes between the confirmed flush position and the last change applied on a shard. Each destination shard skips every transaction at or below its origin progress.+- Each shard updates its origin in the same transaction as the applied changes. Origin progress and data therefore always agree on each shard.+- Changes for a table at or below its copy LSN are skipped, because the copy already has them.++A resume refuses a row in `initializing`, so a resume never needs to create a slot.++### Interface changes++- RESHARD, COPY_DATA and REPLICATE accept `WITH (resume)`. No new command. The command, the publication and the slot name must match the stored row, or the command refuses. `resume` together with `cleanup_on_cancel` is refused.+- CLI `data-sync --resume`.+- Origins of resumable runs keep their progress across restarts.+- A resumed task shows a "resumed" flag in the UI. Its subtasks show it too.++### Rules++- Slot: must exist. Refuse if it is missing or its WAL is lost.+- Origin: create if missing. Never reset the progress of an existing origin.+- Validator: the lock is free, the format version is known, the config snapshot has no critical mismatch, shard counts match, all slots exist. On failure, only abort is possible.++A RESHARD run needs copy LSNs as watermarks. Stage 3 stores them. Until then, only REPLICATE runs resume.++### Tests++|#|Scenario|Expected result|+|---|---|---|+|4|SIGKILL during REPLICATE replication. Restart. Resume.|Paused before resume. Origin progress does not go back. No duplicate keys. Data check passes.|+|5|A second instance resumes an owned run.|Refused with the lock holder.|++Unit tests: creating an existing origin keeps its progress. The validator refuses a missing slot and lost WAL.++Result: forward replication resumes after stop or crash with no duplicates.++### Open questions++- Synced LSNs: we assume they are not needed after table sync. If a test shows duplicates, store a synced LSN per part.++#### Task state across attempts++The exact behavior is not decided yet.++Facts:++- The pgdog task registry is in memory. Task ids start again after a restart.+- The control plane keeps task updates in memory, per token, instance and process run.+- After a crash, the old task sends no final update. The UI can show it as running.+- A resumed run starts a new task tree with new ids and new progress counters.+- A run can be resumed by another pgdog instance, or by an instance with another token. The control plane does not show tasks across tokens.++Needs:++- Link all attempts of one run. The link must not depend on the instance, the token or the task id. The run `id` in the state table is this link: every attempt of the run keeps it.+- Show that a task is resumed, and which attempt it continues.+- End the old attempt in the UI (for example "interrupted") when the run is resumed or aborted.+- Show done parts as done, not as zero progress.+- Decide whether progress counters are restored. Restored counters need more stored data.++## Stage 3. Copy and table sync resume++Goal: COPY_DATA and RESHARD survive a stop or crash during copy and table sync. Complete parts are not copied again. pgdog never truncates destination tables.++### Interface changes++- No new commands.+- The copy always uses 2PC on a multi-shard destination, independent of the client 2PC setting. The destination needs `max_prepared_transactions > 0`.+- The resumable copy runs `PREPARE TRANSACTION` and `COMMIT PREPARED` directly. It does not register with the 2PC `Manager`. The transaction name is `__pgdog_rsh_<run>_<part>_<attempt>_<shard>`. `Manager` recovery must not touch this prefix, and the resume rules must not touch `Manager` names.+- A single-shard destination commits each copy with synchronous commit on.+- Copy progress is stored in a marker table on each destination shard (shard marker).+- A start command with `WITH (resume)` accepts runs stopped in copy, table sync and replication, also RESHARD runs.+- `WITH (abort)` also rolls back the prepared transactions of the run.++### Marking a part complete++To know what is done, resume needs these facts for each part (source publication table, source shard):++|Data|Where|Written when|+|---|---|---|+|Part key: source table schema, source table name, source shard|Shard marker row|With the commit decision|+|Copy LSN|Same row|Same time. It is the watermark for table sync and replication, and it identifies the attempt.|+|Commit decision|A committed marker on any shard|After all shards prepare, before the first commit|+|Prepared transactions|`pg_prepared_xacts` on each destination shard. The name holds the run, the part and the attempt.|By Postgres, at `PREPARE TRANSACTION`|++Nothing else is stored per part. Row counts and attempts are in the logs only.++Resume reads these facts and gives each part one status:++|Status|Facts|Action|+|---|---|---|+|Done|Commit decision exists, no prepared transaction left|Skip. Use its copy LSN.|+|Committing|Commit decision exists, prepared transactions left|`COMMIT PREPARED` on the remaining shards. Then done.|+|Not done|No commit decision. Prepared transactions may exist.|`ROLLBACK PREPARED` on all shards. Copy again.|+|Lost|Markers on some shards, none on others, no prepared transaction|A failover lost data. Refuse.|++A destination table with rows and no done part for any source table that maps to it has rows from outside the run. Refuse. The check runs per source relation, as the copy does today.++A part is one (source publication table, source shard). Child partitions of a partitioned table are separate parts, each with its own copy LSN, also when they map to one destination parent. Its rows come only from that source shard, so each part is its own redo unit. "Complete" must never be true while a shard lacks the data.++Each part is one transaction per destination shard. All shards get the copy, also with no rows. The commit uses 2PC:++1. `PREPARE TRANSACTION` on all destination shards. The transaction name identifies the run, the part and the attempt.+2. `COMMIT PREPARED` on all shards.++Postgres flushes `PREPARE TRANSACTION` and `COMMIT PREPARED` to disk, also with `synchronous_commit = off`. So a committed part is complete and durable on every shard.++Each shard inserts a marker (source table, source shard, copy LSN) in its copy transaction. A committed marker on any shard proves the commit decision, because `COMMIT PREPARED` starts only after all shards prepare. No state write happens between prepare and commit, so the commit depends on the destination only.++|Failure|Destination|Action|+|---|---|---|+|F1: crash before all shards prepare|No shard committed|Roll back prepared transactions. Copy again.|+|F2: crash after all shards prepare, before the first commit|No shard committed|No marker: roll back, copy again.|+|F3: crash during `COMMIT PREPARED`|Some shards committed|Markers on some shards: commit the remaining shards.|+|F4: connection lost during `COMMIT PREPARED`|Unknown on one shard|Resolved as F3|+|F5: failover to an async standby after commit|Part can be missing on one shard|Markers missing on that shard. Refuse.|++Without truncate, a part that a failover lost cannot be copied again, because its rows on other shards stay. Resume can only detect the loss and refuse. The operator then decides.++The marker table is a new object on each shard. Exclude it from the dump and from `FOR ALL TABLES` publications.++Rules:++- Resolve all prepared transactions of the run with the status table before any copy starts.+- Each attempt has a new copy LSN.+- Prepared transactions that a crash leaves behind hold locks and the transaction horizon on the destination until resume or abort.++### Approach++1. Check `max_prepared_transactions` on the destination shards.+2. Resolve the prepared transactions of the run with the rules above.+3. Load publication tables and copy progress. After the copy stage, refuse if a table has no progress or progress exists for a removed table.+4. Check that tables with no complete part have no rows.+5. Copy the parts with no progress. Commit each part with 2PC and record its progress.+6. Give the copy LSNs of all complete parts to table sync and replication.+7. Create missing origins. Write `table_sync`. Run table sync.++A redone part gets a later copy LSN. The forward slot is older, so replay covers every change.++On stop, pgdog commits or rolls back its own prepared transactions. After a crash, the next resume or abort resolves them. Abort otherwise works as in Stage 1.++### Tests++|#|Scenario|Expected result|+|---|---|---|+|6|STOP_TASK during copy. Resume.|Complete parts not copied again. No truncate. Data check passes.|+|7|SIGKILL during copy. Restart. Resume.|As 6. No prepared transactions remain. The source slot count does not grow.|+|8|Pause after copy. Add a table to the publication. Resume.|Refused.|+|13|Kill pgdog after all shards prepare, before the first `COMMIT PREPARED`. Resume.|Rolled back and copied again. Data check passes.|+|14|Kill pgdog after shard 0 commits, before shard 1. Resume.|Shard 1 committed. Part not copied again. Data check passes.|+|15|Kill a destination Postgres right after a part commits. Restart it. Resume.|The part is still there. Data check passes.|++Unit tests: the resolve rule for each mix of prepared, committed and missing shards. Shard markers commit and roll back with the data.++Result: a copy crash never copies complete parts again, never truncates, leaves no prepared transactions, and RESHARD runs resume in replication.++### Open questions++- Schema exclusion: the publication table list and `pg_dump` exclude `pgdog.resharding`. A `FOR ALL TABLES` publication still sends its changes, and apply skips them because the table is not in its list. Stage 3 must exclude the marker tables the same way.++## Stage 4. Schema phase resume++Goal: RESHARD with schema sync resumes in every forward phase.++### Approach++Most schema statements already skip objects that exist. Foreign keys come only in post-data. The one gap: a crash during a concurrent index build leaves an INVALID index, and a rerun skips it.++- Add stage values for the schema phases, and skip pre-data and post-data when the stored stage is past them.+- Pre-data and validation: run again.+- Post-data: drop INVALID indexes on publication tables on all destination shards. Then run again.++### Interface changes++None. A start command with `WITH (resume)` accepts runs stopped in schema phases.++### Tests++|#|Scenario|Expected result|+|---|---|---|+|9|SIGKILL during a concurrent index build. Resume.|No INVALID index stays.|++Result: every forward phase of RESHARD is resumable.++### Open questions++None.++## Stage 5. Cutover recovery++Goal: a crash at any cutover step is recoverable. Reverse replication is never resumed: after the cutover, the operator starts REPLICATE when needed.++Today cutover drops the forward slots before the config swap. A crash between them loses data. The swap is in memory only, so a restart reads the old config.++### Interface changes++- Cutover deletes the row at its end, as in Stage 1. The reverse phase is not recorded.+- The state stores the cutover step. Stage 5 adds the column `cutover_step` with additive DDL and the default `none`. Rows from earlier stages read as `none`.+- The config swap survives a restart (with P1, P2 or P4), or resume prints the needed change (P3).+- CUTOVER is unchanged.++### Approach++New cutover order. Forward slots stay until the swap is durable. The existing traffic stop, cutover schema, reverse slot creation and fleet hooks stay.++```mermaid+sequenceDiagram+ participant C as pgdog+ participant J as State+ participant S as Source+ participant D as Destination+ C->>J: stage cutover, step started+ C->>S: stop traffic, cancel queries, drain+ C->>D: cutover schema, validation+ C->>D: create reverse slots+ C->>S: create reverse origins+ C->>C: durable config swap, fleet sync+ C->>J: step swap_done+ C->>C: resume traffic+ C->>S: drop forward slots+ C->>D: drop forward origins+ C->>J: delete the row+```++The state table is on destination shard 0. Before the swap, pgdog finds it through the destination name. After the swap, through the source name. This shows the swap state with no stored endpoints.++|Step|Live config|Resume action|+|---|---|---|+|`started`|Not swapped|Release fleet traffic. Drop reverse objects. Back to forward replication.|+|`started`|Swapped|Write `swap_done`. Continue below.|+|`swap_done`|Swapped|Release traffic. Drop forward slots and origins. Delete the row. Start reverse replication as a new phase with no row.|+|`swap_done`|Not swapped|Refuse. The swap was not durable.|+|Any|Neither side|Refuse. The operator fixes the config.|++A failed durable swap or fleet sync reverts the swap, sends traffic to the source and pauses. A rollback cutover uses the same steps in reverse.++Stop at `started` releases traffic and pauses. Abort at `started` also drops reverse objects and the forward objects the run created. Abort at `swap_done` drops all remaining objects.++### Config persistence++|Option|How|Pros|Cons|+|---|---|---|---|+|P1|Rewrite config files with a temporary file and rename|Restart reads the right config|Loses comments. Fails on read-only mounts.|+|P2|On startup, apply `swap_done` runs from the state|Works with read-only config|Startup needs the database|+|P3|Refuse on mismatch and print the needed change|No new write path|Manual step after each such crash|+|P4|Control plane stores the config|Hook exists|Enterprise only|++### Reverse slot timing++|Option|When|Pros|Cons|+|---|---|---|---|+|R1|Before the swap, traffic stopped|Rollback path exists before the first new write. A crash before the swap only drops reverse objects.|Both slot sets exist for a short time.|+|R2|After `swap_done`, before traffic resumes|Fewer stale objects|A crash between them leaves no rollback path. Resume may stop traffic again.|++### Tests++|#|Scenario|Expected result|+|---|---|---|+|10|Fail after reverse slots exist, before the swap. Resume.|Traffic on source. Reverse objects dropped. Cutover runs again later.|+|11|Fail after `swap_done`, before the forward drop. Resume.|Traffic on destination. Forward slots dropped. Row gone. Reverse replication runs with no row.|+|12|`WITH (abort)` at `swap_done`.|Row gone. No run slots or origins on either cluster.|++Result: an injected failure at each cutover step, then resume, ends in the action of the cutover table.++### Open questions++- Config persistence: P1, P2, P3 or P4. Not decided. With P3, a restart after `swap_done` serves the old source until the operator fixes the config. Writes made in that window sit in the forward slots. Resume at `swap_done` must then drain the forward slots before it drops them, or P3 is not safe.+- Reverse slot timing: R1 or R2.+- The enterprise release-traffic hook can run alone after a restart. Unverified. If not, resume refuses and asks the operator to release maintenance mode.+- Postgres allows forward and reverse slots at the same time. If a slot limit blocks R1, use R2.++## Stage 6. Operations++### Interface changes++- Slot WAL status and retained bytes in `SHOW RESHARDING`. A paused run keeps its slot, and the slot holds WAL.++Tests: second-instance resume for every stage.++Result: all scenarios of Stages 1 to 5 pass.++### Open questions++None.diff --git a/integration/resharding/cutover.sh b/integration/resharding/cutover.shnew file mode 100755index 000000000..a8f30cd0e--- /dev/null+++ b/integration/resharding/cutover.sh@@ -0,0 +1,61 @@+#!/bin/bash+set -euo pipefail++SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )+source "${SCRIPT_DIR}/shared.sh"++resharding_records() {+ local exists+ exists=$(pg_sql 15434 "SELECT to_regclass('pgdog.resharding') IS NOT NULL") || return 1+ if [ "${exists}" = t ]; then+ pg_sql 15434 'SELECT count(*) FROM pgdog.resharding'+ else+ echo 0+ fi+}++slot_count() {+ pg_sql "$1" "SELECT count(*) FROM pg_replication_slots WHERE starts_with(slot_name::text, '$2')"+}++origin_count() {+ pg_sql "$1" "SELECT count(*) FROM pg_replication_origin WHERE starts_with(roname, '__pgdog_origin_$2')"+}++lock_holders() {+ pg_sql 15434 "SELECT count(*) FROM pg_locks l JOIN pg_stat_activity a ON a.pid = l.pid WHERE l.locktype = 'advisory' AND l.granted AND starts_with(a.application_name, 'pgdog_resharding:')"+}++initialize_resharding+RESHARD_TASK_ID=$(psql admin -tAc 'RESHARD source destination pgdog WITH (cleanup_on_cancel false)')+# The record is written inside the task.+wait_for_value 1 resharding_records+SLOT=$(pg_sql 15434 'SELECT slot_name FROM pgdog.resharding')+prepare_resharding_verification+wait_for_replication_catchup+verify_resharding_data+wait_admin_task_inner_status "${RESHARD_TASK_ID}" 'reverse replicating'++test "$(resharding_records)" = 0+wait_for_value 0 lock_holders+RESHARDING=$(psql admin -tAc 'SHOW RESHARDING')+test -z "${RESHARDING}"+test "$(slot_count 15432 "${SLOT}_0")" = 0+test "$(slot_count 15433 "${SLOT}_1")" = 0+test "$(slot_count 15434 "${SLOT}_0")" = 1+test "$(slot_count 15435 "${SLOT}_1")" = 1+wait_for_value 2 origin_count 15432 "${SLOT}"+wait_for_value 2 origin_count 15433 "${SLOT}"++if psql admin -tAc 'RESHARD source destination pgdog WITH (abort)'; then+ echo "abort after the cutover must fail" >&2+ exit 1+fi++psql admin -c "STOP_TASK ${RESHARD_TASK_ID}"+wait_admin_task_finished "${RESHARD_TASK_ID}"+wait_for_value 0 slot_count 15434 "${SLOT}_0"+wait_for_value 0 slot_count 15435 "${SLOT}_1"+wait_for_value 0 origin_count 15432 "${SLOT}"+wait_for_value 0 origin_count 15433 "${SLOT}"+test "$(resharding_records)" = 0diff --git a/integration/resharding/resumable.sh b/integration/resharding/resumable.shnew file mode 100755index 000000000..5eeeb6383--- /dev/null+++ b/integration/resharding/resumable.sh@@ -0,0 +1,94 @@+#!/bin/bash+set -euo pipefail++SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )+source "${SCRIPT_DIR}/shared.sh"++source_slots() {+ local first second+ first=$(pg_sql 15432 "SELECT count(*) FROM pg_replication_slots WHERE slot_name = '$1_0'") || return 1+ second=$(pg_sql 15433 "SELECT count(*) FROM pg_replication_slots WHERE slot_name = '$1_1'") || return 1+ echo $((first + second))+}++# command|state|replication_slot of the source -> destination resharding.+resharding() {+ local rows+ rows=$(psql admin -tAc 'SHOW RESHARDING') || return 1+ grep '^source|destination|' <<< "${rows}" | cut -d'|' -f3,6,9 || true+}++task_status() {+ local row+ row=$(admin_task_row "$1") || return 1+ echo "${row%%|*}" | cut -d: -f1+}++NO_RESHARDING='no unfinished resharding source -> destination (pgdog)'++# An abort must fail at once with the given error.+abort_fails_with() {+ local error+ if error=$(psql admin -tAc "$1" 2>&1); then+ echo "$1 must fail, returned: ${error}" >&2+ return 1+ fi+ if [[ "${error}" != *"$2"* ]]; then+ echo "$1 failed with: ${error}, expected: $2" >&2+ return 1+ fi+}++initialize_resharding++# Abort before any run is refused.+abort_fails_with 'COPY_DATA source destination pgdog WITH (abort)' "${NO_RESHARDING}"++# A resumable run shows as running while it runs.+COPY_RESULT=$(psql admin -tAc 'COPY_DATA source destination pgdog WITH (cleanup_on_cancel false)')+COPY_TASK_ID=${COPY_RESULT%%|*}+SLOT=${COPY_RESULT#*|}+test -n "${SLOT}"+wait_admin_task_inner_status "${COPY_TASK_ID}" 'replicating'+test "$(resharding)" = "reshard|running|${SLOT}"++# STOP_TASK pauses it and keeps its slots.+psql admin -c "STOP_TASK ${COPY_TASK_ID}"+wait_admin_task_status "${COPY_TASK_ID}" 'cancelled'+wait_for_value "reshard|paused|${SLOT}" resharding+test "$(source_slots "${SLOT}")" = 2++# The pause survives a restart.+kill -TERM "${PGBENCH_PID}" 2>/dev/null || true+wait "${PGBENCH_PID}" || true+PGBENCH_PID=""+restart_pgdog+wait_for_value "reshard|paused|${SLOT}" resharding+test "$(source_slots "${SLOT}")" = 2++# A new run with the same key is refused while the paused run exists.+NEW_TASK_ID=$(psql admin -tAc 'COPY_DATA source destination pgdog' | cut -d'|' -f1)+wait_for_value failed task_status "${NEW_TASK_ID}"+test "$(resharding)" = "reshard|paused|${SLOT}"++# Abort with the wrong command is refused.+abort_fails_with 'REPLICATE source destination pgdog WITH (abort)' \+ 'resharding command is "reshard", not "replicate"'++# Abort removes the run and its slots.+ABORT_TASK_ID=$(psql admin -tAc 'COPY_DATA source destination pgdog WITH (abort)')+wait_admin_task_finished "${ABORT_TASK_ID}"+RESHARDING=$(resharding)+test -z "${RESHARDING}"+test "$(source_slots "${SLOT}")" = 0++# A second abort is refused: the run is gone.+abort_fails_with 'COPY_DATA source destination pgdog WITH (abort)' "${NO_RESHARDING}"++# A plain run removes its slots and leaves no run on STOP_TASK.+PLAIN_TASK_ID=$(psql admin -tAc 'REPLICATE source destination pgdog plain_slot')+wait_for_value 'replicate|running|plain_slot' resharding+psql admin -c "STOP_TASK ${PLAIN_TASK_ID}"+wait_admin_task_status "${PLAIN_TASK_ID}" 'cancelled'+wait_for_value '' resharding+wait_for_value 0 source_slots plain_slotdiff --git a/integration/resharding/run.sh b/integration/resharding/run.shindex c40753767..55778072c 100755--- a/integration/resharding/run.sh+++ b/integration/resharding/run.sh@@ -13,3 +13,5 @@ trap cleanup EXIT INT TERM
timeout --signal=TERM --kill-after=90s 16m bash "${SCRIPT_DIR}/reshard.sh"
timeout --signal=TERM --kill-after=90s 16m bash "${SCRIPT_DIR}/admin.sh"
timeout --signal=TERM --kill-after=90s 16m bash "${SCRIPT_DIR}/cli.sh"
+timeout --signal=TERM --kill-after=90s 16m bash "${SCRIPT_DIR}/resumable.sh"+timeout --signal=TERM --kill-after=90s 16m bash "${SCRIPT_DIR}/cutover.sh"diff --git a/integration/resharding/shared.sh b/integration/resharding/shared.shindex d81713000..727cec675 100755--- a/integration/resharding/shared.sh+++ b/integration/resharding/shared.sh@@ -148,6 +148,25 @@ initialize_resharding() {
sleep 10
}
+restart_pgdog() {+ kill -TERM "${PGDOG_PID}"+ wait "${PGDOG_PID}" || true+ "${PGDOG_BIN}" &+ PGDOG_PID="$!"+ local deadline=$((SECONDS + 60))+ until psql admin -c 'SHOW TASKS' > /dev/null 2>&1; do+ if ! kill -0 "${PGDOG_PID}" 2>/dev/null; then+ echo "ERROR: PgDog exited during restart"+ return 1+ fi+ if [ "${SECONDS}" -ge "${deadline}" ]; then+ echo "ERROR: PgDog did not accept admin connections within 60s after restart"+ return 1+ fi+ sleep 1+ done+}+
replace_copy_with_replicate() {
local table="$1"
local column="$2"
@@ -401,3 +420,55 @@ verify_resharding_data() {
return "${failed}"
}
++pg_sql() { PGPASSWORD=pgdog psql -h 127.0.0.1 -p "$1" -U pgdog -d postgres -v ON_ERROR_STOP=1 -tAc "$2"; }++admin_task_type() {+ local target="$1"+ psql admin -tAc 'SHOW TASKS' | while IFS='|' read -r id parent kind rest; do+ if [ "${id}" = "${target}" ]; then+ printf '%s' "${kind}"+ break+ fi+ done+}++wait_admin_task_status() {+ local task_id="$1"+ local expected="$2"+ local row status+ for _ in $(seq 1 120); do+ row=$(admin_task_row "${task_id}")+ status=${row%%|*}+ if [ "${status}" = "${expected}" ]; then+ return 0+ fi+ case "${status}" in+ finished|failed:*|panicked:*|cancelled)+ echo "task ${task_id} reached ${status} while waiting for ${expected}"+ psql admin -c 'SHOW TASKS'+ return 1+ ;;+ esac+ sleep 1+ done+ echo "task ${task_id} did not reach ${expected} within 120s"+ psql admin -c 'SHOW TASKS'+ return 1+}++wait_for_value() {+ local expected="$1"+ shift+ local value=""+ for _ in $(seq 1 30); do+ value=$("$@") || value=""+ if [ "${value}" = "${expected}" ]; then+ return 0+ fi+ sleep 1+ done+ echo "$* returned ${value}, expected ${expected} within 30s"+ psql admin -c 'SHOW TASKS'+ return 1+}diff --git a/integration/rust/tests/integration/admin/resharding/abort.rs b/integration/rust/tests/integration/admin/resharding/abort.rsnew file mode 100644index 000000000..4d4cce747--- /dev/null+++ b/integration/rust/tests/integration/admin/resharding/abort.rs@@ -0,0 +1,213 @@+use crate::setup::{admin_sqlx, connection_sqlx_direct, connection_sqlx_direct_db};+use pgdog_stats::TaskProgress;+use sqlx::{Executor, Pool, Postgres, Row};++use super::replication::prepare_replication;+use super::table_copies::poll;+use super::{+ TEST_PUB, assert_layout, cleanup, fail_if_task_errored, run_task_command, test_slot_names,+ wait_for_task, wait_for_task_status, with_cleanup,+};++const SLOT: &str = "__pgdog_repl_abort";++fn abort(command: &str, slot: &str) -> String {+ format!("{command} pgdog pgdog_sharded {TEST_PUB}{slot} WITH (abort)")+}++async fn admin_error(admin: &Pool<Postgres>, sql: &str) -> String {+ match admin.execute(sql).await {+ Err(sqlx::Error::Database(err)) => err.message().to_owned(),+ other => panic!("{sql} must fail with a database error, got {other:?}"),+ }+}++async fn resharding_records() -> i64 {+ let shard = connection_sqlx_direct_db("shard_0").await;+ let exists: bool = sqlx::query_scalar("SELECT to_regclass('pgdog.resharding') IS NOT NULL")+ .fetch_one(&shard)+ .await+ .unwrap();+ if !exists {+ return 0;+ }+ sqlx::query_scalar("SELECT count(*) FROM pgdog.resharding")+ .fetch_one(&shard)+ .await+ .unwrap()+}++async fn origin_count(db: &str) -> i64 {+ let shard = connection_sqlx_direct_db(db).await;+ sqlx::query_scalar("SELECT count(*) FROM pg_replication_origin WHERE starts_with(roname, $1)")+ .bind(format!("__pgdog_origin_{SLOT}"))+ .fetch_one(&shard)+ .await+ .expect("replication origins must be readable")+}++const SHOW_RESHARDING_LAYOUT: &[(&str, &str)] = &[+ ("source", "TEXT"),+ ("destination", "TEXT"),+ ("command", "TEXT"),+ ("publication", "TEXT"),+ ("stage", "TEXT"),+ ("state", "TEXT"),+ ("instance", "TEXT"),+ ("task_id", "TEXT"),+ ("replication_slot", "TEXT"),+ ("started_at", "TEXT"),+ ("resumed_at", "TEXT"),+];++const RESHARDING_COLUMNS: [&str; 5] = [+ "command",+ "publication",+ "state",+ "replication_slot",+ "task_id",+];++async fn show_resharding(admin: &Pool<Postgres>) -> Vec<Vec<Option<String>>> {+ let rows = admin.fetch_all("SHOW RESHARDING").await.unwrap();+ if !rows.is_empty() {+ assert_layout(&rows, SHOW_RESHARDING_LAYOUT);+ }+ rows.iter()+ .filter(|row| {+ row.get::<Option<String>, _>("source").as_deref() == Some("pgdog")+ && row.get::<String, _>("destination") == "pgdog_sharded"+ })+ .map(|row| {+ RESHARDING_COLUMNS+ .iter()+ .map(|column| row.get::<Option<String>, _>(*column))+ .collect()+ })+ .collect()+}++fn resharding_row(state: &str, task_id: Option<i64>) -> Vec<Option<String>> {+ vec![+ Some("replicate".into()),+ Some(TEST_PUB.into()),+ Some(state.into()),+ Some(SLOT.into()),+ task_id.map(|id| id.to_string()),+ ]+}++#[tokio::test]+async fn test_abort_refuses_a_running_run_and_cleans_up_a_paused_one() {+ let direct = connection_sqlx_direct().await;+ let admin = admin_sqlx().await;+ cleanup(&admin, &direct).await;++ with_cleanup(&admin, &direct, async {+ prepare_replication(&admin, &direct).await;+ let missing = format!("no unfinished resharding pgdog -> pgdog_sharded ({TEST_PUB})");++ assert_eq!(+ admin_error(&admin, &abort("REPLICATE", "")).await,+ missing,+ "abort before any run"+ );++ let task_id = run_task_command(+ &admin,+ &format!(+ "REPLICATE pgdog pgdog_sharded {TEST_PUB} {SLOT} WITH (cleanup_on_cancel false)"+ ),+ )+ .await;+ wait_for_task(&admin, "replication ready", |task| {+ task.id == Some(task_id)+ && task.status == TaskProgress::Running+ && task.inner_status == "replicating"+ })+ .await;+ poll("the resharding row after slot creation", || async {+ fail_if_task_errored(&admin, task_id).await;+ (resharding_records().await == 1).then_some(())+ })+ .await;+ let running = poll("SHOW RESHARDING to list the running resharding", || async {+ fail_if_task_errored(&admin, task_id).await;+ let rows = show_resharding(&admin).await;+ (rows.len() == 1).then_some(rows)+ })+ .await;+ assert_eq!(running, [resharding_row("running", Some(task_id))]);++ let wrong_command = r#"resharding command is "replicate", not "reshard""#;+ for (sql, expected) in [+ (abort("RESHARD", ""), wrong_command.to_owned()),+ (abort("COPY_DATA", ""), wrong_command.to_owned()),+ (+ abort("REPLICATE", " other"),+ format!(r#"resharding slot is "{SLOT}", not "other""#),+ ),+ ] {+ assert_eq!(admin_error(&admin, &sql).await, expected, "{sql}");+ }+ wait_for_task(&admin, "replication still running", |task| {+ task.id == Some(task_id)+ && task.status == TaskProgress::Running+ && task.inner_status == "replicating"+ })+ .await;++ let abort_slot = abort("REPLICATE", &format!(" {SLOT}"));+ let locked = admin_error(&admin, &abort_slot).await;+ assert!(+ locked.starts_with(&format!(+ "resharding pgdog -> pgdog_sharded ({TEST_PUB}) is owned by another session ("+ )) && locked.ends_with("; stop it with STOP_TASK first"),+ "abort of a running resharding: {locked}"+ );+ wait_for_task(&admin, "replication not stopped by the abort", |task| {+ task.id == Some(task_id)+ && task.status == TaskProgress::Running+ && task.inner_status == "replicating"+ })+ .await;+ assert_eq!(resharding_records().await, 1);++ let stopped = admin+ .fetch_one(format!("STOP_TASK {task_id}").as_str())+ .await+ .expect("replication stop must succeed");+ assert_eq!(stopped.get::<String, _>("stop_task"), "OK");+ wait_for_task_status(&admin, task_id, TaskProgress::Cancelled).await;+ assert_eq!(resharding_records().await, 1);+ assert_eq!(test_slot_names(&direct).await, [format!("{SLOT}_0")]);+ let paused = poll("SHOW RESHARDING to list the paused resharding", || async {+ let rows = show_resharding(&admin).await;+ (rows.len() == 1 && rows[0][2].as_deref() == Some("paused")).then_some(rows)+ })+ .await;+ assert_eq!(paused, [resharding_row("paused", None)]);++ let abort_id = run_task_command(&admin, &abort_slot).await;+ wait_for_task(&admin, "abort task listed", |task| {+ task.id == Some(abort_id) && task.kind == "abort"+ })+ .await;+ wait_for_task_status(&admin, abort_id, TaskProgress::Finished).await;++ assert_eq!(resharding_records().await, 0);+ assert_eq!(+ show_resharding(&admin).await,+ Vec::<Vec<Option<String>>>::new()+ );+ assert_eq!(test_slot_names(&direct).await, Vec::<String>::new());+ assert_eq!(origin_count("shard_0").await, 0);+ assert_eq!(origin_count("shard_1").await, 0);+ assert_eq!(+ admin_error(&admin, &abort("REPLICATE", "")).await,+ missing,+ "abort after the abort"+ );+ })+ .await;+}diff --git a/integration/rust/tests/integration/admin/resharding/mod.rs b/integration/rust/tests/integration/admin/resharding/mod.rsindex d8f42d7ad..230418232 100644--- a/integration/rust/tests/integration/admin/resharding/mod.rs+++ b/integration/rust/tests/integration/admin/resharding/mod.rs@@ -1,3 +1,4 @@+pub mod abort;
pub mod copy_data;
pub mod replication;
pub mod replication_slots;
@@ -11,7 +12,7 @@ use std::time::Duration;
use futures_util::FutureExt;
-use crate::setup::connection_sqlx_direct_db;+use crate::setup::{connection_sqlx_direct, connection_sqlx_direct_db};
use sqlx::{Executor, Pool, Postgres, Row};
use tokio::time::{sleep, timeout};
@@ -117,6 +118,31 @@ async fn test_slot_names(pool: &Pool<Postgres>) -> Vec<String> {
.expect("replication slots must be readable")
}
+/// Wait until the forward slot of shard 0 exists on the source and has a+/// confirmed flush position. Changes are streamed only after the slot is+/// ready, and the task status can come first.+async fn wait_for_source_slot(admin: &Pool<Postgres>, task_id: i64, slot: Option<&str>) {+ let (condition, value) = match slot {+ Some(slot) => ("slot_name = $1", format!("{slot}_0")),+ None => ("slot_name LIKE $1", r"\_\_pgdog\_repl\_%\_0".to_owned()),+ };+ let query = format!(+ "SELECT count(*) FROM pg_replication_slots \+ WHERE {condition} AND confirmed_flush_lsn IS NOT NULL"+ );+ let direct = connection_sqlx_direct().await;+ table_copies::poll("the source replication slot", || async {+ fail_if_task_errored(admin, task_id).await;+ let slots: i64 = sqlx::query_scalar(&query)+ .bind(&value)+ .fetch_one(&direct)+ .await+ .expect("replication slots must be readable");+ (slots > 0).then_some(())+ })+ .await;+}+
async fn drop_all_test_slots(direct: &Pool<Postgres>) {
drop_test_slots(direct).await;
for db in &["shard_0", "shard_1"] {
@@ -124,6 +150,15 @@ async fn drop_all_test_slots(direct: &Pool<Postgres>) {
}
}
+async fn drop_test_origins(pool: &Pool<Postgres>) {+ let _ = pool+ .execute(+ r"SELECT pg_replication_origin_drop(roname) FROM pg_replication_origin+ WHERE roname LIKE '\_\_pgdog\_origin\_\_\_pgdog\_repl\_%'",+ )+ .await;+}+
async fn cleanup(admin: &Pool<Postgres>, direct: &Pool<Postgres>) {
drain_tasks(admin).await;
@@ -131,6 +166,13 @@ async fn cleanup(admin: &Pool<Postgres>, direct: &Pool<Postgres>) {
sleep(Duration::from_millis(500)).await;
drop_all_test_slots(direct).await;
+ for db in &["shard_0", "shard_1"] {+ drop_test_origins(&connection_sqlx_direct_db(db).await).await;+ }+ let _ = connection_sqlx_direct_db("shard_0")+ .await+ .execute("DROP TABLE IF EXISTS pgdog.resharding")+ .await;
let _ = direct
.execute(format!("DROP PUBLICATION IF EXISTS {TEST_PUB}").as_str())
diff --git a/integration/rust/tests/integration/admin/resharding/replication.rs b/integration/rust/tests/integration/admin/resharding/replication.rsindex 1e398c13b..f0ee51101 100644--- a/integration/rust/tests/integration/admin/resharding/replication.rs+++ b/integration/rust/tests/integration/admin/resharding/replication.rs@@ -11,7 +11,7 @@ use super::table_copies::poll;
use super::{
POLL, TEST_PUB, TEST_SCHEMA, TEST_TABLE, cleanup, create_publication, create_test_table,
fail_if_task_errored, run_task_command, seed_rows, task_status_line, test_slot_names,
- wait_for_task, wait_for_task_status, with_cleanup,+ wait_for_source_slot, wait_for_task, wait_for_task_status, with_cleanup,
};
pub(super) async fn prepare_replication(admin: &Pool<Postgres>, direct: &Pool<Postgres>) {
@@ -34,6 +34,8 @@ pub(super) async fn start_replication(admin: &Pool<Postgres>, slot: Option<&str>
})
.await;
+ wait_for_source_slot(admin, task_id, slot).await;+
task_id
}
diff --git a/pgdog-stats/Cargo.toml b/pgdog-stats/Cargo.tomlindex 647124101..effd33a2c 100644--- a/pgdog-stats/Cargo.toml+++ b/pgdog-stats/Cargo.toml@@ -13,6 +13,7 @@ indexmap.workspace = true
schemars.workspace = true
pgdog-util.workspace = true
derive_more = { workspace = true, features = ["error", "from_str"] }
+uuid.workspace = true
[dev-dependencies]
serde_json.workspace = true
diff --git a/pgdog-stats/src/identity.rs b/pgdog-stats/src/identity.rsnew file mode 100644index 000000000..f349e7b66--- /dev/null+++ b/pgdog-stats/src/identity.rs@@ -0,0 +1,80 @@+// TODO: the ee has the same implementation, migrate the ee to this one.++use std::sync::LazyLock;++use derive_more::{Display, From, FromStr};+use schemars::JsonSchema;+use serde::{Deserialize, Serialize};+use uuid::Uuid;++pub const INSTANCE_ID_HEADER: &str = "x-pgdog-instance-id";+pub const RUN_ID_HEADER: &str = "x-pgdog-run-id";++#[derive(+ Debug, Clone, PartialEq, Eq, Hash, Display, From, FromStr, Serialize, Deserialize, JsonSchema,+)]+#[serde(transparent)]+pub struct InstanceId(String);++impl InstanceId {+ pub fn as_str(&self) -> &str {+ &self.0+ }+}++#[derive(+ Debug, Clone, Copy, PartialEq, Eq, Hash, Display, FromStr, Serialize, Deserialize, JsonSchema,+)]+#[serde(transparent)]+pub struct RunId(Uuid);++impl RunId {+ pub fn generate() -> Self {+ Self(Uuid::new_v4())+ }+}++static RUN_ID: LazyLock<RunId> = LazyLock::new(RunId::generate);++pub fn run_id() -> RunId {+ *RUN_ID+}++/// Identifier of one resharding run, kept across its pauses and resumes.+#[derive(+ Debug, Clone, Copy, PartialEq, Eq, Hash, Display, FromStr, Serialize, Deserialize, JsonSchema,+)]+#[serde(transparent)]+pub struct ReshardingId(Uuid);++impl ReshardingId {+ pub fn generate() -> Self {+ Self(Uuid::now_v7())+ }+}++#[cfg(test)]+mod test {+ use super::*;++ #[test]+ fn test_ids_round_trip_through_json() {+ let run_id = RunId::generate();+ let json = serde_json::to_string(&run_id).unwrap();+ assert_eq!(json, format!("\"{run_id}\""));+ assert_eq!(serde_json::from_str::<RunId>(&json).unwrap(), run_id);++ let instance_id = InstanceId::from("pgdog-0".to_owned());+ let json = serde_json::to_string(&instance_id).unwrap();+ assert_eq!(json, r#""pgdog-0""#);+ assert_eq!(+ serde_json::from_str::<InstanceId>(&json).unwrap(),+ instance_id+ );+ }++ #[test]+ fn test_run_id_is_stable_within_the_process() {+ assert_eq!(run_id(), run_id());+ }+}diff --git a/pgdog-stats/src/lib.rs b/pgdog-stats/src/lib.rsindex 130cdbe31..9566f08ab 100644--- a/pgdog-stats/src/lib.rs+++ b/pgdog-stats/src/lib.rs@@ -1,16 +1,20 @@
pub mod client;
+pub mod identity;
pub mod memory;
pub mod pool;
pub mod replication;
+pub mod resharding;
pub mod schema;
pub mod server;
pub mod state;
pub mod task;
pub mod user;
+pub use identity::*;
pub use memory::*;
pub use pool::*;
pub use replication::*;
+pub use resharding::*;
pub use schema::*;
pub use task::*;
pub use user::*;
diff --git a/pgdog-stats/src/resharding.rs b/pgdog-stats/src/resharding.rsnew file mode 100644index 000000000..122d4d77b--- /dev/null+++ b/pgdog-stats/src/resharding.rs@@ -0,0 +1,17 @@+//! Resharding types shared with the control plane.++use derive_more::Display;+use schemars::JsonSchema;+use serde::{Deserialize, Serialize};++/// The kind of command that started a resharding run.+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Display, Serialize, Deserialize, JsonSchema)]+#[display(rename_all = "snake_case")]+#[serde(rename_all = "snake_case")]+pub enum CommandKind {+ Reshard,+ Replicate,+ /// A kind this build does not know, sent by a newer pgdog.+ #[serde(other)]+ Unknown,+}diff --git a/pgdog-stats/src/task.rs b/pgdog-stats/src/task.rsindex 91b580e1d..d63b360eb 100644--- a/pgdog-stats/src/task.rs+++ b/pgdog-stats/src/task.rs@@ -12,12 +12,14 @@ use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
use serde_with::{TimestampMilliSeconds, serde_as, skip_serializing_none};
+pub mod abort_resharding;
pub mod copy_data;
pub mod replication;
pub mod reshard;
pub mod schema_sync;
pub mod synchronize_tables;
+pub use abort_resharding::*;
pub use copy_data::*;
pub use replication::*;
pub use reshard::*;
@@ -360,6 +362,7 @@ pub enum TaskDefinitionKind {
ReplicationCluster(ReplicationClusterDefinition),
ReplicationShard(ReplicationShardDefinition),
Reshard(ReshardDefinition),
+ AbortResharding(AbortReshardingDefinition),
/// No detail beyond the name, or a `kind` this build does not know.
#[default]
#[display("")]
@@ -372,6 +375,7 @@ impl TaskDefinitionKind {
pub const fn kind(&self) -> &'static str {
match self {
Self::Reshard(_) => "reshard",
+ Self::AbortResharding(_) => "abort_resharding",
Self::CopyData(_) => "copy_data",
Self::TableCopy(_) => "table_copy",
Self::SynchronizeTables(_) => "synchronize_tables",
@@ -386,7 +390,7 @@ impl TaskDefinitionKind {
}
/// The two ends of a migration.
-#[derive(Debug, Clone, PartialEq, Display, Serialize, Deserialize, JsonSchema)]+#[derive(Debug, Clone, PartialEq, Eq, Hash, Display, Serialize, Deserialize, JsonSchema)]
#[display("{source} -> {destination}")]
pub struct Databases {
pub source: String,
@@ -405,7 +409,7 @@ pub struct RatioProgress {
#[cfg(test)]
mod test {
use super::*;
- use crate::{Lsn, MissedRows, SyncState};+ use crate::{CommandKind, Lsn, MissedRows, SyncState};
use pgdog_config::CopyFormat;
use std::time::{Duration, UNIX_EPOCH};
@@ -426,7 +430,7 @@ mod test {
/// One definition per kind. The exhaustive `match` in
/// [`test_definition_round_trip`] forces a new kind into this list.
- fn definitions() -> [TaskDefinition; 10] {+ fn definitions() -> [TaskDefinition; 11] {
[
"test task".into(),
ReshardDefinition {
@@ -435,6 +439,7 @@ mod test {
replicate_only: false,
sync_only: false,
auto_cutover: true,
+ cleanup_on_cancel: false,
}
.into(),
CopyDataDefinition {
@@ -458,6 +463,7 @@ mod test {
ReplicationDefinition {
databases: databases(),
auto_cutover: false,
+ cleanup_on_cancel: true,
}
.into(),
ReplicationClusterDefinition {
@@ -482,6 +488,11 @@ mod test {
sync_state: SyncState::PostData,
}
.into(),
+ AbortReshardingDefinition {+ databases: databases(),+ command: CommandKind::Reshard,+ }+ .into(),
]
}
@@ -500,6 +511,7 @@ mod test {
match back.kind {
TaskDefinitionKind::Reshard(_)
+ | TaskDefinitionKind::AbortResharding(_)
| TaskDefinitionKind::CopyData(_)
| TaskDefinitionKind::TableCopy(_)
| TaskDefinitionKind::SynchronizeTables(_)
@@ -526,6 +538,7 @@ mod test {
replicate_only: false,
sync_only: false,
auto_cutover: true,
+ cleanup_on_cancel: true,
})
.to_string(),
"reshard prod -> prod_sharded"
@@ -841,6 +854,54 @@ mod test {
);
}
+ #[test]+ fn test_missing_cleanup_on_cancel_reads_as_true() {+ assert_eq!(+ serde_json::from_str::<TaskDefinition>(+ r#"{"name":"reshard","kind":"reshard","databases":{"source":"prod","destination":"prod_sharded"},"skip_schema_sync":false,"replicate_only":false,"sync_only":false,"auto_cutover":true}"#+ )+ .unwrap(),+ TaskDefinition::from(ReshardDefinition {+ databases: databases(),+ skip_schema_sync: false,+ replicate_only: false,+ sync_only: false,+ auto_cutover: true,+ cleanup_on_cancel: true,+ })+ );+ assert_eq!(+ serde_json::from_str::<TaskDefinition>(+ r#"{"name":"replication","kind":"replication","databases":{"source":"prod","destination":"prod_sharded"},"auto_cutover":true}"#+ )+ .unwrap(),+ TaskDefinition::from(ReplicationDefinition {+ databases: databases(),+ auto_cutover: true,+ cleanup_on_cancel: true,+ })+ );+ }++ #[test]+ fn test_abort_resharding_definition() {+ let definition = TaskDefinition::from(AbortReshardingDefinition {+ databases: databases(),+ command: CommandKind::Reshard,+ });+ assert_eq!(definition.to_string(), "abort reshard prod -> prod_sharded");++ let json = r#"{"name":"abort_resharding","kind":"abort_resharding","databases":{"source":"prod","destination":"prod_sharded"},"command":"reshard"}"#;+ assert_eq!(+ serde_json::to_value(&definition).unwrap(),+ serde_json::from_str::<serde_json::Value>(json).unwrap()+ );+ assert_eq!(+ serde_json::from_str::<TaskDefinition>(json).unwrap(),+ definition+ );+ }+
/// `Other` is a real tag on the wire, not an absence.
#[test]
fn test_other_round_trips() {
diff --git a/pgdog-stats/src/task/abort_resharding.rs b/pgdog-stats/src/task/abort_resharding.rsnew file mode 100644index 000000000..c286cd8bf--- /dev/null+++ b/pgdog-stats/src/task/abort_resharding.rs@@ -0,0 +1,14 @@+//! Abort resharding task definition.++use derive_more::Display;+use schemars::JsonSchema;+use serde::{Deserialize, Serialize};++use crate::{CommandKind, Databases};++#[derive(Debug, Clone, PartialEq, Display, Serialize, Deserialize, JsonSchema)]+#[display("abort {command} {databases}")]+pub struct AbortReshardingDefinition {+ pub databases: Databases,+ pub command: CommandKind,+}diff --git a/pgdog-stats/src/task/replication.rs b/pgdog-stats/src/task/replication.rsindex 8dd8e66ef..bf383f89a 100644--- a/pgdog-stats/src/task/replication.rs+++ b/pgdog-stats/src/task/replication.rs@@ -74,6 +74,8 @@ pub enum ReplicationCutoverReason {
pub struct ReplicationDefinition {
pub databases: Databases,
pub auto_cutover: bool,
+ #[serde(default = "crate::task::reshard::default_cleanup_on_cancel")]+ pub cleanup_on_cancel: bool,
}
/// Stages of logical replication, reported as the task's status.
diff --git a/pgdog-stats/src/task/reshard.rs b/pgdog-stats/src/task/reshard.rsindex 0da1ddeb6..1eda87761 100644--- a/pgdog-stats/src/task/reshard.rs+++ b/pgdog-stats/src/task/reshard.rs@@ -16,6 +16,12 @@ pub struct ReshardDefinition {
pub replicate_only: bool,
pub sync_only: bool,
pub auto_cutover: bool,
+ #[serde(default = "default_cleanup_on_cancel")]+ pub cleanup_on_cancel: bool,+}++pub(crate) fn default_cleanup_on_cancel() -> bool {+ true
}
/// Stages of the migration, reported as the task's status. The fine-grained
diff --git a/pgdog/src/admin/copy_data.rs b/pgdog/src/admin/copy_data.rsindex 99b7c9ff0..6bdfe961a 100644--- a/pgdog/src/admin/copy_data.rs+++ b/pgdog/src/admin/copy_data.rs@@ -5,9 +5,11 @@ use tracing::info;
use crate::api::resharding::ReshardTask;
use crate::api::run_task;
use crate::backend::replication::resharding_state::ReshardingState;
+use pgdog_stats::CommandKind;-use super::options::Options;+use super::options::{Options, RunMode};
use super::prelude::*;
+use super::reshard::check_abort;
pub(crate) struct CopyData {
pub(crate) from_database: String,
@@ -16,6 +18,7 @@ pub(crate) struct CopyData {
pub(crate) replication_slot: Option<String>,
/// `tables_only` option: copy only the publication's tables, without cutover.
pub(crate) tables_only: bool,
+ pub(crate) mode: RunMode,
}
#[async_trait]
@@ -51,7 +54,11 @@ impl Command for CopyData {
(slot, option) => slot.or(option).map(str::to_owned),
};
let tables_only = options.flag("tables_only")?;
+ let mode = options.run_mode()?;
options.finish()?;
+ if mode == RunMode::Abort && tables_only {+ return Err(Error::InvalidOption("abort with tables_only".into()));+ }
Ok(Self {
from_database: from_database.to_owned(),
@@ -59,21 +66,23 @@ impl Command for CopyData {
publication: publication.to_owned(),
replication_slot,
tables_only,
+ mode,
})
}
async fn execute(&self) -> Result<Vec<Message>, Error> {
- info!(- r#"copy_data "{}" to "{}", publication="{}", tables_only={}"#,- self.from_database, self.to_database, self.publication, self.tables_only- );-
let state = ReshardingState::builder()
.source(&self.from_database)
.destination(&self.to_database)
.publication(&self.publication)
.maybe_replication_slot(self.replication_slot.clone())
+ .cleanup_on_cancel(self.mode.cleanup_on_cancel())
.build()?;
+ check_abort!(self, state, CommandKind::Reshard);+ info!(+ r#"copy_data "{}" to "{}", publication="{}", tables_only={}"#,+ self.from_database, self.to_database, self.publication, self.tables_only+ );
let slot_name = state.replication_slot().to_owned();
@@ -138,4 +147,26 @@ mod test {
Err(Error::Syntax)
));
}
++ #[test]+ fn parses_run_mode() {+ for (sql, slot, mode) in [+ (+ "copy_data a b pub slot with (cleanup_on_cancel false)",+ Some("slot"),+ RunMode::Run {+ cleanup_on_cancel: false,+ },+ ),+ ("copy_data a b pub (abort)", None, RunMode::Abort),+ ] {+ let cmd = CopyData::parse(sql).unwrap();+ assert_eq!(cmd.replication_slot.as_deref(), slot, "{sql}");+ assert_eq!(cmd.mode, mode, "{sql}");+ }+ assert!(matches!(+ CopyData::parse("copy_data a b pub (abort, tables_only)"),+ Err(Error::InvalidOption(n)) if n == "abort with tables_only"+ ));+ }
}
diff --git a/pgdog/src/admin/mod.rs b/pgdog/src/admin/mod.rsindex 83dbec0af..f2c58770b 100644--- a/pgdog/src/admin/mod.rs+++ b/pgdog/src/admin/mod.rs@@ -43,6 +43,7 @@ pub(crate) mod show_prepared_statements;
pub(crate) mod show_query_cache;
pub(crate) mod show_replication;
pub(crate) mod show_replication_slots;
+pub(crate) mod show_resharding;
pub(crate) mod show_schema_sync;
pub(crate) mod show_server_memory;
pub(crate) mod show_servers;
@@ -88,6 +89,7 @@ pub(crate) use show_prepared_statements::*;
pub(crate) use show_query_cache::*;
pub(crate) use show_replication::*;
pub(crate) use show_replication_slots::*;
+pub(crate) use show_resharding::*;
pub(crate) use show_schema_sync::*;
pub(crate) use show_server_memory::*;
pub(crate) use show_servers::*;
diff --git a/pgdog/src/admin/options.rs b/pgdog/src/admin/options.rsindex 58182504e..cf5e20d34 100644--- a/pgdog/src/admin/options.rs+++ b/pgdog/src/admin/options.rs@@ -2,6 +2,26 @@
use super::error::Error;
+/// What a resharding admin command does.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub(crate) enum RunMode {+ /// Start a run. With `cleanup_on_cancel` false, a stop pauses it.+ Run { cleanup_on_cancel: bool },+ /// Abort the unfinished run with the same key.+ Abort,+}++impl RunMode {+ pub(crate) fn cleanup_on_cancel(self) -> bool {+ !matches!(+ self,+ Self::Run {+ cleanup_on_cancel: false+ }+ )+ }+}+
/// Options after a command's positional arguments. Each command takes the
/// options it knows, then calls [`Options::finish`] to reject the rest.
#[derive(Debug, Default)]
@@ -62,6 +82,26 @@ impl<'a> Options<'a> {
}
}
+ pub(crate) fn run_mode(&mut self) -> Result<RunMode, Error> {+ let abort = self.flag("abort")?;+ let cleanup_on_cancel = self.take("cleanup_on_cancel");+ if abort && cleanup_on_cancel.is_some() {+ return Err(Error::InvalidOption("abort with cleanup_on_cancel".into()));+ }+ if abort {+ return Ok(RunMode::Abort);+ }+ match cleanup_on_cancel {+ None | Some(None) | Some(Some("true" | "on")) => Ok(RunMode::Run {+ cleanup_on_cancel: true,+ }),+ Some(Some("false" | "off")) => Ok(RunMode::Run {+ cleanup_on_cancel: false,+ }),+ Some(Some(_)) => Err(Error::InvalidOption("cleanup_on_cancel".into())),+ }+ }+
/// Reject options the command didn't take.
pub(crate) fn finish(self) -> Result<(), Error> {
match self.options.first() {
@@ -108,6 +148,62 @@ mod test {
options.finish().unwrap();
}
+ const CLEANUP: RunMode = RunMode::Run {+ cleanup_on_cancel: true,+ };+ const PAUSE: RunMode = RunMode::Run {+ cleanup_on_cancel: false,+ };++ #[test]+ fn parses_run_mode() {+ for (sql, mode) in [+ ("reshard a b", CLEANUP),+ ("reshard a b ()", CLEANUP),+ ("reshard a b (cleanup_on_cancel)", CLEANUP),+ ("reshard a b (cleanup_on_cancel true)", CLEANUP),+ ("reshard a b (cleanup_on_cancel 'true')", CLEANUP),+ ("reshard a b (cleanup_on_cancel on)", CLEANUP),+ ("reshard a b (cleanup_on_cancel false)", PAUSE),+ ("reshard a b with (cleanup_on_cancel off)", PAUSE),+ ("reshard a b (cleanup_on_cancel 'false')", PAUSE),+ ("reshard a b (abort)", RunMode::Abort),+ ] {+ let (_, mut options) = Options::parse(sql).unwrap();+ assert_eq!(options.run_mode().unwrap(), mode, "{sql}");+ options.finish().unwrap();+ }++ for (sql, name) in [+ ("reshard a b (cleanup_on_cancel maybe)", "cleanup_on_cancel"),+ ("reshard a b (cleanup_on_cancel 1)", "cleanup_on_cancel"),+ ("reshard a b (abort true)", "abort"),+ (+ "reshard a b (abort, cleanup_on_cancel)",+ "abort with cleanup_on_cancel",+ ),+ (+ "reshard a b (cleanup_on_cancel false, abort)",+ "abort with cleanup_on_cancel",+ ),+ (+ "reshard a b (abort, cleanup_on_cancel true)",+ "abort with cleanup_on_cancel",+ ),+ ] {+ let (_, mut options) = Options::parse(sql).unwrap();+ assert!(+ matches!(options.run_mode(), Err(Error::InvalidOption(n)) if n == name),+ "{sql}"+ );+ }++ let (_, mut options) =+ Options::parse("reshard a b (cleanup_on_cancel false, other)").unwrap();+ assert_eq!(options.run_mode().unwrap(), PAUSE);+ assert!(matches!(options.finish(), Err(Error::InvalidOption(n)) if n == "other"));+ }+
#[test]
fn rejects_invalid_options() {
for sql in [
diff --git a/pgdog/src/admin/parser.rs b/pgdog/src/admin/parser.rsindex 15e24ba14..bbec8d820 100644--- a/pgdog/src/admin/parser.rs+++ b/pgdog/src/admin/parser.rs@@ -36,6 +36,7 @@ pub(crate) enum ParseResult {
ShowClientMemory(ShowClientMemory),
ShowTableCopies(ShowTableCopies),
ShowReplicationSlots(ShowReplicationSlots),
+ ShowResharding(ShowResharding),
ShowSchemaSync(ShowSchemaSync),
Set(Set),
Ban(Ban),
@@ -87,6 +88,7 @@ impl ParseResult {
ShowClientMemory(show_client_memory) => show_client_memory.execute().await,
ShowTableCopies(show_table_copies) => show_table_copies.execute().await,
ShowReplicationSlots(cmd) => cmd.execute().await,
+ ShowResharding(cmd) => cmd.execute().await,
ShowSchemaSync(cmd) => cmd.execute().await,
Set(set) => set.execute().await,
Ban(ban) => ban.execute().await,
@@ -138,6 +140,7 @@ impl ParseResult {
ShowClientMemory(show_client_memory) => show_client_memory.name(),
ShowTableCopies(show_table_copies) => show_table_copies.name(),
ShowReplicationSlots(cmd) => cmd.name(),
+ ShowResharding(cmd) => cmd.name(),
ShowSchemaSync(cmd) => cmd.name(),
Set(set) => set.name(),
Ban(ban) => ban.name(),
@@ -194,6 +197,7 @@ impl Parser {
"replication_slots" => {
ParseResult::ShowReplicationSlots(ShowReplicationSlots::parse(sql)?)
}
+ "resharding" => ParseResult::ShowResharding(ShowResharding::parse(sql)?),
"schema_sync" => ParseResult::ShowSchemaSync(ShowSchemaSync::parse(sql)?),
"table_copies" => ParseResult::ShowTableCopies(ShowTableCopies::parse(sql)?),
"tasks" => ParseResult::ShowTasks(ShowTasks::parse(sql)?),
@@ -328,6 +332,7 @@ mod tests {
"SHOW REPLICATION_SLOTS",
ParseResult::ShowReplicationSlots(_)
);
+ assert_parses!("SHOW RESHARDING", ParseResult::ShowResharding(_));
assert_parses!("SHOW SCHEMA_SYNC", ParseResult::ShowSchemaSync(_));
assert_parses!("SHOW TABLE_COPIES", ParseResult::ShowTableCopies(_));
assert_parses!("SHOW TASKS", ParseResult::ShowTasks(_));
@@ -364,6 +369,30 @@ mod tests {
"REPLICATE source target publication",
ParseResult::Replicate(_)
);
+ assert_parses!(+ "RESHARD source target publication WITH (cleanup_on_cancel false)",+ ParseResult::Reshard(_)+ );+ assert_parses!(+ "COPY_DATA source target publication WITH (cleanup_on_cancel false)",+ ParseResult::CopyData(_)+ );+ assert_parses!(+ "REPLICATE source target publication slot WITH (cleanup_on_cancel false)",+ ParseResult::Replicate(_)+ );+ assert_parses!(+ "RESHARD source target publication WITH (abort)",+ ParseResult::Reshard(_)+ );+ assert_parses!(+ "COPY_DATA source target publication WITH (abort)",+ ParseResult::CopyData(_)+ );+ assert_parses!(+ "REPLICATE source target publication WITH (abort)",+ ParseResult::Replicate(_)+ );
assert_parses!("STOP_TASK 1", ParseResult::StopTask(_));
assert_parses!("CUTOVER", ParseResult::Cutover(_));
}
diff --git a/pgdog/src/admin/replicate.rs b/pgdog/src/admin/replicate.rsindex 72846545c..dcacecd47 100644--- a/pgdog/src/admin/replicate.rs+++ b/pgdog/src/admin/replicate.rs@@ -6,14 +6,18 @@ use crate::api::replication::ReplicationTask;
use crate::api::run_task;
use crate::api::schema_sync::SchemaSyncTask;
use crate::backend::replication::resharding_state::ReshardingState;
+use pgdog_stats::CommandKind;+use super::options::{Options, RunMode};
use super::prelude::*;
+use super::reshard::check_abort;
pub(crate) struct Replicate {
pub(crate) from_database: String,
pub(crate) to_database: String,
pub(crate) publication: String,
pub(crate) replication_slot: Option<String>,
+ pub(crate) mode: RunMode,
}
#[async_trait]
@@ -23,43 +27,51 @@ impl Command for Replicate {
}
fn parse(sql: &str) -> Result<Self, Error> {
- let parts = sql.split(" ").collect::<Vec<_>>();+ let (words, mut options) = Options::parse(sql)?;- match parts[..] {- ["replicate", from_database, to_database, publication] => Ok(Self {- from_database: from_database.to_owned(),- to_database: to_database.to_owned(),- publication: publication.to_owned(),- replication_slot: None,- }),+ let (from_database, to_database, publication, replication_slot) = match words[..] {+ ["replicate", from_database, to_database, publication] => {+ (from_database, to_database, publication, None)+ }
[
"replicate",
from_database,
to_database,
publication,
replication_slot,
- ] => Ok(Self {- from_database: from_database.to_owned(),- to_database: to_database.to_owned(),- publication: publication.to_owned(),- replication_slot: Some(replication_slot.to_owned()),- }),- _ => Err(Error::Syntax),- }+ ] => (+ from_database,+ to_database,+ publication,+ Some(replication_slot),+ ),+ _ => return Err(Error::Syntax),+ };+ let mode = options.run_mode()?;+ options.finish()?;++ Ok(Self {+ from_database: from_database.to_owned(),+ to_database: to_database.to_owned(),+ publication: publication.to_owned(),+ replication_slot: replication_slot.map(str::to_owned),+ mode,+ })
}
async fn execute(&self) -> Result<Vec<Message>, Error> {
- info!(- r#"replicate "{}" to "{}", publication="{}""#,- self.from_database, self.to_database, self.publication- );-
let state = ReshardingState::builder()
.source(&self.from_database)
.destination(&self.to_database)
.publication(&self.publication)
.maybe_replication_slot(self.replication_slot.clone())
+ .cleanup_on_cancel(self.mode.cleanup_on_cancel())
.build()?;
+ check_abort!(self, state, CommandKind::Replicate);+ info!(+ r#"replicate "{}" to "{}", publication="{}""#,+ self.from_database, self.to_database, self.publication+ );
let schema_sync = SchemaSyncTask::builder()
.publication_tables_only(false)
@@ -84,3 +96,38 @@ impl Command for Replicate {
])
}
}
++#[cfg(test)]+mod test {+ use super::*;++ #[test]+ fn parses_slot_and_run_mode() {+ for (sql, slot, mode) in [+ (+ "replicate a b pub",+ None,+ RunMode::Run {+ cleanup_on_cancel: true,+ },+ ),+ (+ "replicate a b pub slot with (cleanup_on_cancel false)",+ Some("slot"),+ RunMode::Run {+ cleanup_on_cancel: false,+ },+ ),+ ("replicate a b pub (abort)", None, RunMode::Abort),+ ] {+ let cmd = Replicate::parse(sql).unwrap();+ assert_eq!(cmd.publication, "pub", "{sql}");+ assert_eq!(cmd.replication_slot.as_deref(), slot, "{sql}");+ assert_eq!(cmd.mode, mode, "{sql}");+ }+ assert!(matches!(+ Replicate::parse("replicate a b pub (unknown)"),+ Err(Error::InvalidOption(_))+ ));+ }+}diff --git a/pgdog/src/admin/reshard.rs b/pgdog/src/admin/reshard.rsindex 5b9916b4b..2883bfae8 100644--- a/pgdog/src/admin/reshard.rs+++ b/pgdog/src/admin/reshard.rs@@ -2,10 +2,13 @@
use tracing::info;
+use crate::api::abort_resharding::AbortReshardingTask;
use crate::api::resharding::ReshardTask;
use crate::api::run_task;
use crate::backend::replication::resharding_state::ReshardingState;
+use pgdog_stats::CommandKind;+use super::options::{Options, RunMode};
use super::prelude::*;
pub(crate) struct Reshard {
@@ -13,6 +16,7 @@ pub(crate) struct Reshard {
pub(crate) to_database: String,
pub(crate) publication: String,
pub(crate) replication_slot: Option<String>,
+ pub(crate) mode: RunMode,
}
#[async_trait]
@@ -22,42 +26,51 @@ impl Command for Reshard {
}
fn parse(sql: &str) -> Result<Self, Error> {
- let parts = sql.split(" ").collect::<Vec<_>>();-- match parts[..] {- ["reshard", from_database, to_database, publication] => Ok(Self {- from_database: from_database.to_owned(),- to_database: to_database.to_owned(),- publication: publication.to_owned(),- replication_slot: None,- }),+ let (words, mut options) = Options::parse(sql)?;++ let (from_database, to_database, publication, replication_slot) = match words[..] {+ ["reshard", from_database, to_database, publication] => {+ (from_database, to_database, publication, None)+ }
[
"reshard",
from_database,
to_database,
publication,
replication_slot,
- ] => Ok(Self {- from_database: from_database.to_owned(),- to_database: to_database.to_owned(),- publication: publication.to_owned(),- replication_slot: Some(replication_slot.to_owned()),- }),- _ => Err(Error::Syntax),- }+ ] => (+ from_database,+ to_database,+ publication,+ Some(replication_slot),+ ),+ _ => return Err(Error::Syntax),+ };+ let mode = options.run_mode()?;+ options.finish()?;++ Ok(Self {+ from_database: from_database.to_owned(),+ to_database: to_database.to_owned(),+ publication: publication.to_owned(),+ replication_slot: replication_slot.map(str::to_owned),+ mode,+ })
}
async fn execute(&self) -> Result<Vec<Message>, Error> {
- info!(- r#"resharding "{}" to "{}", publication="{}""#,- self.from_database, self.to_database, self.publication- );
let state = ReshardingState::builder()
.source(&self.from_database)
.destination(&self.to_database)
.publication(&self.publication)
.maybe_replication_slot(self.replication_slot.clone())
+ .cleanup_on_cancel(self.mode.cleanup_on_cancel())
.build()?;
+ check_abort!(self, state, CommandKind::Reshard);+ info!(+ r#"resharding "{}" to "{}", publication="{}""#,+ self.from_database, self.to_database, self.publication+ );
let task_id = run_task(
ReshardTask::builder()
@@ -76,3 +89,65 @@ impl Command for Reshard {
])
}
}
++/// Return from the command with the abort task's reply when the command runs `WITH (abort)`.+macro_rules! check_abort {+ ($command:expr, $state:expr, $kind:expr) => {+ if $command.mode == $crate::admin::options::RunMode::Abort {+ return $crate::admin::reshard::abort($state, $kind, $command.replication_slot.clone())+ .await;+ }+ };+}+pub(super) use check_abort;++pub(super) async fn abort(+ state: ReshardingState,+ command: CommandKind,+ slot: Option<String>,+) -> Result<Vec<Message>, Error> {+ let task_id = run_task(AbortReshardingTask::checked(state, command, slot).await?).id();++ let mut dr = DataRow::new();+ dr.add(task_id.to_string());++ Ok(vec![+ RowDescription::new(&[Field::text("task_id")]).message(),+ dr.message(),+ ])+}++#[cfg(test)]+mod test {+ use super::*;++ #[test]+ fn parses_slot_and_run_mode() {+ for (sql, slot, mode) in [+ (+ "reshard a b pub",+ None,+ RunMode::Run {+ cleanup_on_cancel: true,+ },+ ),+ (+ "reshard a b pub slot with (cleanup_on_cancel false)",+ Some("slot"),+ RunMode::Run {+ cleanup_on_cancel: false,+ },+ ),+ ("reshard a b pub (abort)", None, RunMode::Abort),+ ] {+ let cmd = Reshard::parse(sql).unwrap();+ assert_eq!(cmd.publication, "pub", "{sql}");+ assert_eq!(cmd.replication_slot.as_deref(), slot, "{sql}");+ assert_eq!(cmd.mode, mode, "{sql}");+ }+ assert!(matches!(+ Reshard::parse("reshard a b pub (unknown)"),+ Err(Error::InvalidOption(_))+ ));+ }+}diff --git a/pgdog/src/admin/show_resharding.rs b/pgdog/src/admin/show_resharding.rsnew file mode 100644index 000000000..0287187a7--- /dev/null+++ b/pgdog/src/admin/show_resharding.rs@@ -0,0 +1,102 @@+use std::time::SystemTime;++use chrono::{DateTime, Local};++use crate::backend::replication::logical::run::scan::{+ ReshardingReport, ReshardingScan, RunState, scan,+};+use crate::util::format_time;++use super::prelude::*;++pub(crate) struct ShowResharding;++#[async_trait]+impl Command for ShowResharding {+ fn name(&self) -> String {+ "SHOW RESHARDING".into()+ }++ fn parse(_sql: &str) -> Result<Self, Error> {+ Ok(ShowResharding)+ }++ async fn execute(&self) -> Result<Vec<Message>, Error> {+ let rd = RowDescription::new(&[+ Field::text("source"),+ Field::text("destination"),+ Field::text("command"),+ Field::text("publication"),+ Field::text("stage"),+ Field::text("state"),+ Field::text("instance"),+ Field::text("task_id"),+ Field::text("replication_slot"),+ Field::text("started_at"),+ Field::text("resumed_at"),+ ]);+ let mut messages = vec![rd.message()];++ for found in scan().await {+ let row = match found {+ ReshardingScan::Found(report) => found_row(&report),+ ReshardingScan::Unreadable { key, error } => problem_row(+ Some(&key.databases.source),+ &key.databases.destination,+ Some(&key.publication),+ format!("unsupported: {error}"),+ ),+ ReshardingScan::Failed { database, error } => {+ problem_row(None, &database, None, format!("error: {error}"))+ }+ };+ messages.push(row.message());+ }++ Ok(messages)+ }+}++fn found_row(report: &ReshardingReport) -> DataRow {+ let record = &report.record;++ let mut row = DataRow::new();+ row.add(record.key.databases.source.as_str())+ .add(record.key.databases.destination.as_str())+ .add(record.command.to_string())+ .add(record.key.publication.as_str())+ .add(record.stage.to_string())+ .add(report.state.to_string())+ .add(record.owner.instance_id.to_string())+ .add(matches!(report.state, RunState::Running).then(|| record.owner.task_id.to_string()))+ .add(record.slot_name.as_str())+ .add(local_time(record.started_at))+ .add(record.resumed_at.map(local_time));+ row+}++fn problem_row(+ source: Option<&str>,+ destination: &str,+ publication: Option<&str>,+ state: String,+) -> DataRow {+ let none: Option<&str> = None;+ let mut row = DataRow::new();+ row.add(source)+ .add(destination)+ .add(none)+ .add(publication)+ .add(none)+ .add(state)+ .add(none)+ .add(none)+ .add(none)+ .add(none)+ .add(none);+ row+}++fn local_time(time: SystemTime) -> String {+ format_time(DateTime::<Local>::from(time))+}diff --git a/pgdog/src/api/abort_resharding.rs b/pgdog/src/api/abort_resharding.rsnew file mode 100644index 000000000..0a3978067--- /dev/null+++ b/pgdog/src/api/abort_resharding.rs@@ -0,0 +1,51 @@+use pgdog_stats::{AbortReshardingDefinition, CommandKind, TaskDefinition, TaskStatus};+use tracing::info;++use crate::api::Task;+use crate::api::task::TaskContext;+use crate::backend::replication::logical::Error;+use crate::backend::replication::logical::resharding_state::ReshardingState;++#[derive(Debug)]+pub(crate) struct AbortReshardingTask {+ state: ReshardingState,+ command: CommandKind,+ slot: Option<String>,+}++impl AbortReshardingTask {+ pub(crate) async fn checked(+ state: ReshardingState,+ command: CommandKind,+ slot: Option<String>,+ ) -> Result<Self, Error> {+ state.check_record(command, slot.as_deref()).await?;+ Ok(Self {+ state,+ command,+ slot,+ })+ }+}++impl Task for AbortReshardingTask {+ type Status = TaskStatus;+ type Output = ();+ type Error = Error;++ fn definition(&self) -> impl Into<TaskDefinition> {+ AbortReshardingDefinition {+ databases: self.state.databases(),+ command: self.command,+ }+ }++ async fn run(self, _ctx: TaskContext<Self>) -> Result<(), Error> {+ let record = self+ .state+ .abort_run(self.command, self.slot.as_deref())+ .await?;+ info!("resharding {} aborted", record.key);+ Ok(())+ }+}diff --git a/pgdog/src/api/copy_data.rs b/pgdog/src/api/copy_data.rsindex d4fa5fff7..45c457411 100644--- a/pgdog/src/api/copy_data.rs+++ b/pgdog/src/api/copy_data.rs@@ -13,6 +13,7 @@ use crate::backend::replication::data_sync::{
};
use crate::backend::replication::logical::Error;
use crate::backend::replication::logical::resharding_state::ReshardingState;
+use crate::backend::replication::logical::run::Stage;
use crate::backend::replication::publisher::{Table, resolve_resharding_replicas};
use crate::tasks;
use crate::util::safe_sleep;
@@ -73,6 +74,7 @@ impl Task for CopyDataTask {
ctx.set_status(CopyDataStage::CreatingSlots.into());
state.create_slots(&cancel).await?;
+ state.set_stage(Stage::CopyTables).await?;
let mut tables = state.tables();
ctx.set_status(CopyDataStatus {
diff --git a/pgdog/src/api/mod.rs b/pgdog/src/api/mod.rsindex 997117030..1e55f8af5 100644--- a/pgdog/src/api/mod.rs+++ b/pgdog/src/api/mod.rs@@ -10,6 +10,7 @@ use crate::backend::replication::logical::Error;
use crate::backend::schema::sync::SchemaSyncError;
use task::{TaskError, TaskStorage, TaskWaiter};
+pub(crate) mod abort_resharding;
pub(crate) mod copy_data;
pub(crate) mod ee;
pub(crate) mod replication;
@@ -37,11 +38,11 @@ pub(crate) fn run_task<T: Task + 'static>(task: T) -> TaskWaiter<T::Output, T::E
tasks_storage().run(task)
}
-/// Error returned by the API migration tasks: either an error from the+/// Error returned by the resharding task: either an error from the
/// replication/orchestrator machinery, or a child task's [`TaskError`]
/// (failure, cancellation, panic, or abandonment) surfaced to its parent.
#[derive(Debug, Display, Error, From)]
-pub(crate) enum MigrationError {+pub(crate) enum ReshardingError {
Replication(Error),
SchemaSync(SchemaSyncError),
Task(TaskError<Error>),
diff --git a/pgdog/src/api/replication.rs b/pgdog/src/api/replication.rsindex fedf6440e..27378c79b 100644--- a/pgdog/src/api/replication.rs+++ b/pgdog/src/api/replication.rs@@ -24,6 +24,7 @@ use crate::backend::replication::logical::publisher::replication_stream::{
};
use crate::backend::replication::logical::publisher::{Lsn, Permanent, ReplicationSlot, Table};
use crate::backend::replication::logical::resharding_state::ReshardingState;
+use crate::backend::replication::logical::run::{LockWatchdog, ReshardingCommand};
use crate::backend::replication::logical::subscriber::replication_origin::ReplicationOrigin;
use crate::backend::schema::sync::SchemaSyncError;
use crate::backend::{
@@ -87,6 +88,7 @@ impl Task for ReplicationTask {
ReplicationDefinition {
databases: self.state.databases(),
auto_cutover: self.cutover == CutoverMode::Auto,
+ cleanup_on_cancel: self.state.cleanup_on_cancel(),
}
}
@@ -168,24 +170,35 @@ impl<T: Task<Status: From<ReplicationStatus>>> Replication<T> {
/// and the operator ended the rollback window. The same signal during a
/// forward phase returns [`Error::ReplicationAborted`].
async fn run(mut self, ctx: &TaskContext<T>) -> Result<(), Error> {
+ let cancel = ctx.cancellation_token();+ let _watchdog = if self.state.run_store().is_some() {+ None+ } else {+ let command = ReshardingCommand::Replicate {+ auto_cutover: self.cutover == CutoverMode::Auto,+ };+ let root_id = ctx.root_id();+ self.state.start_run(command, root_id).await?;+ Some(LockWatchdog::start(&self.state, root_id))+ };
let state = self.state.clone();
let result = self.replicate_loop(ctx, self.cutover).await;
self.resume_traffic().await;
- let cancelled = ctx.cancellation_token().is_cancelled();+ let cancelled = cancel.is_cancelled();
let stopped_in_rollback_window = cancelled
&& self.state.direction() == ReshardingDirection::Reverse
&& matches!(result, Err(Error::ReplicationAborted));
let completed = result.is_ok() || stopped_in_rollback_window;
let cleanup = if completed {
- state.drop_slots().await+ state.complete_run().await
} else {
- state.drop_slots_if_owned().await+ state.cancel_run().await
};
if let Err(err) = cleanup {
- warn!("failed to clean up replication slots: {err}");+ warn!("failed to clean up the resharding run: {err}");
}
if stopped_in_rollback_window {
@@ -462,11 +475,8 @@ impl<T: Task<Status: From<ReplicationStatus>>> Replication<T> {
warn!("cutting over");
ctx.set_status(ReplicationStatus::PreparingReverseReplication.into());
- // remove slots on the current source- Box::pin(self.state.drop_slots()).await?;- // create replication slots on the destination to allow rollback
self.state
- .create_reverse_slots(&ctx.cancellation_token())+ .prepare_reverse_replication(&ctx.cancellation_token())
.await?;
ctx.set_status(
match self.state.direction() {
diff --git a/pgdog/src/api/resharding.rs b/pgdog/src/api/resharding.rsindex a54825f0d..75858d875 100644--- a/pgdog/src/api/resharding.rs+++ b/pgdog/src/api/resharding.rs@@ -14,8 +14,9 @@ use crate::api::replication::{CutoverMode, ReplicationTask};
use crate::api::schema_sync::{SchemaSyncPhase, SchemaSyncTask};
use crate::api::synchronize_tables::SynchronizeTablesTask;
use crate::api::task::TaskContext;
-use crate::api::{MigrationError, Task, run_task};+use crate::api::{ReshardingError, Task, run_task};
use crate::backend::replication::logical::resharding_state::ReshardingState;
+use crate::backend::replication::logical::run::{LockWatchdog, ReshardingCommand};
use crate::config::config;
use pgdog_stats::{ReshardDefinition, ReshardStatus, TaskDefinition};
@@ -48,7 +49,7 @@ pub(crate) struct ReshardTask {
impl Task for ReshardTask {
type Status = ReshardStatus;
type Output = ();
- type Error = MigrationError;+ type Error = ReshardingError;
fn cancel_timeout() -> Duration {
Duration::from_secs(60)
@@ -61,40 +62,47 @@ impl Task for ReshardTask {
replicate_only: self.replicate_only,
sync_only: self.sync_only,
auto_cutover: self.auto_cutover,
+ cleanup_on_cancel: self.state.cleanup_on_cancel(),
}
}
- async fn run(self, ctx: TaskContext<Self>) -> Result<(), MigrationError> {+ async fn run(self, ctx: TaskContext<Self>) -> Result<(), ReshardingError> {
let cancel = ctx.cancellation_token();
let cutover = self.cutover_mode();
+ let command = self.command();
let mut state = self.state;
+ let root_id = ctx.root_id();+ state.start_run(command, root_id).await?;+ let _watchdog = LockWatchdog::start(&state, root_id);
let schema_sync = SchemaSyncTask::builder()
.publication_tables_only(self.tables_only)
.databases(state.databases())
.publication(state.publication.clone());
- // Pre-data schema sync, unless skipped. It runs before any replication- // slots exist, so it stays outside the cleanup guard below.- if !self.skip_schema_sync {- ctx.set_status(ReshardStatus::SchemaSync);- ctx.run(- schema_sync- .clone()- .phase(SchemaSyncPhase::Pre)- .ignore_errors(true)- .build(),- )- .await?;+ let mut copied = false;+ let mut result: Result<(), ReshardingError> = Box::pin(async {+ // Pre-data schema sync, unless skipped. It runs inside the cleanup+ // guard below, so a failure here also reaches the cleanup.+ if !self.skip_schema_sync {+ ctx.set_status(ReshardStatus::SchemaSync);+ ctx.run(+ schema_sync+ .clone()+ .phase(SchemaSyncPhase::Pre)+ .ignore_errors(true)+ .build(),+ )+ .await?;- // The pre-data sync changed the destination's schema, so its pools- // reloaded. `SchemaSync::reload_destination` refreshes only its own- // cluster refs, so the state still holds stale ones.- state.reload()?;- }+ // The pre-data sync changed the destination's schema, so its pools+ // reloaded. `SchemaSync::reload_destination` refreshes only its own+ // cluster refs, so the state still holds stale ones.+ state.reload()?;+ }++ // Replicate-only continues an earlier copy.+ copied = self.replicate_only;- // Replicate-only continues an earlier copy.- let mut copied = self.replicate_only;- let result: Result<(), MigrationError> = Box::pin(async {
// Copy the data, unless replicate-only.
if !self.replicate_only {
ctx.set_status(ReshardStatus::SyncingData);
@@ -161,19 +169,22 @@ impl Task for ReshardTask {
})
.await;
- if result.is_err() || cancel.is_cancelled() {- // on error or cancellation make sure we drop the slots,- // but only the slots we're actually created during this run.- // If the slots were created outside or by calling pgdog- // separate runs, we leave the slots.- if let Err(err) = state.drop_slots_if_owned().await {- warn!("failed to clean up replication slots after migration: {err}");+ let stopped = result.is_err() || cancel.is_cancelled();+ if stopped {+ // With cleanup_on_cancel true, drop the slots this run created and+ // all its origins, then delete the row. With false, keep them and+ // pause the run. A live run keeps the slots that existed before it.+ if let Err(err) = state.cancel_run().await {+ warn!("failed to clean up the resharding run: {err}");
}
} else {
// on success we either already removed slots on replication,
// or replication was not called and we should leave the
// slots for the future calls
state.detach_slots();
+ if let Err(err) = state.finish_run().await {+ result = Err(err.into());+ }
}
if self.tables_only && copied {
@@ -207,4 +218,14 @@ impl ReshardTask {
CutoverMode::Manual
}
}
++ fn command(&self) -> ReshardingCommand {+ ReshardingCommand::Reshard {+ skip_schema_sync: self.skip_schema_sync,+ replicate_only: self.replicate_only,+ sync_only: self.sync_only,+ auto_cutover: self.auto_cutover,+ tables_only: self.tables_only,+ }+ }
}
diff --git a/pgdog/src/backend/replication/logical/error.rs b/pgdog/src/backend/replication/logical/error.rsindex 147013450..9e09a6626 100644--- a/pgdog/src/backend/replication/logical/error.rs+++ b/pgdog/src/backend/replication/logical/error.rs@@ -10,6 +10,30 @@ use crate::{
net::ErrorResponse,
};
+use super::run::snapshot::ConfigMismatch;+use pgdog_stats::CommandKind;++fn abort_hint(command: CommandKind) -> &'static str {+ match command {+ CommandKind::Reshard => "abort it with WITH (abort) or data-sync --abort",+ CommandKind::Replicate => "abort it with REPLICATE ... WITH (abort)",+ CommandKind::Unknown => "abort it with the command that started it, WITH (abort)",+ }+}++#[derive(Debug, Display, Error)]+pub(crate) enum RecordMismatch {+ #[display("resharding command is \"{stored}\", not \"{given}\"")]+ Command { stored: String, given: String },+ #[display("resharding slot is \"{stored}\", not \"{given}\"")]+ Slot { stored: String, given: String },+ #[display(+ "sharding config changed since the resharding started: {}",+ _0.iter().map(ToString::to_string).collect::<Vec<_>>().join(", ")+ )]+ Config(#[error(ignore)] Vec<ConfigMismatch>),+}+
/// The kind of validation failure, decoupled from which table it occurred on.
#[derive(Debug, Display)]
pub(crate) enum TableValidationErrorKind {
@@ -144,6 +168,9 @@ pub(crate) enum Error {
#[error("replication slot \"{0}\" doesn't exist, but it should")]
MissingReplicationSlot(String),
+ #[error("replication origin {0} doesn't exist, but it should")]+ MissingReplicationOrigin(String),+
#[error("parse int")]
ParseInt(#[from] ParseIntError),
@@ -273,6 +300,33 @@ pub(crate) enum Error {
#[source]
source: Box<Error>,
},
++ #[error("resharding {key} is owned by another session ({owner}); stop it with STOP_TASK first")]+ ReshardingLocked { key: String, owner: String },++ #[error("unsupported resharding state: {0}")]+ ReshardingFormat(String),++ #[error("resharding {key} is unfinished at stage {stage}; {}", abort_hint(*command))]+ ReshardingExists {+ key: String,+ stage: String,+ command: CommandKind,+ },++ #[error("only some replication slots \"{0}_<shard>\" exist; create all of them or none")]+ ReshardingSlotsPartial(String),++ #[error(+ "resharding {key} stopped at stage {stage} and cannot resume; abort it with WITH (abort) and start again"+ )]+ ReshardingNotResumable { key: String, stage: String },++ #[error("no unfinished resharding {0}")]+ NoResharding(String),++ #[error(transparent)]+ RecordMismatch(#[from] RecordMismatch),
}
impl From<ErrorResponse> for Error {
diff --git a/pgdog/src/backend/replication/logical/mod.rs b/pgdog/src/backend/replication/logical/mod.rsindex 1a3f41f93..0ecf4e047 100644--- a/pgdog/src/backend/replication/logical/mod.rs+++ b/pgdog/src/backend/replication/logical/mod.rs@@ -3,6 +3,7 @@ pub(crate) mod data_sync;
pub(crate) mod error;
pub(crate) mod publisher;
pub(crate) mod resharding_state;
+pub(crate) mod run;
pub(crate) mod schema_sync;
pub(crate) mod subscriber;
pub(crate) mod tables_sync;
diff --git a/pgdog/src/backend/replication/logical/publisher/queries.rs b/pgdog/src/backend/replication/logical/publisher/queries.rsindex 75abb5a06..3a7fa7b27 100644--- a/pgdog/src/backend/replication/logical/publisher/queries.rs+++ b/pgdog/src/backend/replication/logical/publisher/queries.rs@@ -40,6 +40,7 @@ JOIN (
LEFT JOIN pg_inherits i ON i.inhrelid = c.oid -- only present if c is a child partition
LEFT JOIN pg_class p ON p.oid = i.inhparent -- immediate parent partitioned table
LEFT JOIN pg_namespace pn ON pn.oid = p.relnamespace
+WHERE NOT (n.nspname = 'pgdog' AND c.relname = 'resharding')
ORDER BY n.nspname, c.relname;";
/// Table included in a publication.
@@ -409,6 +410,30 @@ mod test {
server.execute("ROLLBACK").await.unwrap();
}
+ #[tokio::test]+ async fn test_publication_tables_exclude_resharding_state_table() {+ let mut server = test_server().await;++ server.execute("BEGIN").await.unwrap();+ for query in [+ "SELECT pg_advisory_xact_lock(hashtext('pgdog.resharding'))",+ "CREATE SCHEMA IF NOT EXISTS pgdog",+ "CREATE TABLE IF NOT EXISTS pgdog.resharding (id UUID PRIMARY KEY)",+ "CREATE TABLE users_logical_pub_state (id BIGINT PRIMARY KEY)",+ "CREATE PUBLICATION users_logical_pub_state+ FOR TABLE users_logical_pub_state, pgdog.resharding",+ ] {+ server.execute(query).await.unwrap();+ }++ let tables = PublicationTable::load("users_logical_pub_state", &mut server)+ .await+ .unwrap();+ let names: Vec<_> = tables.iter().map(|table| table.name.as_str()).collect();+ assert_eq!(names, ["users_logical_pub_state"]);+ server.execute("ROLLBACK").await.unwrap();+ }+
/// Table with no unique index: must appear in missing set.
#[tokio::test]
async fn test_has_unique_index_no_index() {
diff --git a/pgdog/src/backend/replication/logical/publisher/replication_slot.rs b/pgdog/src/backend/replication/logical/publisher/replication_slot.rsindex e9dd4c348..6ed2a8687 100644--- a/pgdog/src/backend/replication/logical/publisher/replication_slot.rs+++ b/pgdog/src/backend/replication/logical/publisher/replication_slot.rs@@ -119,6 +119,10 @@ impl<K> Deref for ReplicationSlot<K> {
}
}
+pub(crate) fn permanent_slot_name(base: &str, shard: usize) -> String {+ format!("{base}_{shard}")+}+
impl ReplicationSlot<Permanent> {
/// Create new [`Permanent`] slot that should be managed explicitly.
pub(crate) fn new_permanent(
@@ -128,7 +132,7 @@ impl ReplicationSlot<Permanent> {
shard: usize,
) -> Self {
let name = name.unwrap_or(format!("__pgdog_repl_{}", random_string(18).to_lowercase()));
- let name = format!("{}_{}", name, shard);+ let name = permanent_slot_name(&name, shard);
Self::wrap(ReplicationSlotInner {
address: address.clone(),
@@ -170,7 +174,10 @@ impl ReplicationSlot<Permanent> {
pub(crate) async fn reuse_or_create(&self) -> Result<(), Error> {
match self.verify_exists().await {
- Err(Error::MissingReplicationSlot(_)) => self.create().await,+ Err(Error::MissingReplicationSlot(_)) => {+ self.mark_owned();+ self.create().await+ }
result => result,
}
}
@@ -419,16 +426,21 @@ impl ReplicationSlotInner {
self.remove_on_drop.store(false, Ordering::Release);
}
- /// Whether the slot was already on the backend, so this process reused- /// it instead of creating it.+ /// Whether cleanup must keep the slot: it was on the backend before the+ /// run, so the run does not own it.
pub(crate) fn existing(&self) -> bool {
self.existing.load(Ordering::Acquire)
}
- fn mark_existing(&self) {+ pub(crate) fn mark_existing(&self) {
self.existing.store(true, Ordering::Release);
}
+ /// This run creates the slot, so cleanup may drop it.+ pub(crate) fn mark_owned(&self) {+ self.existing.store(false, Ordering::Release);+ }+
fn status_key(&self) -> String {
format!("{}@{}", self.name, self.address)
}
diff --git a/pgdog/src/backend/replication/logical/publisher/replication_stream.rs b/pgdog/src/backend/replication/logical/publisher/replication_stream.rsindex 47834f77d..28662908d 100644--- a/pgdog/src/backend/replication/logical/publisher/replication_stream.rs+++ b/pgdog/src/backend/replication/logical/publisher/replication_stream.rs@@ -396,7 +396,7 @@ mod tests {
.address(self.server.addr())
.shard(0)
.build();
- origin.recreate_origin().await?;+ origin.recreate().await?;
let mut guard = slot.get_existing().await?;
if self.replication.progress().replication_lag.is_some() {
return Err("lag was measured before replication started".into());
diff --git a/pgdog/src/backend/replication/logical/resharding_state.rs b/pgdog/src/backend/replication/logical/resharding_state.rsindex d32c559c9..b501ed600 100644--- a/pgdog/src/backend/replication/logical/resharding_state.rs+++ b/pgdog/src/backend/replication/logical/resharding_state.rs@@ -1,17 +1,28 @@
use futures::future::try_join_all;
use futures::prelude::future::join_all;
use parking_lot::Mutex;
-use pgdog_stats::{Databases, ReshardingDirection};-use std::{collections::HashMap, fmt::Display, sync::Arc, time::Duration};+use pgdog_stats::{CommandKind, Databases, ReshardingDirection, ReshardingId, TaskId};+use std::{+ collections::HashMap,+ fmt::Display,+ sync::{Arc, Weak},+ time::Duration,+};
use tokio::task::JoinHandle;
use tokio_util::sync::CancellationToken;
-use tracing::warn;--use super::publisher::{Permanent, ReplicationSlot, Table};+use tracing::{info, warn};++use super::publisher::{Permanent, ReplicationSlot, Table, permanent_slot_name};+use super::run::{+ ConfigSnapshot, FORMAT_VERSION, ReshardingCommand, ReshardingKey, ReshardingRecord,+ ReshardingStore, Stage, TaskOwner, existing_slots,+};+use super::subscriber::connect_address;
use super::subscriber::replication_origin::ReplicationOrigin;
use super::tables_sync::tables_sync;
use super::*;
-use crate::backend::pool::Request;+use crate::backend::Error as BackendError;+use crate::backend::pool::{Address, Request};
use crate::tasks;
use crate::util::safe_timeout;
use crate::{backend::Cluster, util::random_string};
@@ -32,25 +43,54 @@ pub(crate) struct ReshardingState {
inner: Arc<Mutex<ReshardingStateInner>>,
/// Slot name actually used: the supplied one or a generated one.
slot_name: String,
+ cleanup_on_cancel: bool,
}
-#[derive(Debug, bon::Builder)]+#[derive(Debug, Default)]
struct ReshardingStateInner {
- /// Name of the publication.- #[builder(into)]- publication: String,
/// Shard -> Tables mapping.
- #[builder(skip)]
tables: HashMap<usize, Vec<Table>>,
/// Shard -> Replication slot mapping.
- #[builder(skip)]
slots: HashMap<usize, ReplicationSlot<Permanent>>,
- #[builder(skip)]
origins: Origins,
- #[builder(into)]- slot_name: String,- #[builder(skip)]
direction: ReshardingDirection,
+ run: Option<RecordedRun>,+}++#[derive(Debug)]+struct RecordedRun {+ store: Arc<ReshardingStore>,+ id: ReshardingId,+ stage: Stage,+ resumed: bool,+ keep_on_stop: bool,+}++impl RecordedRun {+ fn row(&self) -> RunRow {+ RunRow {+ store: self.store.clone(),+ id: self.id,+ }+ }++ fn into_row(self) -> RunRow {+ RunRow {+ store: self.store,+ id: self.id,+ }+ }+}++#[derive(Debug)]+struct RunRow {+ store: Arc<ReshardingStore>,+ id: ReshardingId,+}++struct AcceptedRow {+ id: ReshardingId,+ stage: Stage,
}
type Slots = HashMap<usize, ReplicationSlot<Permanent>>;
@@ -74,12 +114,30 @@ impl ReshardingStateInner {
fn take_origins(&mut self) -> Origins {
std::mem::take(&mut self.origins)
}
++ fn keeps_slots(&self) -> bool {+ self.run+ .as_ref()+ .is_some_and(|run| run.stage > Stage::Initializing && run.keep_on_stop)+ }
}
impl Drop for ReshardingStateInner {
fn drop(&mut self) {
- remove_slots(self.take_owned_slots());- remove_origins(self.take_origins());+ if self.keeps_slots() {+ return;+ }+ if self.slots.is_empty() && self.origins.is_empty() && self.run.is_none() {+ return;+ }+ let slots = self.take_owned_slots();+ let origins = self.take_origins();+ let run = self.run.take().map(RecordedRun::into_row);+ tasks::spawn("resharding run cleanup", async move {+ if let Err(err) = close(run, slots, origins).await {+ warn!("resharding cleanup failed: {err}");+ }+ });
}
}
@@ -91,21 +149,18 @@ impl ReshardingState {
destination: &str,
publication: &str,
replication_slot: Option<String>,
+ #[builder(default = true)] cleanup_on_cancel: bool,
) -> Result<Self, Error> {
let slot_name = replication_slot
.unwrap_or(format!("__pgdog_repl_{}", random_string(19).to_lowercase()));
- let inner = ReshardingStateInner::builder()- .publication(publication)- .slot_name(slot_name.clone())- .build();-
Ok(Self {
source: databases().schema_owner(source)?,
destination: databases().schema_owner(destination)?,
publication: publication.to_owned(),
- inner: Arc::new(Mutex::new(inner)),+ inner: Arc::new(Mutex::new(ReshardingStateInner::default())),
slot_name,
+ cleanup_on_cancel,
})
}
}
@@ -213,51 +268,78 @@ impl ReshardingState {
self.create_slots_on(&self.source, cancel).await
}
- /// Create the slots a rollback would replicate from, on the destination.- pub(crate) async fn create_reverse_slots(+ /// End the run, then create the slots a rollback would replicate from, on the destination.+ pub(crate) async fn prepare_reverse_replication(
&self,
cancel: &CancellationToken,
) -> Result<(), Error> {
+ if let Err(err) = Box::pin(self.drop_slots()).await {+ warn!(+ "resharding {}: forward slot cleanup failed, cutover continues: {err}; drop the remaining slots \"{}_<shard>\" on the source and origins \"__pgdog_origin_{}_<shard>\" on the destination by hand",+ self.run_key(),+ self.slot_name,+ self.slot_name,+ );+ }+ self.finish_run().await?;
self.create_slots_on(&self.destination, cancel).await
}
+ fn loads_run_objects(&self) -> bool {+ self.inner+ .lock()+ .run+ .as_ref()+ .is_some_and(|run| run.resumed)+ }+
async fn create_slots_on(
&self,
cluster: &Cluster,
cancel: &CancellationToken,
) -> Result<(), Error> {
- let (publication, slot_name) = {- let inner = self.inner.lock();- if !inner.slots.is_empty() {- return Err(Error::SlotsAlreadyCreated(inner.slot_name.clone()));- }- (inner.publication.clone(), inner.slot_name.clone())- };+ let load = self.loads_run_objects();+ if !self.inner.lock().slots.is_empty() {+ return Err(Error::SlotsAlreadyCreated(self.slot_name.clone()));+ }+ let (publication, slot_name) = (&self.publication, &self.slot_name);
if cancel.is_cancelled() {
return Err(Error::ReplicationAborted);
}
- join_all(cluster.shards().iter().enumerate().map(|(number, shard)| {- let publication = publication.clone();- let slot_name = slot_name.clone();-- async move {- let addr = shard.primary(&Request::default()).await?.addr().clone();-- let slot =- ReplicationSlot::new_permanent(&publication, &addr, Some(slot_name), number);- self.inner.lock().slots.insert(number, slot.clone());-- slot.reuse_or_create().await- }- }))+ join_all(+ cluster+ .shards()+ .iter()+ .enumerate()+ .map(|(number, shard)| async move {+ let addr = shard.primary(&Request::default()).await?.addr().clone();++ let slot = ReplicationSlot::new_permanent(+ publication,+ &addr,+ Some(slot_name.clone()),+ number,+ );+ slot.mark_existing();+ self.inner.lock().slots.insert(number, slot.clone());++ if load {+ slot.verify_exists().await?;+ } else {+ slot.reuse_or_create().await?;+ }+ Ok(())+ }),+ )
.await
.into_iter()
.collect()
}
async fn create_origins(&self) -> Result<(), Error> {
+ let keep_progress = self.loads_run_objects();
let slots: Vec<(usize, String)> = self
.inner
.lock()
@@ -266,29 +348,41 @@ impl ReshardingState {
.map(|(number, slot)| (*number, slot.name().to_owned()))
.collect();
- let mut addresses = vec![];- for shard in self.destination.shards() {- addresses.push(shard.primary(&Request::default()).await?.addr().clone());+ for (number, origins) in self.origins_for(slots).await? {+ self.inner.lock().origins.insert(number, origins.clone());+ if keep_progress {+ try_join_all(origins.iter().map(ReplicationOrigin::ensure_exists)).await?;+ } else {+ try_join_all(origins.iter().map(ReplicationOrigin::recreate)).await?;+ }
}
- for (number, slot_name) in slots {- let origins: Vec<ReplicationOrigin> = addresses- .iter()- .enumerate()- .map(|(shard, address)| {- ReplicationOrigin::builder()- .name(&slot_name)- .address(address)- .shard(shard)- .build()- })- .collect();- self.inner.lock().origins.insert(number, origins.clone());+ Ok(())+ }- try_join_all(origins.iter().map(ReplicationOrigin::recreate_origin)).await?;+ async fn origins_for(&self, slots: Vec<(usize, String)>) -> Result<Origins, Error> {+ let mut addresses = vec![];+ for shard in self.destination.shards() {+ addresses.push(shard.primary(&Request::default()).await?.addr().clone());
}
- Ok(())+ Ok(slots+ .into_iter()+ .map(|(number, slot_name)| {+ let origins = addresses+ .iter()+ .enumerate()+ .map(|(shard, address)| {+ ReplicationOrigin::builder()+ .name(&slot_name)+ .address(address)+ .shard(shard)+ .build()+ })+ .collect();+ (number, origins)+ })+ .collect())
}
pub(crate) async fn prepare_replication(
@@ -313,7 +407,7 @@ impl ReshardingState {
self.create_origins().await?;
}
- Ok(())+ self.set_stage(Stage::Replication).await
}
/// Remove every slot this state holds, and its origins.
@@ -322,22 +416,7 @@ impl ReshardingState {
let mut inner = self.inner.lock();
(inner.take_slots(), inner.take_origins())
};
-- let slots = remove_slots(slots);- let origins = remove_origins(origins);- slots.await?.and(origins.await?)- }-- /// Remove only the slots this state created, and every origin.- pub(crate) async fn drop_slots_if_owned(&self) -> Result<(), Error> {- let (slots, origins) = {- let mut inner = self.inner.lock();- (inner.take_owned_slots(), inner.take_origins())- };-- let slots = remove_slots(slots);- let origins = remove_origins(origins);- slots.await?.and(origins.await?)+ remove_objects(slots, origins).await
}
/// Forget the slots without removing them from the backend, and remove
@@ -350,15 +429,375 @@ impl ReshardingState {
};
remove_origins(origins);
}
++ pub(crate) fn cleanup_on_cancel(&self) -> bool {+ self.cleanup_on_cancel+ }++ pub(crate) async fn start_run(+ &self,+ command: ReshardingCommand,+ task_id: TaskId,+ ) -> Result<(), Error> {+ let store = self.lock_run().await?;+ let row = Box::pin(self.insert_run(&store, command, task_id)).await;+ Box::pin(self.accept_run(store, row, false)).await+ }++ #[allow(dead_code)]+ pub(crate) async fn resume_run(+ &self,+ command: ReshardingCommand,+ task_id: TaskId,+ ) -> Result<(), Error> {+ let store = self.lock_run().await?;+ let row = Box::pin(self.load_run(&store, command, task_id)).await;+ Box::pin(self.accept_run(store, row, true)).await+ }++ async fn lock_run(&self) -> Result<ReshardingStore, Error> {+ ReshardingStore::open(&self.state_address().await?, self.run_key()).await+ }++ async fn insert_run(+ &self,+ store: &ReshardingStore,+ command: ReshardingCommand,+ task_id: TaskId,+ ) -> Result<AcceptedRow, Error> {+ let key = store.key();+ if let Some(record) = store.load().await? {+ return Err(Error::ReshardingExists {+ key: key.to_string(),+ stage: record.stage.to_string(),+ command: record.command.kind(),+ });+ }+ let existing = existing_slots(&self.source, &self.publication, &self.slot_name)+ .await?+ .len();+ if existing != 0 && existing != self.source.shards().len() {+ return Err(Error::ReshardingSlotsPartial(self.slot_name.clone()));+ }+ let id = ReshardingId::generate();+ let stage = Stage::Initializing;+ store+ .insert(&ReshardingRecord {+ id,+ key: key.clone(),+ command,+ slot_name: self.slot_name.clone(),+ stage,+ config: self.config_snapshot()?,+ owner: TaskOwner::current(task_id),+ format_version: FORMAT_VERSION,+ started_at: std::time::SystemTime::now(),+ resumed_at: None,+ })+ .await?;+ Ok(AcceptedRow { id, stage })+ }++ async fn load_run(+ &self,+ store: &ReshardingStore,+ command: ReshardingCommand,+ task_id: TaskId,+ ) -> Result<AcceptedRow, Error> {+ let key = store.key();+ let record = store+ .load()+ .await?+ .ok_or_else(|| Error::NoResharding(key.to_string()))?;+ self.check_resumed(&record, command, &self.config_snapshot()?)?;+ if record.stage < Stage::Replication {+ return Err(Error::ReshardingNotResumable {+ key: key.to_string(),+ stage: record.stage.to_string(),+ });+ }+ store+ .resume(+ record.id,+ &TaskOwner::current(task_id),+ std::time::SystemTime::now(),+ )+ .await?;+ Ok(AcceptedRow {+ id: record.id,+ stage: record.stage,+ })+ }++ async fn accept_run(+ &self,+ store: ReshardingStore,+ row: Result<AcceptedRow, Error>,+ resumed: bool,+ ) -> Result<(), Error> {+ let key = store.key();+ let AcceptedRow { id, stage } = match row {+ Ok(row) => row,+ Err(err) => {+ warn!("resharding {key}: refused to start: {err}");+ unlock(&store).await;+ return Err(err);+ }+ };+ info!(+ "resharding {key}: {} (stage {stage})",+ if resumed { "resumed" } else { "started" },+ );+ run::scan::check_other_runs(store.address(), key).await;++ self.inner.lock().run = Some(RecordedRun {+ store: Arc::new(store),+ id,+ stage,+ resumed,+ keep_on_stop: resumed || !self.cleanup_on_cancel,+ });+ Ok(())+ }++ fn config_snapshot(&self) -> Result<ConfigSnapshot, Error> {+ ConfigSnapshot::capture(+ &self.source,+ &self.destination,+ &crate::config::config().config,+ )+ }++ fn check_resumed(+ &self,+ record: &ReshardingRecord,+ command: ReshardingCommand,+ config: &ConfigSnapshot,+ ) -> Result<(), RecordMismatch> {+ if record.command != command {+ return Err(RecordMismatch::Command {+ stored: command_text(&record.command),+ given: command_text(&command),+ });+ }+ if record.slot_name != self.slot_name {+ return Err(RecordMismatch::Slot {+ stored: record.slot_name.clone(),+ given: self.slot_name.clone(),+ });+ }+ record+ .config+ .validate_equal(config)+ .map_err(RecordMismatch::Config)+ }++ pub(crate) fn run_store(&self) -> Option<Weak<ReshardingStore>> {+ self.inner+ .lock()+ .run+ .as_ref()+ .map(|run| Arc::downgrade(&run.store))+ }++ pub(crate) async fn set_stage(&self, stage: Stage) -> Result<(), Error> {+ let row = match &self.inner.lock().run {+ Some(run) if stage > run.stage => run.row(),+ _ => return Ok(()),+ };+ row.store.set_stage(row.id, stage).await?;+ if let Some(run) = self.inner.lock().run.as_mut() {+ run.stage = stage;+ }+ Ok(())+ }++ pub(crate) async fn cancel_run(&self) -> Result<(), Error> {+ let (keep, slots, origins, run) = {+ let mut inner = self.inner.lock();+ let keep = inner.keeps_slots();+ let slots = if keep {+ inner.take_slots()+ } else {+ inner.take_owned_slots()+ };+ (keep, slots, inner.take_origins(), inner.run.take())+ };+ if keep {+ if let Some(RecordedRun { store, .. }) = run {+ unlock(&store).await;+ info!(+ "resharding {} paused, replication slots and origins kept",+ store.key()+ );+ }+ return Ok(());+ }+ close(run.map(RecordedRun::into_row), slots, origins).await+ }++ pub(crate) async fn complete_run(&self) -> Result<(), Error> {+ let (slots, origins, run) = {+ let mut inner = self.inner.lock();+ (+ inner.take_slots(),+ inner.take_origins(),+ inner.run.take().map(RecordedRun::into_row),+ )+ };+ close(run, slots, origins).await+ }++ pub(crate) async fn finish_run(&self) -> Result<(), Error> {+ let row = self.inner.lock().run.as_ref().map(RecordedRun::row);+ let Some(row) = row else {+ return Ok(());+ };+ delete_row(&row.store, row.id).await?;+ self.inner.lock().run = None;+ unlock(&row.store).await;+ Ok(())+ }++ pub(crate) fn run_key(&self) -> ReshardingKey {+ ReshardingKey::new(self.databases(), &self.publication)+ }++ async fn state_address(&self) -> Result<Address, Error> {+ run::state_address(&self.destination).await+ }++ pub(crate) async fn check_record(+ &self,+ command: CommandKind,+ slot: Option<&str>,+ ) -> Result<ReshardingRecord, Error> {+ let key = self.run_key();+ let mut server = connect_address(&self.state_address().await?).await?;+ let record = ReshardingStore::read(&mut server, &key)+ .await?+ .ok_or_else(|| Error::NoResharding(key.to_string()))?;+ record_matches(&record, command, slot)?;+ if let Some(owner) = ReshardingStore::lock_owner(&mut server, &key).await? {+ return Err(Error::ReshardingLocked {+ key: key.to_string(),+ owner,+ });+ }+ Ok(record)+ }++ pub(crate) async fn abort_run(+ &self,+ command: CommandKind,+ slot: Option<&str>,+ ) -> Result<ReshardingRecord, Error> {+ let store = self.lock_run().await?;+ let result = async {+ let key = store.key();+ let record = store+ .load()+ .await?+ .ok_or_else(|| Error::NoResharding(key.to_string()))?;+ record_matches(&record, command, slot)?;+ let slots = existing_slots(&self.source, &self.publication, &record.slot_name).await?;+ let names = (0..self.source.shards().len())+ .map(|number| (number, permanent_slot_name(&record.slot_name, number)))+ .collect();+ let origins = self.origins_for(names).await?;+ remove_run(&store, record.id, slots, origins).await?;+ Ok(record)+ }+ .await;+ unlock(&store).await;+ result+ }+}++fn command_text(command: &ReshardingCommand) -> String {+ serde_json::to_string(command).expect("command is valid JSON")+}++fn record_matches(+ record: &ReshardingRecord,+ command: CommandKind,+ slot: Option<&str>,+) -> Result<(), RecordMismatch> {+ if record.command.kind() != command {+ return Err(RecordMismatch::Command {+ stored: record.command.to_string(),+ given: command.to_string(),+ });+ }+ match slot {+ Some(slot) if slot != record.slot_name => Err(RecordMismatch::Slot {+ stored: record.slot_name.clone(),+ given: slot.to_owned(),+ }),+ _ => Ok(()),+ }+}++async fn unlock(store: &ReshardingStore) {+ if let Err(err) = store.unlock().await {+ warn!(+ "resharding {}: failed to release the lock: {err}",+ store.key()+ );+ }+}++async fn remove_objects(slots: Slots, origins: Origins) -> Result<(), Error> {+ let slots = remove_slots(slots);+ let origins = remove_origins(origins);+ slots.await?.and(origins.await?)+}++async fn remove_run(+ store: &ReshardingStore,+ id: ReshardingId,+ slots: Slots,+ origins: Origins,+) -> Result<(), Error> {+ remove_objects(slots, origins).await?;+ delete_row(store, id).await+}++async fn close(run: Option<RunRow>, slots: Slots, origins: Origins) -> Result<(), Error> {+ let Some(run) = run else {+ return remove_objects(slots, origins).await;+ };+ let result = remove_run(&run.store, run.id, slots, origins).await;+ unlock(&run.store).await;+ result+}++async fn delete_row(store: &ReshardingStore, id: ReshardingId) -> Result<(), Error> {+ if let Err(err) = store.delete(id).await {+ warn!(+ "resharding {}: state connection failed, reconnecting: {err}",+ store.key()+ );+ store.reconnect().await?;+ store.delete(id).await?;+ }+ Ok(())
}
fn remove_slots(slots: Slots) -> JoinHandle<Result<(), Error>> {
tasks::spawn("replication slot cleanup", async move {
let removals = slots.into_values().map(|slot| async move {
let slot_name = slot.name().to_owned();
- safe_timeout(SLOT_DROP_TIMEOUT, slot.drop_slot())- .await- .unwrap_or(Err(Error::SlotDropTimeout(slot_name)))+ match safe_timeout(SLOT_DROP_TIMEOUT, slot.drop_slot()).await {+ Ok(Err(Error::Backend(BackendError::ExecutionError(error))))+ if error.code == "42704" =>+ {+ warn!("replication slot \"{slot_name}\" was already gone");+ Ok(())+ }+ Ok(result) => result,+ Err(_) => Err(Error::SlotDropTimeout(slot_name)),+ }
});
let result: Result<Vec<_>, _> = join_all(removals).await.into_iter().collect();
@@ -399,10 +838,11 @@ impl Display for ReshardingState {
#[cfg(test)]
mod test {
- use super::super::publisher::test::setup_publication;+ use super::super::publisher::test::{setup_publication, setup_publication_tables};
use super::*;
use crate::backend::Server;
use crate::config::config;
+ use pgdog_stats::TaskId;
async fn slot_count(server: &mut Server, name: &str) -> i64 {
let rows: Vec<i64> = server
@@ -425,10 +865,7 @@ mod test {
}
async fn inner_with_slots(source: &Cluster, slot_name: &str) -> ReshardingStateInner {
- let mut inner = ReshardingStateInner::builder()- .publication("publication_test")- .slot_name(slot_name)- .build();+ let mut inner = ReshardingStateInner::default();
for (number, shard) in source.shards().iter().enumerate() {
let addr = shard
@@ -453,17 +890,21 @@ mod test {
}
fn state_for(source: &Cluster, slot_name: &str) -> ReshardingState {
- let inner = ReshardingStateInner::builder()- .publication("publication_test")- .slot_name(slot_name)- .build();+ state_with_publication(source, slot_name, "publication_test")+ }+ fn state_with_publication(+ source: &Cluster,+ slot_name: &str,+ publication: &str,+ ) -> ReshardingState {
ReshardingState {
source: source.clone(),
destination: source.clone(),
- publication: "publication_test".into(),- inner: Arc::new(Mutex::new(inner)),+ publication: publication.into(),+ inner: Arc::new(Mutex::new(ReshardingStateInner::default())),
slot_name: slot_name.to_owned(),
+ cleanup_on_cancel: true,
}
}
@@ -660,4 +1101,1232 @@ mod test {
source.shutdown();
publication.cleanup().await;
}
++ async fn resharding_rows(server: &mut Server, name: &str) -> i64 {+ let rows: Vec<i64> = server+ .fetch_all(format!(+ "SELECT count(*) FROM pgdog.resharding WHERE slot_name = '{name}'"+ ))+ .await+ .unwrap();+ rows[0]+ }++ async fn remove_resumable(server: &mut Server, name: &str) {+ server+ .execute_checked(format!(+ "SELECT pg_drop_replication_slot(slot_name) FROM pg_replication_slots WHERE slot_name LIKE '{name}%'"+ ))+ .await+ .unwrap();+ server+ .execute_checked(format!(+ "SELECT pg_replication_origin_drop(roname) FROM pg_replication_origin WHERE roname LIKE '__pgdog_origin_{name}%'"+ ))+ .await+ .unwrap();+ server+ .execute_checked(format!(+ "DO $$ BEGIN IF to_regclass('pgdog.resharding') IS NOT NULL THEN \+ DELETE FROM pgdog.resharding WHERE slot_name = '{name}'; END IF; END $$"+ ))+ .await+ .unwrap();+ }++ #[tokio::test]+ async fn paused_resumable_run_keeps_slots_origins_and_record() {+ crate::logger();+ let mut publication = setup_publication().await;+ let name = "state_resumable_pause";+ let source = Cluster::new_test(&config());+ source.launch();+ let shards = source.shards().len() as i64;+ remove_resumable(&mut publication.server, name).await;++ let mut state = state_for(&source, name);+ state.cleanup_on_cancel = false;+ start(&state, 1).await.unwrap();+ assert!(!state.keeps_slots());+ assert_eq!(+ stored_stage(&mut publication.server, name).await,+ Stage::Initializing.to_string()+ );++ state+ .prepare_replication(&CancellationToken::new())+ .await+ .unwrap();+ assert!(state.keeps_slots());+ assert_eq!(resharding_rows(&mut publication.server, name).await, 1);+ state.cancel_run().await.unwrap();+ assert!(!state.keeps_slots());+ assert!(state.run_store().is_none());+ assert_paused_state_is_empty(&state);+ drop(state);++ assert_eq!(slot_count(&mut publication.server, name).await, shards);+ assert_eq!(+ origin_count(&mut publication.server, name).await,+ shards * shards+ );+ assert_eq!(resharding_rows(&mut publication.server, name).await, 1);++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ const REPLICATE: ReshardingCommand = ReshardingCommand::Replicate {+ auto_cutover: false,+ };++ fn keyed_state(source: &Cluster, name: &str, cleanup_on_cancel: bool) -> ReshardingState {+ let mut state = state_with_publication(source, name, &format!("{name}_pub"));+ state.cleanup_on_cancel = cleanup_on_cancel;+ state+ }++ impl ReshardingState {+ fn keeps_slots(&self) -> bool {+ self.inner.lock().keeps_slots()+ }+ }++ fn assert_paused_state_is_empty(state: &ReshardingState) {+ let inner = state.inner.lock();+ assert!(inner.slots.is_empty());+ assert!(inner.origins.is_empty());+ assert!(inner.run.is_none());+ }++ async fn terminate_state_backend(server: &mut Server, key: &ReshardingKey) {+ let killed: Vec<String> = server+ .fetch_all(format!(+ "SELECT pg_terminate_backend(pid)::text FROM pg_locks \+ WHERE locktype = 'advisory' AND objsubid = 2 \+ AND objid = (hashtext('{}')::bigint & 4294967295)::oid",+ lock_name(key)+ ))+ .await+ .unwrap();+ assert_eq!(killed, ["true"]);+ }++ fn lock_name(key: &ReshardingKey) -> String {+ format!(+ "{}/{}/{}",+ key.databases.source, key.databases.destination, key.publication+ )+ }++ async fn start(state: &ReshardingState, task_id: u64) -> Result<(), Error> {+ state.start_run(REPLICATE, TaskId::new(task_id)).await+ }++ async fn resume(state: &ReshardingState, task_id: u64) -> Result<(), Error> {+ state.resume_run(REPLICATE, TaskId::new(task_id)).await+ }++ async fn recreate_keyed_publication(server: &mut Server, name: &str, create: bool) {+ server+ .execute_checked(format!("DROP PUBLICATION IF EXISTS {name}_pub"))+ .await+ .unwrap();+ if create {+ server+ .execute_checked(format!(+ "CREATE PUBLICATION {name}_pub FOR TABLE publication_test_one"+ ))+ .await+ .unwrap();+ }+ }++ #[tokio::test]+ async fn finish_reconnects_after_state_connection_loss() {+ crate::logger();+ let mut publication = setup_publication().await;+ let name = "state_finish_reconnects";+ let source = Cluster::new_test(&config());+ source.launch();+ remove_resumable(&mut publication.server, name).await;+ recreate_keyed_publication(&mut publication.server, name, true).await;++ let state = keyed_state(&source, name, false);+ start(&state, 1).await.unwrap();+ state+ .prepare_replication(&CancellationToken::new())+ .await+ .unwrap();+ assert_eq!(resharding_rows(&mut publication.server, name).await, 1);++ terminate_state_backend(&mut publication.server, &state.run_key()).await;++ state.finish_run().await.unwrap();+ assert!(!state.keeps_slots());+ assert!(state.run_store().is_none());+ assert_eq!(resharding_rows(&mut publication.server, name).await, 0);+ drop(state);+ wait_until_removed(&mut publication.server, name).await;++ remove_resumable(&mut publication.server, name).await;+ recreate_keyed_publication(&mut publication.server, name, false).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn cancel_removes_the_record_when_a_slot_is_already_gone() {+ crate::logger();+ let mut publication = setup_publication().await;+ let name = "state_slot_gone";+ let source = Cluster::new_test(&config());+ source.launch();+ remove_resumable(&mut publication.server, name).await;+ recreate_keyed_publication(&mut publication.server, name, true).await;++ let state = keyed_state(&source, name, true);+ start(&state, 1).await.unwrap();+ state+ .prepare_replication(&CancellationToken::new())+ .await+ .unwrap();+ publication+ .server+ .execute_checked(format!("SELECT pg_drop_replication_slot('{name}_0')"))+ .await+ .unwrap();++ state.cancel_run().await.unwrap();++ assert_eq!(slot_count(&mut publication.server, name).await, 0);+ assert_eq!(origin_count(&mut publication.server, name).await, 0);+ assert_eq!(resharding_rows(&mut publication.server, name).await, 0);++ recreate_keyed_publication(&mut publication.server, name, false).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn other_unfinished_runs_do_not_block_a_new_run() {+ crate::logger();+ let mut publication = setup_publication().await;+ let paused_name = "state_other_paused";+ let new_name = "state_other_new";+ let source = Cluster::new_test(&config());+ source.launch();+ for name in [paused_name, new_name] {+ remove_resumable(&mut publication.server, name).await;+ }++ let paused = keyed_state(&source, paused_name, false);+ start(&paused, 1).await.unwrap();+ paused+ .create_slots(&CancellationToken::new())+ .await+ .unwrap();+ paused.set_stage(Stage::Replication).await.unwrap();+ paused.cancel_run().await.unwrap();++ let other = keyed_state(&source, new_name, false);+ start(&other, 3).await.unwrap();+ assert_eq!(resharding_rows(&mut publication.server, new_name).await, 1);+ other.create_slots(&CancellationToken::new()).await.unwrap();+ assert_eq!(+ resharding_rows(&mut publication.server, paused_name).await,+ 1+ );+ assert_eq!(resharding_rows(&mut publication.server, new_name).await, 1);+ other.finish_run().await.unwrap();++ for name in [paused_name, new_name] {+ remove_resumable(&mut publication.server, name).await;+ }+ source.shutdown();+ publication.cleanup().await;+ }++ async fn pause_after_replication(state: &ReshardingState) {+ start(state, 1).await.unwrap();+ state+ .prepare_replication(&CancellationToken::new())+ .await+ .unwrap();+ state.cancel_run().await.unwrap();+ }++ async fn abort(state: &ReshardingState) -> ReshardingRecord {+ state+ .abort_run(CommandKind::Replicate, Some(state.replication_slot()))+ .await+ .unwrap()+ }++ #[tokio::test]+ async fn abort_drops_every_slot_of_the_run() {+ crate::logger();+ let name = "state_abort";+ let mut publication =+ setup_publication_tables(&format!("{name}_pub"), &["state_abort_one"]).await;+ let source = Cluster::new_test(&config());+ source.launch();+ let shards = source.shards().len() as i64;+ remove_resumable(&mut publication.server, name).await;++ let state = keyed_state(&source, name, false);+ start(&state, 1).await.unwrap();+ state+ .prepare_replication(&CancellationToken::new())+ .await+ .unwrap();+ for locked in [+ state+ .check_record(CommandKind::Replicate, None)+ .await+ .map(drop),+ state+ .abort_run(CommandKind::Replicate, None)+ .await+ .map(drop),+ ] {+ assert!(+ matches!(locked, Err(Error::ReshardingLocked { .. })),+ "{locked:?}"+ );+ }+ state.cancel_run().await.unwrap();+ assert_eq!(slot_count(&mut publication.server, name).await, shards);++ for (command, slot) in [+ (CommandKind::Reshard, None),+ (CommandKind::Replicate, Some("other")),+ ] {+ let mismatch = state.check_record(command, slot).await;+ assert!(+ matches!(mismatch, Err(Error::RecordMismatch(_))),+ "{mismatch:?}"+ );+ }+ state+ .check_record(CommandKind::Replicate, None)+ .await+ .unwrap();++ abort(&state).await;+ wait_until_removed(&mut publication.server, name).await;+ assert_eq!(origin_count(&mut publication.server, name).await, 0);+ assert_eq!(resharding_rows(&mut publication.server, name).await, 0);+ assert!(matches!(+ state.check_record(CommandKind::Replicate, None).await,+ Err(Error::NoResharding(_))+ ));++ let owner = keyed_state(&source, name, true);+ owner.create_slots(&CancellationToken::new()).await.unwrap();+ owner.detach_slots();++ let state = keyed_state(&source, name, false);+ pause_after_replication(&state).await;+ abort(&state).await;+ wait_until_removed(&mut publication.server, name).await;+ assert_eq!(origin_count(&mut publication.server, name).await, 0);+ assert_eq!(resharding_rows(&mut publication.server, name).await, 0);++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn resume_state_refuses_parallel_and_unfinished_runs() {+ crate::logger();+ let mut publication = setup_publication().await;+ let name = "state_resume_refuse";+ let source = Cluster::new_test(&config());+ source.launch();+ remove_resumable(&mut publication.server, name).await;+ let state = keyed_state(&source, name, false);++ start(&state, 1).await.unwrap();+ state.create_slots(&CancellationToken::new()).await.unwrap();+ state.set_stage(Stage::Replication).await.unwrap();++ let parallel = start(&keyed_state(&source, name, false), 2).await;+ assert!(+ matches!(parallel, Err(Error::ReshardingLocked { .. })),+ "{parallel:?}"+ );++ state.cancel_run().await.unwrap();+ let unfinished = start(&keyed_state(&source, name, false), 3).await;+ assert!(+ matches!(unfinished, Err(Error::ReshardingExists { .. })),+ "{unfinished:?}"+ );+ assert_eq!(resharding_rows(&mut publication.server, name).await, 1);++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn finished_resume_state_frees_the_key() {+ crate::logger();+ let mut publication = setup_publication().await;+ let name = "state_resume_finish";+ let source = Cluster::new_test(&config());+ source.launch();+ remove_resumable(&mut publication.server, name).await;+ let state = keyed_state(&source, name, false);++ start(&state, 1).await.unwrap();+ assert!(!state.keeps_slots());+ state.create_slots(&CancellationToken::new()).await.unwrap();+ state.set_stage(Stage::Replication).await.unwrap();+ assert!(state.keeps_slots());++ state.finish_run().await.unwrap();+ assert!(!state.keeps_slots());+ assert!(state.run_store().is_none());+ assert_eq!(resharding_rows(&mut publication.server, name).await, 0);++ remove_resumable(&mut publication.server, name).await;+ let again = keyed_state(&source, name, false);+ start(&again, 2).await.unwrap();+ again.create_slots(&CancellationToken::new()).await.unwrap();+ again.set_stage(Stage::Replication).await.unwrap();+ assert!(again.keeps_slots());+ again.finish_run().await.unwrap();++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn cancelled_plain_run_drops_owned_slots_and_record() {+ crate::logger();+ let name = "state_plain_cancel";+ let mut publication =+ setup_publication_tables(&format!("{name}_pub"), &["state_plain_cancel_one"]).await;+ let source = Cluster::new_test(&config());+ source.launch();+ remove_resumable(&mut publication.server, name).await;+ let state = keyed_state(&source, name, true);++ start(&state, 1).await.unwrap();+ assert!(!state.keeps_slots());+ assert!(state.run_store().is_some());+ assert_eq!(resharding_rows(&mut publication.server, name).await, 1);++ state+ .prepare_replication(&CancellationToken::new())+ .await+ .unwrap();+ assert!(slot_count(&mut publication.server, name).await > 0);+ assert_eq!(resharding_rows(&mut publication.server, name).await, 1);++ state.cancel_run().await.unwrap();+ assert_eq!(slot_count(&mut publication.server, name).await, 0);+ assert_eq!(origin_count(&mut publication.server, name).await, 0);+ assert_eq!(resharding_rows(&mut publication.server, name).await, 0);++ start(&state, 3).await.unwrap();+ state.finish_run().await.unwrap();++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn dropped_plain_run_removes_its_record() {+ crate::logger();+ let name = "state_plain_drop";+ let mut publication =+ setup_publication_tables(&format!("{name}_pub"), &["state_plain_drop_one"]).await;+ let source = Cluster::new_test(&config());+ source.launch();+ remove_resumable(&mut publication.server, name).await;++ let state = keyed_state(&source, name, true);+ start(&state, 1).await.unwrap();+ state+ .prepare_replication(&CancellationToken::new())+ .await+ .unwrap();+ assert!(slot_count(&mut publication.server, name).await > 0);+ drop(state);++ tokio::time::timeout(Duration::from_secs(10), async {+ while resharding_rows(&mut publication.server, name).await != 0 {+ tokio::time::sleep(Duration::from_millis(50)).await;+ }+ })+ .await+ .expect("the row of a dropped run must be deleted");+ assert_eq!(slot_count(&mut publication.server, name).await, 0);+ assert_eq!(origin_count(&mut publication.server, name).await, 0);++ let again = keyed_state(&source, name, true);+ tokio::time::timeout(Duration::from_secs(10), async {+ while let Err(Error::ReshardingLocked { .. }) = start(&again, 2).await {+ tokio::time::sleep(Duration::from_millis(50)).await;+ }+ })+ .await+ .expect("the lock of a dropped run must be released");+ assert!(again.run_store().is_some());+ again.finish_run().await.unwrap();++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ async fn stored_stage(server: &mut Server, name: &str) -> String {+ server+ .fetch_all::<String>(format!(+ "SELECT stage FROM pgdog.resharding WHERE slot_name = '{name}'"+ ))+ .await+ .unwrap()+ .remove(0)+ }++ #[tokio::test]+ async fn stored_stage_moves_only_forward() {+ crate::logger();+ let mut publication = setup_publication().await;+ let name = "state_stage_forward";+ let source = Cluster::new_test(&config());+ source.launch();+ remove_resumable(&mut publication.server, name).await;++ let state = keyed_state(&source, name, true);+ state.start_run(RESHARD, TaskId::new(1)).await.unwrap();+ assert_eq!(+ stored_stage(&mut publication.server, name).await,+ Stage::Initializing.to_string()+ );+ for (write, stored) in [+ (Stage::CopyTables, Stage::CopyTables),+ (Stage::Replication, Stage::Replication),+ (Stage::CopyTables, Stage::Replication),+ (Stage::Initializing, Stage::Replication),+ ] {+ state.set_stage(write).await.unwrap();+ assert_eq!(+ stored_stage(&mut publication.server, name).await,+ stored.to_string(),+ "after writing {write}"+ );+ }+ state.complete_run().await.unwrap();++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn failed_load_keeps_the_slot_unowned() {+ crate::logger();+ let name = "state_user_slot_flag";+ let mut publication = keyed_publication(name).await;+ let source = Cluster::new_test(&config());+ source.launch();+ remove_resumable(&mut publication.server, name).await;++ pause_after_replication(&keyed_state(&source, name, false)).await;+ publication+ .server+ .execute_checked(format!("SELECT pg_drop_replication_slot('{name}_0')"))+ .await+ .unwrap();++ let resumed = keyed_state(&source, name, false);+ resume(&resumed, 2).await.unwrap();+ let loaded = resumed.create_slots(&CancellationToken::new()).await;+ assert!(+ matches!(loaded, Err(Error::MissingReplicationSlot(_))),+ "{loaded:?}"+ );+ assert!(resumed.slot(0).unwrap().existing());+ resumed.cancel_run().await.unwrap();++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ const RESHARD: ReshardingCommand = ReshardingCommand::Reshard {+ skip_schema_sync: false,+ replicate_only: false,+ sync_only: false,+ auto_cutover: false,+ tables_only: false,+ };++ async fn origin_progress(server: &mut Server, origin: &str) -> String {+ let rows: Vec<String> = server+ .fetch_all(format!(+ "SELECT pg_replication_origin_progress('{origin}', false)::text"+ ))+ .await+ .unwrap();+ rows[0].clone()+ }++ #[tokio::test]+ async fn resume_needs_a_matching_record() {+ crate::logger();+ let mut publication = setup_publication().await;+ let name = "state_resume_match";+ let source = Cluster::new_test(&config());+ source.launch();+ remove_resumable(&mut publication.server, name).await;++ let missing = resume(&keyed_state(&source, name, false), 1).await;+ assert!(+ matches!(missing, Err(Error::NoResharding(_))),+ "{missing:?}"+ );++ let state = keyed_state(&source, name, false);+ start(&state, 1).await.unwrap();+ state.create_slots(&CancellationToken::new()).await.unwrap();+ state.set_stage(Stage::Replication).await.unwrap();+ state.cancel_run().await.unwrap();++ let other_command = keyed_state(&source, name, false)+ .resume_run(+ ReshardingCommand::Replicate { auto_cutover: true },+ TaskId::new(2),+ )+ .await;+ assert!(+ matches!(+ other_command,+ Err(Error::RecordMismatch(RecordMismatch::Command { .. }))+ ),+ "{other_command:?}"+ );++ let mut other_slot = keyed_state(&source, name, false);+ other_slot.slot_name = "other".into();+ let other_slot = resume(&other_slot, 3).await;+ assert!(+ matches!(+ other_slot,+ Err(Error::RecordMismatch(RecordMismatch::Slot { .. }))+ ),+ "{other_slot:?}"+ );++ let resumed = keyed_state(&source, name, false);+ resume(&resumed, 7).await.unwrap();+ let owners: Vec<i64> = publication+ .server+ .fetch_all(format!(+ "SELECT task_id FROM pgdog.resharding \+ WHERE slot_name = '{name}' AND resumed_at IS NOT NULL"+ ))+ .await+ .unwrap();+ assert_eq!(owners, [7]);+ resumed.finish_run().await.unwrap();+ assert_eq!(resharding_rows(&mut publication.server, name).await, 0);++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn resumed_run_loads_slots_and_never_recreates_them() {+ crate::logger();+ let name = "state_resume_load";+ let mut publication =+ setup_publication_tables(&format!("{name}_pub"), &["state_resume_load_one"]).await;+ let source = Cluster::new_test(&config());+ source.launch();+ let shards = source.shards().len() as i64;+ remove_resumable(&mut publication.server, name).await;+ let cancel = CancellationToken::new();++ let state = keyed_state(&source, name, false);+ start(&state, 1).await.unwrap();+ state.prepare_replication(&cancel).await.unwrap();+ state.cancel_run().await.unwrap();++ let resumed = keyed_state(&source, name, false);+ resume(&resumed, 2).await.unwrap();+ resumed.prepare_replication(&cancel).await.unwrap();+ assert_eq!(slot_count(&mut publication.server, name).await, shards);+ resumed.cancel_run().await.unwrap();++ publication+ .server+ .execute_checked(format!("SELECT pg_drop_replication_slot('{name}_0')"))+ .await+ .unwrap();+ for task_id in [3, 4] {+ let resumed = keyed_state(&source, name, false);+ resume(&resumed, task_id).await.unwrap();+ let missing_slot = resumed.prepare_replication(&cancel).await;+ assert!(+ matches!(missing_slot, Err(Error::MissingReplicationSlot(_))),+ "{missing_slot:?}"+ );+ assert_eq!(slot_count(&mut publication.server, name).await, shards - 1);+ resumed.cancel_run().await.unwrap();+ assert_eq!(resharding_rows(&mut publication.server, name).await, 1);+ }++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn resumed_run_recreates_missing_origins_and_keeps_progress() {+ crate::logger();+ let name = "state_resume_reuse";+ let mut publication =+ setup_publication_tables(&format!("{name}_pub"), &["state_resume_reuse_one"]).await;+ let source = Cluster::new_test(&config());+ source.launch();+ let shards = source.shards().len() as i64;+ let last = shards - 1;+ remove_resumable(&mut publication.server, name).await;+ let kept = format!("__pgdog_origin_{name}_0_0");+ let lost = format!("__pgdog_origin_{name}_{last}_{last}");+ let cancel = CancellationToken::new();++ let state = keyed_state(&source, name, false);+ start(&state, 1).await.unwrap();+ state.prepare_replication(&cancel).await.unwrap();+ state.cancel_run().await.unwrap();+ publication+ .server+ .execute_checked(format!(+ "SELECT pg_replication_origin_advance('{kept}', '0/2000'); \+ SELECT pg_replication_origin_drop('{lost}')"+ ))+ .await+ .unwrap();++ let resumed = keyed_state(&source, name, false);+ resume(&resumed, 2).await.unwrap();+ resumed.prepare_replication(&cancel).await.unwrap();+ assert_eq!(+ origin_progress(&mut publication.server, &kept).await,+ "0/2000"+ );+ assert_eq!(+ origin_count(&mut publication.server, name).await,+ shards * shards+ );++ resumed.cancel_run().await.unwrap();++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ async fn keyed_publication(name: &str) -> super::super::publisher::test::PublicationTest {+ setup_publication_tables(&format!("{name}_pub"), &[&format!("{name}_one")]).await+ }++ async fn wait_for_lock(server: &mut Server, key: &ReshardingKey) {+ tokio::time::timeout(Duration::from_secs(10), async {+ loop {+ let locked: Vec<String> = server+ .fetch_all(format!(+ "SELECT pg_try_advisory_lock(hashtext('pgdog.resharding'), hashtext('{}'))::text",+ lock_name(key)+ ))+ .await+ .unwrap();+ if locked == ["true"] {+ break;+ }+ tokio::time::sleep(Duration::from_millis(50)).await;+ }+ })+ .await+ .expect("the lock of a terminated backend must be released");+ }++ async fn release_lock(server: &mut Server, key: &ReshardingKey) {+ server+ .execute_checked(format!(+ "SELECT pg_advisory_unlock(hashtext('pgdog.resharding'), hashtext('{}'))",+ lock_name(key)+ ))+ .await+ .unwrap();+ }++ #[tokio::test]+ async fn cancel_after_lost_state_connection_removes_the_record() {+ crate::logger();+ let name = "state_lost_cancel";+ let mut publication = keyed_publication(name).await;+ let source = Cluster::new_test(&config());+ source.launch();+ remove_resumable(&mut publication.server, name).await;++ let state = keyed_state(&source, name, true);+ start(&state, 1).await.unwrap();+ state+ .prepare_replication(&CancellationToken::new())+ .await+ .unwrap();+ terminate_state_backend(&mut publication.server, &state.run_key()).await;++ state.cancel_run().await.unwrap();+ assert!(state.run_store().is_none());+ assert_eq!(slot_count(&mut publication.server, name).await, 0);+ assert_eq!(origin_count(&mut publication.server, name).await, 0);+ assert_eq!(resharding_rows(&mut publication.server, name).await, 0);++ let again = keyed_state(&source, name, true);+ start(&again, 2).await.unwrap();+ again.finish_run().await.unwrap();++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn cancel_keeps_the_record_when_another_session_takes_the_lock() {+ crate::logger();+ let name = "state_lost_taken";+ let mut publication = keyed_publication(name).await;+ let source = Cluster::new_test(&config());+ source.launch();+ remove_resumable(&mut publication.server, name).await;++ let state = keyed_state(&source, name, true);+ let key = state.run_key();+ start(&state, 1).await.unwrap();+ state+ .prepare_replication(&CancellationToken::new())+ .await+ .unwrap();+ terminate_state_backend(&mut publication.server, &key).await;+ wait_for_lock(&mut publication.server, &key).await;++ let cancelled = state.cancel_run().await;+ assert!(+ matches!(cancelled, Err(Error::ReshardingLocked { .. })),+ "{cancelled:?}"+ );+ assert!(state.run_store().is_none());+ assert_eq!(slot_count(&mut publication.server, name).await, 0);+ assert_eq!(origin_count(&mut publication.server, name).await, 0);+ assert_eq!(resharding_rows(&mut publication.server, name).await, 1);++ release_lock(&mut publication.server, &key).await;+ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn complete_run_drops_slots_and_origins_then_the_record() {+ crate::logger();+ let name = "state_complete";+ let mut publication = keyed_publication(name).await;+ let source = Cluster::new_test(&config());+ source.launch();+ remove_resumable(&mut publication.server, name).await;++ let state = keyed_state(&source, name, false);+ start(&state, 1).await.unwrap();+ state+ .prepare_replication(&CancellationToken::new())+ .await+ .unwrap();++ state.complete_run().await.unwrap();+ assert!(!state.keeps_slots());+ assert_paused_state_is_empty(&state);+ assert_eq!(slot_count(&mut publication.server, name).await, 0);+ assert_eq!(origin_count(&mut publication.server, name).await, 0);+ assert_eq!(resharding_rows(&mut publication.server, name).await, 0);++ let again = keyed_state(&source, name, false);+ start(&again, 2).await.unwrap();+ again.finish_run().await.unwrap();++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn user_slots_stay_on_cancel_of_a_live_run_and_of_a_resumed_run() {+ crate::logger();+ let name = "state_user_slots_stay";+ let mut publication = keyed_publication(name).await;+ let source = Cluster::new_test(&config());+ source.launch();+ let shards = source.shards().len() as i64;+ remove_resumable(&mut publication.server, name).await;+ let cancel = CancellationToken::new();++ let owner = keyed_state(&source, name, true);+ owner.create_slots(&cancel).await.unwrap();+ owner.detach_slots();++ let fresh = keyed_state(&source, name, true);+ start(&fresh, 1).await.unwrap();+ fresh.prepare_replication(&cancel).await.unwrap();+ assert!(fresh.inner.lock().take_owned_slots().is_empty());+ fresh.cancel_run().await.unwrap();+ assert_eq!(slot_count(&mut publication.server, name).await, shards);+ assert_eq!(origin_count(&mut publication.server, name).await, 0);+ assert_eq!(resharding_rows(&mut publication.server, name).await, 0);++ let paused = keyed_state(&source, name, false);+ start(&paused, 2).await.unwrap();+ paused.prepare_replication(&cancel).await.unwrap();+ paused.cancel_run().await.unwrap();++ let resumed = keyed_state(&source, name, true);+ resume(&resumed, 3).await.unwrap();+ resumed.prepare_replication(&cancel).await.unwrap();+ resumed.cancel_run().await.unwrap();+ assert_eq!(slot_count(&mut publication.server, name).await, shards);+ assert_eq!(resharding_rows(&mut publication.server, name).await, 1);++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn partial_user_slots_refuse_a_new_run() {+ crate::logger();+ let mut publication = setup_publication().await;+ let name = "state_partial_slots";+ let source = Cluster::new_test(&config());+ source.launch();+ remove_resumable(&mut publication.server, name).await;+ publication+ .server+ .execute_checked(format!(+ "SELECT pg_create_logical_replication_slot('{name}_0', 'pgoutput')"+ ))+ .await+ .unwrap();++ for task_id in [1, 2] {+ let refused = start(&keyed_state(&source, name, true), task_id).await;+ assert!(+ matches!(refused, Err(Error::ReshardingSlotsPartial(_))),+ "{refused:?}"+ );+ }+ assert_eq!(resharding_rows(&mut publication.server, name).await, 0);+ assert_eq!(slot_count(&mut publication.server, name).await, 1);++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn crashed_resumable_run_keeps_its_objects_and_resumes() {+ crate::logger();+ let name = "state_crash";+ let mut publication = keyed_publication(name).await;+ let source = Cluster::new_test(&config());+ source.launch();+ let shards = source.shards().len() as i64;+ remove_resumable(&mut publication.server, name).await;+ let cancel = CancellationToken::new();++ let state = keyed_state(&source, name, false);+ start(&state, 1).await.unwrap();+ state.prepare_replication(&cancel).await.unwrap();+ drop(state);++ let resumed = keyed_state(&source, name, false);+ tokio::time::timeout(Duration::from_secs(10), async {+ while let Err(Error::ReshardingLocked { .. }) = resume(&resumed, 2).await {+ tokio::time::sleep(Duration::from_millis(50)).await;+ }+ })+ .await+ .expect("the lock of a crashed run must be released");+ assert!(resumed.run_store().is_some());+ assert_eq!(slot_count(&mut publication.server, name).await, shards);+ assert_eq!(+ origin_count(&mut publication.server, name).await,+ shards * shards+ );+ assert_eq!(resharding_rows(&mut publication.server, name).await, 1);++ resumed.prepare_replication(&cancel).await.unwrap();+ resumed.cancel_run().await.unwrap();+ assert_eq!(slot_count(&mut publication.server, name).await, shards);++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn abort_releases_the_lock_on_every_exit() {+ crate::logger();+ let name = "state_abort_unlock";+ let mut publication = keyed_publication(name).await;+ let source = Cluster::new_test(&config());+ source.launch();+ remove_resumable(&mut publication.server, name).await;++ let state = keyed_state(&source, name, false);+ pause_after_replication(&state).await;++ for (command, slot) in [+ (CommandKind::Reshard, None),+ (CommandKind::Replicate, Some("other")),+ ] {+ let mismatch = state.abort_run(command, slot).await.map(drop);+ assert!(+ matches!(mismatch, Err(Error::RecordMismatch(_))),+ "{mismatch:?}"+ );+ state+ .check_record(CommandKind::Replicate, None)+ .await+ .unwrap();+ }++ abort(&state).await;+ assert_eq!(resharding_rows(&mut publication.server, name).await, 0);+ let again = keyed_state(&source, name, false);+ start(&again, 2).await.unwrap();+ again.finish_run().await.unwrap();++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn resumed_run_keeps_its_stored_stage() {+ crate::logger();+ let mut publication = setup_publication().await;+ let name = "state_resume_stage";+ let source = Cluster::new_test(&config());+ source.launch();+ remove_resumable(&mut publication.server, name).await;++ let state = keyed_state(&source, name, false);+ start(&state, 1).await.unwrap();+ state.create_slots(&CancellationToken::new()).await.unwrap();+ state.set_stage(Stage::Replication).await.unwrap();+ state.cancel_run().await.unwrap();++ let resumed = keyed_state(&source, name, false);+ resume(&resumed, 2).await.unwrap();+ assert_eq!(+ resumed.inner.lock().run.as_ref().map(|run| run.stage),+ Some(Stage::Replication)+ );+ resumed.set_stage(Stage::CopyTables).await.unwrap();+ assert_eq!(+ stored_stage(&mut publication.server, name).await,+ Stage::Replication.to_string()+ );+ resumed.finish_run().await.unwrap();++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn row_is_written_at_start_and_staged_once_slots_exist() {+ crate::logger();+ let name = "state_row_at_start";+ let mut publication = keyed_publication(name).await;+ let source = Cluster::new_test(&config());+ source.launch();+ let cancel = CancellationToken::new();+ remove_resumable(&mut publication.server, name).await;++ let state = keyed_state(&source, name, false);+ start(&state, 1).await.unwrap();+ assert!(!state.keeps_slots());+ assert_eq!(resharding_rows(&mut publication.server, name).await, 1);+ assert_eq!(+ stored_stage(&mut publication.server, name).await,+ Stage::Initializing.to_string()+ );+ state.prepare_replication(&cancel).await.unwrap();+ assert!(state.keeps_slots());+ assert_eq!(+ stored_stage(&mut publication.server, name).await,+ Stage::Replication.to_string()+ );+ state.finish_run().await.unwrap();++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn resumable_run_stopped_while_initializing_drops_its_slots_and_row() {+ crate::logger();+ let mut publication = setup_publication().await;+ let name = "state_stopped_initializing";+ let source = Cluster::new_test(&config());+ source.launch();+ let shards = source.shards().len() as i64;+ remove_resumable(&mut publication.server, name).await;++ let state = keyed_state(&source, name, false);+ start(&state, 1).await.unwrap();+ state.create_slots(&CancellationToken::new()).await.unwrap();+ assert_eq!(slot_count(&mut publication.server, name).await, shards);+ assert!(!state.keeps_slots());++ state.cancel_run().await.unwrap();+ assert_eq!(slot_count(&mut publication.server, name).await, 0);+ assert_eq!(resharding_rows(&mut publication.server, name).await, 0);++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn run_stopped_before_slots_leaves_nothing() {+ crate::logger();+ let mut publication = setup_publication().await;+ let name = "state_stopped_early";+ let source = Cluster::new_test(&config());+ source.launch();+ remove_resumable(&mut publication.server, name).await;++ let state = keyed_state(&source, name, false);+ start(&state, 1).await.unwrap();+ assert!(!state.keeps_slots());+ assert!(state.run_store().is_some());+ state.cancel_run().await.unwrap();+ assert!(state.run_store().is_none());+ assert_eq!(slot_count(&mut publication.server, name).await, 0);+ assert_eq!(resharding_rows(&mut publication.server, name).await, 0);++ let dropped = keyed_state(&source, name, false);+ start(&dropped, 2).await.unwrap();+ drop(dropped);++ let again = keyed_state(&source, name, false);+ tokio::time::timeout(Duration::from_secs(10), async {+ while let Err(Error::ReshardingLocked { .. }) = start(&again, 3).await {+ tokio::time::sleep(Duration::from_millis(50)).await;+ }+ })+ .await+ .expect("the lock of a dropped run must be released");+ assert!(again.run_store().is_some());+ assert_eq!(slot_count(&mut publication.server, name).await, 0);+ assert_eq!(resharding_rows(&mut publication.server, name).await, 1);+ again.finish_run().await.unwrap();++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn copy_tables_row_refuses_resume() {+ crate::logger();+ let mut publication = setup_publication().await;+ let name = "state_copy_resume";+ let source = Cluster::new_test(&config());+ source.launch();+ let shards = source.shards().len() as i64;+ remove_resumable(&mut publication.server, name).await;++ let state = keyed_state(&source, name, false);+ state.start_run(RESHARD, TaskId::new(1)).await.unwrap();+ state.create_slots(&CancellationToken::new()).await.unwrap();+ state.set_stage(Stage::CopyTables).await.unwrap();+ state.cancel_run().await.unwrap();+ assert_eq!(slot_count(&mut publication.server, name).await, shards);++ let resumed = keyed_state(&source, name, false);+ let refused = resumed.resume_run(RESHARD, TaskId::new(2)).await;+ assert!(+ matches!(refused, Err(Error::ReshardingNotResumable { .. })),+ "{refused:?}"+ );+ assert!(resumed.run_store().is_none());+ assert_eq!(resharding_rows(&mut publication.server, name).await, 1);+ assert_eq!(+ stored_stage(&mut publication.server, name).await,+ Stage::CopyTables.to_string()+ );++ state.abort_run(CommandKind::Reshard, None).await.unwrap();+ wait_until_removed(&mut publication.server, name).await;+ assert_eq!(resharding_rows(&mut publication.server, name).await, 0);++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }++ #[tokio::test]+ async fn cutover_ends_the_run() {+ crate::logger();+ let name = "state_cutover";+ let mut publication = keyed_publication(name).await;+ let source = Cluster::new_test(&config());+ source.launch();+ let shards = source.shards().len() as i64;+ remove_resumable(&mut publication.server, name).await;+ let cancel = CancellationToken::new();++ let state = keyed_state(&source, name, false);+ start(&state, 1).await.unwrap();+ state.prepare_replication(&cancel).await.unwrap();+ assert_eq!(resharding_rows(&mut publication.server, name).await, 1);++ state.prepare_reverse_replication(&cancel).await.unwrap();+ state.prepare_replication(&cancel).await.unwrap();+ assert_eq!(resharding_rows(&mut publication.server, name).await, 0);+ assert_eq!(slot_count(&mut publication.server, name).await, shards);+ assert_eq!(+ origin_count(&mut publication.server, name).await,+ shards * shards+ );+ assert!(!state.keeps_slots());+ assert!(matches!(+ state.check_record(CommandKind::Replicate, None).await,+ Err(Error::NoResharding(_))+ ));++ state.cancel_run().await.unwrap();+ wait_until_removed(&mut publication.server, name).await;+ assert_eq!(origin_count(&mut publication.server, name).await, 0);++ remove_resumable(&mut publication.server, name).await;+ source.shutdown();+ publication.cleanup().await;+ }
}
diff --git a/pgdog/src/backend/replication/logical/run/mod.rs b/pgdog/src/backend/replication/logical/run/mod.rsnew file mode 100644index 000000000..ef12572a3--- /dev/null+++ b/pgdog/src/backend/replication/logical/run/mod.rs@@ -0,0 +1,176 @@+use std::collections::HashMap;+use std::time::SystemTime;++use derive_more::{Display, FromStr, FromStrError};++use pgdog_stats::{CommandKind, Databases, InstanceId, ReshardingId, RunId, TaskId, run_id};+use serde::{Deserialize, Serialize};++use super::Error;+use super::publisher::{Permanent, ReplicationSlot};+use crate::backend::{+ Cluster,+ pool::{Address, Request},+};++mod queries;+pub(crate) mod scan;+pub(crate) mod snapshot;+mod store;+pub(crate) mod watchdog;++pub(crate) use snapshot::ConfigSnapshot;+pub(crate) use store::ReshardingStore;+pub(crate) use watchdog::LockWatchdog;++pub(crate) async fn state_address(destination: &Cluster) -> Result<Address, Error> {+ Ok(destination+ .shards()+ .first()+ .ok_or_else(|| Error::NoPrimary)?+ .primary(&Request::default())+ .await?+ .addr()+ .clone())+}++pub(crate) const RESHARDING_TABLE: &str = "resharding";++/// Version of the data in [`RESHARDING_TABLE`]. Increase it on breaking changes.+pub(crate) const FORMAT_VERSION: i32 = 1;++/// Identifier of the specific resharding.+#[derive(Clone, Debug, Hash, Eq, PartialEq, Display)]+#[display("{databases} ({publication})")]+pub(crate) struct ReshardingKey {+ pub(crate) databases: Databases,+ pub(crate) publication: String,+}++impl ReshardingKey {+ pub(crate) fn new(databases: Databases, publication: &str) -> Self {+ Self {+ databases,+ publication: publication.to_owned(),+ }+ }+}++/// The command that started the run, with its options.+#[derive(Clone, Copy, Debug, PartialEq, Eq, Display, Serialize, Deserialize)]+#[serde(tag = "command", rename_all = "snake_case")]+pub(crate) enum ReshardingCommand {+ #[display("reshard")]+ Reshard {+ skip_schema_sync: bool,+ replicate_only: bool,+ sync_only: bool,+ auto_cutover: bool,+ tables_only: bool,+ },+ #[display("replicate")]+ Replicate { auto_cutover: bool },+}++impl ReshardingCommand {+ pub(crate) fn kind(&self) -> CommandKind {+ match self {+ Self::Reshard { .. } => CommandKind::Reshard,+ Self::Replicate { .. } => CommandKind::Replicate,+ }+ }+}++/// The last known stage of the resharding run.+#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Display, FromStr)]+#[display(rename_all = "snake_case")]+#[from_str(rename_all = "snake_case")]+#[from_str(error(Error, format_error))]+pub(crate) enum Stage {+ Initializing,+ CopyTables,+ Replication,+}++fn format_error(err: FromStrError) -> Error {+ Error::ReshardingFormat(err.to_string())+}++#[derive(Clone, Debug, PartialEq, Eq, Display)]+#[display("task {task_id} on {instance_id} (run {run_id})")]+pub(crate) struct TaskOwner {+ pub(crate) instance_id: InstanceId,+ pub(crate) run_id: RunId,+ pub(crate) task_id: TaskId,+}++impl TaskOwner {+ pub(crate) fn current(task_id: TaskId) -> Self {+ Self {+ instance_id: InstanceId::from(crate::util::instance_id().to_owned()),+ run_id: run_id(),+ task_id,+ }+ }+}++/// A single row record for the one resharding process.+#[derive(Clone, Debug, PartialEq)]+pub(crate) struct ReshardingRecord {+ pub(crate) id: ReshardingId,+ pub(crate) key: ReshardingKey,+ pub(crate) command: ReshardingCommand,+ pub(crate) slot_name: String,+ pub(crate) stage: Stage,+ pub(crate) config: ConfigSnapshot,+ pub(crate) owner: TaskOwner,+ pub(crate) format_version: i32,+ pub(crate) started_at: SystemTime,+ pub(crate) resumed_at: Option<SystemTime>,+}++pub(crate) async fn existing_slots(+ source: &Cluster,+ publication: &str,+ slot_name: &str,+) -> Result<HashMap<usize, ReplicationSlot<Permanent>>, Error> {+ let mut slots = HashMap::new();+ for (number, shard) in source.shards().iter().enumerate() {+ let addr = shard.primary(&Request::default()).await?.addr().clone();+ let slot =+ ReplicationSlot::new_permanent(publication, &addr, Some(slot_name.to_owned()), number);+ match slot.verify_exists().await {+ Ok(()) => {+ slots.insert(number, slot);+ }+ Err(Error::MissingReplicationSlot(_)) => (),+ Err(err) => return Err(err),+ }+ }+ Ok(slots)+}++#[cfg(test)]+mod test {+ use super::*;++ #[test]+ fn stage_round_trips_its_stored_text_in_pipeline_order() {+ let stages = [+ (Stage::Initializing, "initializing"),+ (Stage::CopyTables, "copy_tables"),+ (Stage::Replication, "replication"),+ ];+ for (stage, text) in stages {+ assert_eq!(stage.to_string(), text);+ assert_eq!(text.parse::<Stage>().unwrap(), stage);+ }+ for pair in stages.windows(2) {+ assert!(pair[0].0 < pair[1].0, "{} < {}", pair[0].1, pair[1].1);+ }+ assert!(matches!(+ "new_stage".parse::<Stage>(),+ Err(Error::ReshardingFormat(_))+ ));+ }+}diff --git a/pgdog/src/backend/replication/logical/run/queries.rs b/pgdog/src/backend/replication/logical/run/queries.rsnew file mode 100644index 000000000..7f494789b--- /dev/null+++ b/pgdog/src/backend/replication/logical/run/queries.rs@@ -0,0 +1,286 @@+use std::time::{SystemTime, UNIX_EPOCH};++use pgdog_stats::ReshardingId;++use super::{ReshardingKey, ReshardingRecord, Stage, TaskOwner};+use crate::backend::server::ServerRequest;+use crate::net::bind::Parameter;++pub(crate) const CREATE_TABLE: &str = include_str!("resharding.sql");++pub(crate) trait Query {+ const SQL: &'static str;++ type Params<'a>;++ fn params(params: Self::Params<'_>) -> Vec<Parameter>;++ fn request(params: Self::Params<'_>) -> ServerRequest {+ ServerRequest::parameterized(Self::SQL, &Self::params(params))+ }+}++pub(crate) struct SetupSession;++impl Query for SetupSession {+ // change tcp keepalive settings to react faster on the+ // client socket abruption+ const SQL: &'static str = "+ SELECT set_config('application_name', $1, false),+ set_config('tcp_keepalives_idle', '60', false),+ set_config('tcp_keepalives_interval', '15', false),+ set_config('tcp_keepalives_count', '4', false),+ set_config('tcp_user_timeout', '120000', false)";++ type Params<'a> = &'a str;++ fn params(application_name: &str) -> Vec<Parameter> {+ vec![text(application_name)]+ }+}++pub(crate) struct TableExists;++impl Query for TableExists {+ const SQL: &'static str = "SELECT (to_regclass('pgdog.resharding') IS NOT NULL)::int";++ type Params<'a> = ();++ fn params(_: ()) -> Vec<Parameter> {+ vec![]+ }+}++pub(crate) struct TryLock;++impl Query for TryLock {+ const SQL: &'static str =+ "SELECT pg_try_advisory_lock(hashtext('pgdog.resharding'), hashtext($1))::int";++ type Params<'a> = &'a ReshardingKey;++ fn params(key: &ReshardingKey) -> Vec<Parameter> {+ vec![lock_name(key)]+ }+}++pub(crate) struct Unlock;++impl Query for Unlock {+ const SQL: &'static str =+ "SELECT pg_advisory_unlock(hashtext('pgdog.resharding'), hashtext($1))";++ type Params<'a> = &'a ReshardingKey;++ fn params(key: &ReshardingKey) -> Vec<Parameter> {+ vec![lock_name(key)]+ }+}++pub(crate) struct LockHolder;++impl Query for LockHolder {+ const SQL: &'static str = "+ SELECT a.application_name+ FROM pg_locks l+ JOIN pg_stat_activity a ON a.pid = l.pid+ WHERE l.locktype = 'advisory'+ AND l.granted+ AND l.database = (SELECT oid FROM pg_database WHERE datname = current_database())+ AND l.classid = (hashtext('pgdog.resharding')::bigint & 4294967295)::oid+ AND l.objid = (hashtext($1)::bigint & 4294967295)::oid+ AND l.objsubid = 2";++ type Params<'a> = &'a ReshardingKey;++ fn params(key: &ReshardingKey) -> Vec<Parameter> {+ vec![lock_name(key)]+ }+}++pub(crate) struct SelectRow;++impl Query for SelectRow {+ const SQL: &'static str = "+ SELECT source_db, destination_db, publication, command::text, slot_name,+ stage, config::text, instance_id, run_id::text,+ task_id::text, format_version::text, id::text,+ (extract(epoch FROM started_at) * 1000)::bigint::text,+ coalesce((extract(epoch FROM resumed_at) * 1000)::bigint::text, '')+ FROM pgdog.resharding+ WHERE source_db = $1 AND destination_db = $2 AND publication = $3";++ type Params<'a> = &'a ReshardingKey;++ fn params(key: &ReshardingKey) -> Vec<Parameter> {+ key_params(key).into()+ }+}++pub(crate) struct ListRows;++impl Query for ListRows {+ const SQL: &'static str = "+ SELECT source_db, destination_db, publication, command::text, slot_name,+ stage, config::text, instance_id, run_id::text,+ task_id::text, format_version::text, id::text,+ (extract(epoch FROM started_at) * 1000)::bigint::text,+ coalesce((extract(epoch FROM resumed_at) * 1000)::bigint::text, '')+ FROM pgdog.resharding+ ORDER BY source_db, destination_db, publication";++ type Params<'a> = ();++ fn params(_: ()) -> Vec<Parameter> {+ vec![]+ }+}++pub(crate) struct InsertRow;++impl Query for InsertRow {+ const SQL: &'static str = "+ INSERT INTO pgdog.resharding (+ source_db, destination_db, publication, command, slot_name,+ stage, config, instance_id, run_id, task_id, format_version, id,+ started_at+ )+ VALUES ($1, $2, $3, $4::jsonb, $5, $6, $7::jsonb, $8, $9::uuid,+ $10::bigint, $11::integer, $12::uuid, to_timestamp($13::bigint / 1000.0))";++ type Params<'a> = &'a ReshardingRecord;++ fn params(row: &ReshardingRecord) -> Vec<Parameter> {+ let [source, destination, publication] = key_params(&row.key);+ vec![+ source,+ destination,+ publication,+ text(&serde_json::to_string(&row.command).expect("command is valid JSON")),+ text(&row.slot_name),+ text(&row.stage.to_string()),+ text(&serde_json::to_string(&row.config).expect("config snapshot is valid JSON")),+ text(row.owner.instance_id.as_str()),+ text(&row.owner.run_id.to_string()),+ text(&row.owner.task_id.to_string()),+ text(&row.format_version.to_string()),+ text(&row.id.to_string()),+ text(&unix_millis(row.started_at).to_string()),+ ]+ }+}++pub(crate) struct UpdateStage;++pub(crate) struct StageUpdate {+ pub(crate) id: ReshardingId,+ pub(crate) stage: Stage,+}++impl Query for UpdateStage {+ const SQL: &'static str = "+ UPDATE pgdog.resharding+ SET stage = $2+ WHERE id = $1::uuid";++ type Params<'a> = StageUpdate;++ fn params(update: StageUpdate) -> Vec<Parameter> {+ vec![+ text(&update.id.to_string()),+ text(&update.stage.to_string()),+ ]+ }+}++pub(crate) struct ResumeRow;++pub(crate) struct ResumeUpdate<'a> {+ pub(crate) id: ReshardingId,+ pub(crate) owner: &'a TaskOwner,+ pub(crate) resumed_at: SystemTime,+}++impl Query for ResumeRow {+ const SQL: &'static str = "+ UPDATE pgdog.resharding+ SET instance_id = $2, run_id = $3::uuid, task_id = $4::bigint,+ resumed_at = to_timestamp($5::bigint / 1000.0)+ WHERE id = $1::uuid";++ type Params<'a> = ResumeUpdate<'a>;++ fn params(update: ResumeUpdate<'_>) -> Vec<Parameter> {+ vec![+ text(&update.id.to_string()),+ text(update.owner.instance_id.as_str()),+ text(&update.owner.run_id.to_string()),+ text(&update.owner.task_id.to_string()),+ text(&unix_millis(update.resumed_at).to_string()),+ ]+ }+}++pub(crate) struct Heartbeat;++impl Query for Heartbeat {+ const SQL: &'static str = "SELECT 1";++ type Params<'a> = ();++ fn params(_: ()) -> Vec<Parameter> {+ vec![]+ }+}++pub(crate) struct DeleteRow;++impl Query for DeleteRow {+ const SQL: &'static str = "DELETE FROM pgdog.resharding WHERE id = $1::uuid";++ type Params<'a> = ReshardingId;++ fn params(id: ReshardingId) -> Vec<Parameter> {+ vec![text(&id.to_string())]+ }+}++#[cfg(test)]+pub(crate) struct DeleteKey;++#[cfg(test)]+impl Query for DeleteKey {+ const SQL: &'static str = "+ DELETE FROM pgdog.resharding+ WHERE source_db = $1 AND destination_db = $2 AND publication = $3";++ type Params<'a> = &'a ReshardingKey;++ fn params(key: &ReshardingKey) -> Vec<Parameter> {+ key_params(key).into()+ }+}++fn text(value: &str) -> Parameter {+ Parameter::new(value.as_bytes())+}++fn key_params(key: &ReshardingKey) -> [Parameter; 3] {+ [+ text(&key.databases.source),+ text(&key.databases.destination),+ text(&key.publication),+ ]+}++fn lock_name(key: &ReshardingKey) -> Parameter {+ text(&format!(+ "{}/{}/{}",+ key.databases.source, key.databases.destination, key.publication+ ))+}++fn unix_millis(time: SystemTime) -> i64 {+ time.duration_since(UNIX_EPOCH)+ .map_or(0, |since| since.as_millis() as i64)+}diff --git a/pgdog/src/backend/replication/logical/run/resharding.sql b/pgdog/src/backend/replication/logical/run/resharding.sqlnew file mode 100644index 000000000..93f68c481--- /dev/null+++ b/pgdog/src/backend/replication/logical/run/resharding.sql@@ -0,0 +1,27 @@+BEGIN;++SET LOCAL client_min_messages = warning;++SELECT pg_advisory_xact_lock(hashtext('pgdog.resharding'));++CREATE SCHEMA IF NOT EXISTS pgdog;++CREATE TABLE IF NOT EXISTS pgdog.resharding (+ id UUID PRIMARY KEY,+ source_db TEXT NOT NULL,+ destination_db TEXT NOT NULL,+ publication TEXT NOT NULL,+ command JSONB NOT NULL,+ slot_name TEXT NOT NULL,+ stage TEXT NOT NULL,+ config JSONB NOT NULL,+ instance_id TEXT NOT NULL,+ run_id UUID NOT NULL,+ task_id BIGINT NOT NULL,+ format_version INTEGER NOT NULL,+ started_at TIMESTAMPTZ NOT NULL,+ resumed_at TIMESTAMPTZ,+ UNIQUE (source_db, destination_db, publication)+);++COMMIT;diff --git a/pgdog/src/backend/replication/logical/run/scan.rs b/pgdog/src/backend/replication/logical/run/scan.rsnew file mode 100644index 000000000..fe884851e--- /dev/null+++ b/pgdog/src/backend/replication/logical/run/scan.rs@@ -0,0 +1,231 @@+use std::collections::BTreeMap;+use std::time::{Duration, SystemTime};++use derive_more::Display;+use futures::future::join_all;++use super::super::{Error, subscriber::connect_address};+use super::{ReshardingKey, ReshardingRecord, ReshardingStore, state_address};+use crate::backend::databases::databases;+use crate::backend::pool::Address;+use crate::util::{human_duration_display, safe_timeout};+use tracing::{info, warn};++const SCAN_TIMEOUT: Duration = Duration::from_secs(10);++#[derive(Debug)]+pub(crate) enum ReshardingScan {+ Found(Box<ReshardingReport>),+ Unreadable { key: ReshardingKey, error: String },+ Failed { database: String, error: String },+}++#[derive(Debug)]+pub(crate) struct ReshardingReport {+ pub(crate) record: ReshardingRecord,+ pub(crate) state: RunState,+}++#[derive(Debug, Display)]+pub(crate) enum RunState {+ #[display("running")]+ Running,+ #[display("paused")]+ Paused,+}++pub(crate) async fn scan() -> Vec<ReshardingScan> {+ let clusters = databases().schema_owners();+ let addresses = join_all(clusters.iter().map(state_address)).await;++ let mut results = vec![];+ let mut by_address: BTreeMap<String, (Address, Vec<String>)> = BTreeMap::new();+ for (cluster, address) in clusters.iter().zip(addresses) {+ let database = cluster.identifier().database.clone();+ match address {+ Ok(address) => by_address+ .entry(address.to_string())+ .or_insert_with(|| (address, vec![]))+ .1+ .push(database),+ Err(err) => results.push(ReshardingScan::Failed {+ database,+ error: err.to_string(),+ }),+ }+ }++ let scans = join_all(+ by_address+ .into_values()+ .map(|(address, mut databases)| async move {+ match scan_address(&address).await {+ Ok(scans) => scans,+ Err(error) => {+ databases.sort();+ vec![ReshardingScan::Failed {+ database: databases.join(", "),+ error,+ }]+ }+ }+ }),+ )+ .await;+ results.extend(scans.into_iter().flatten());+ results+}++/// Never fails the run: other reshardings on the destination are only logged.+pub(crate) async fn check_other_runs(address: &Address, own: &ReshardingKey) {+ info!("resharding {own}: checking {address} for other unfinished reshardings");+ let scans = match scan_address(address).await {+ Ok(scans) => scans,+ Err(err) => {+ warn!("resharding {own}: checking other reshardings on {address} failed: {err}");+ return;+ }+ };++ let mut others = 0;+ for entry in scans {+ match entry {+ ReshardingScan::Found(report) if report.record.key != *own => {+ others += 1;+ log_other_run(&report);+ }+ ReshardingScan::Unreadable { key, error } if key != *own => {+ others += 1;+ warn!("unfinished resharding {key} cannot be read by this pgdog version: {error}")+ }+ _ => (),+ }+ }+ if others == 0 {+ info!("resharding {own}: no other unfinished reshardings on {address}");+ }+}++fn log_other_run(report: &ReshardingReport) {+ let record = &report.record;+ let times = match record.resumed_at {+ Some(resumed_at) => format!(+ "started {} ago, resumed {} ago",+ ago(record.started_at),+ ago(resumed_at)+ ),+ None => format!("started {} ago", ago(record.started_at)),+ };+ match &report.state {+ RunState::Running => info!(+ "resharding {} is in progress ({}, stage {}, running on {}, {times})",+ record.key, record.command, record.stage, record.owner.instance_id,+ ),+ RunState::Paused => warn!(+ "unfinished resharding {} ({}, stage {}, paused, {times}, slot {}); resume it or abort it with WITH (abort)",+ record.key, record.command, record.stage, record.slot_name,+ ),+ }+}++fn ago(time: SystemTime) -> String {+ human_duration_display(SystemTime::now().duration_since(time).unwrap_or_default())+}++async fn scan_address(address: &Address) -> Result<Vec<ReshardingScan>, String> {+ match safe_timeout(SCAN_TIMEOUT, list_table(address)).await {+ Ok(scans) => scans.map_err(|err| err.to_string()),+ Err(_) => Err(format!("timed out after {}s", SCAN_TIMEOUT.as_secs())),+ }+}++async fn list_table(address: &Address) -> Result<Vec<ReshardingScan>, Error> {+ let mut server = connect_address(address).await?;+ let mut found = vec![];+ let mut unreadable = vec![];+ for row in ReshardingStore::list(&mut server).await? {+ match row {+ Ok(record) => {+ let state = match ReshardingStore::lock_owner(&mut server, &record.key).await? {+ Some(_) => RunState::Running,+ None => RunState::Paused,+ };+ found.push(ReshardingScan::Found(Box::new(ReshardingReport {+ record,+ state,+ })));+ }+ Err(row) => unreadable.push(ReshardingScan::Unreadable {+ key: row.key,+ error: row.error.to_string(),+ }),+ }+ }+ found.extend(unreadable);+ Ok(found)+}++#[cfg(test)]+mod test {+ use super::super::store::test::{key, row};+ use super::*;++ fn found<'a>(scans: &'a [ReshardingScan], key: &ReshardingKey) -> &'a ReshardingReport {+ scans+ .iter()+ .find_map(|scan| match scan {+ ReshardingScan::Found(report) if report.record.key == *key => Some(report.as_ref()),+ _ => None,+ })+ .unwrap_or_else(|| panic!("{key} not found in {scans:?}"))+ }++ #[tokio::test]+ async fn table_scan_reports_running_paused_and_unreadable_rows() {+ let addr = Address::new_test();+ let running_key = key("scan_running");+ let paused_key = key("scan_paused");+ let newer_key = key("scan_newer");++ let running = ReshardingStore::open(&addr, running_key.clone())+ .await+ .unwrap();+ let paused = ReshardingStore::open(&addr, paused_key.clone())+ .await+ .unwrap();+ let newer = ReshardingStore::open(&addr, newer_key.clone())+ .await+ .unwrap();+ for store in [&running, &paused, &newer] {+ store.clear().await.unwrap();+ }+ running.insert(&row(running_key.clone())).await.unwrap();+ paused.insert(&row(paused_key.clone())).await.unwrap();+ newer+ .insert(&ReshardingRecord {+ format_version: super::super::FORMAT_VERSION + 1,+ ..row(newer_key.clone())+ })+ .await+ .unwrap();+ paused.unlock().await.unwrap();+ newer.unlock().await.unwrap();++ let scans = list_table(&addr).await.unwrap();+ assert!(matches!(+ found(&scans, &running_key).state,+ RunState::Running+ ));+ assert!(matches!(found(&scans, &paused_key).state, RunState::Paused));+ assert!(+ scans.iter().any(+ |scan| matches!(scan, ReshardingScan::Unreadable { key, .. } if *key == newer_key)+ ),+ "{scans:?}"+ );++ for store in [&running, &paused, &newer] {+ store.clear().await.unwrap();+ }+ }+}diff --git a/pgdog/src/backend/replication/logical/run/snapshot.rs b/pgdog/src/backend/replication/logical/run/snapshot.rsnew file mode 100644index 000000000..d949bc1a1--- /dev/null+++ b/pgdog/src/backend/replication/logical/run/snapshot.rs@@ -0,0 +1,291 @@+use derive_more::Display;+use pgdog_config::{+ OmnishardedTables, ShardedMappingDeprecated, ShardedSchema, ShardedTableConfig,+};+use serde::{Deserialize, Serialize, de::DeserializeOwned};+use serde_json::Value;++use super::super::Error;+use crate::backend::Cluster;++/// Represent the cluster configuration that could affect+/// the resharding if changed+#[derive(Serialize, Deserialize, Debug, Clone, PartialEq)]+pub(crate) struct ClusterSnapshot {+ shards: usize,+ sharded_tables: Vec<Value>,+ sharded_mappings: Vec<Value>,+ omnisharded_tables: Vec<Value>,+ sharded_schemas: Vec<Value>,+}++impl ClusterSnapshot {+ pub(crate) fn capture(cluster: &Cluster, config: &pgdog_config::Config) -> Result<Self, Error> {+ let name = cluster.name();+ Ok(Self {+ shards: cluster.shards().len(),+ sharded_tables: sorted_rules(+ config.sharded_tables.iter().filter(|t| t.database == name),+ )?,+ sharded_mappings: sorted_rules(+ config+ .sharded_mappings+ .iter()+ .filter(|m| m.database == name),+ )?,+ omnisharded_tables: sorted_rules(+ config+ .omnisharded_tables+ .iter()+ .filter(|t| t.database == name),+ )?,+ sharded_schemas: sorted_rules(+ config.sharded_schemas.iter().filter(|s| s.database == name),+ )?,+ })+ }++ fn changed_fields(&self, current: &Self) -> Vec<ConfigField> {+ [+ (ConfigField::Shards, self.shards != current.shards),+ (+ ConfigField::ShardedTables,+ !same_rules::<ShardedTableConfig>(&self.sharded_tables, ¤t.sharded_tables),+ ),+ (+ ConfigField::ShardedMappings,+ !same_rules::<ShardedMappingDeprecated>(+ &self.sharded_mappings,+ ¤t.sharded_mappings,+ ),+ ),+ (+ ConfigField::OmnishardedTables,+ !same_rules::<OmnishardedTables>(+ &self.omnisharded_tables,+ ¤t.omnisharded_tables,+ ),+ ),+ (+ ConfigField::ShardedSchemas,+ !same_rules::<ShardedSchema>(&self.sharded_schemas, ¤t.sharded_schemas),+ ),+ ]+ .into_iter()+ .filter_map(|(field, changed)| changed.then_some(field))+ .collect()+ }+}++#[derive(Debug, Clone, Copy, PartialEq, Eq, Display)]+#[display(rename_all = "snake_case")]+pub(crate) enum ConfigField {+ Shards,+ ShardedTables,+ ShardedMappings,+ OmnishardedTables,+ ShardedSchemas,+}++#[derive(Debug, Clone, Copy, PartialEq, Eq, Display)]+pub(crate) enum ConfigMismatch {+ #[display("source.{_0}")]+ Source(ConfigField),+ #[display("destination.{_0}")]+ Destination(ConfigField),+}++/// Combined snapshot of source and destination cluster to+/// catch any misaligns during resharding reruns.+#[derive(Serialize, Deserialize, Debug, Clone, PartialEq)]+pub(crate) struct ConfigSnapshot {+ source: ClusterSnapshot,+ destination: ClusterSnapshot,+}++impl ConfigSnapshot {+ pub(crate) fn capture(+ source: &Cluster,+ destination: &Cluster,+ config: &pgdog_config::Config,+ ) -> Result<Self, Error> {+ Ok(Self {+ source: ClusterSnapshot::capture(source, config)?,+ destination: ClusterSnapshot::capture(destination, config)?,+ })+ }++ pub(crate) fn validate_equal(+ &self,+ current: &ConfigSnapshot,+ ) -> Result<(), Vec<ConfigMismatch>> {+ let mismatches: Vec<_> = self+ .source+ .changed_fields(¤t.source)+ .into_iter()+ .map(ConfigMismatch::Source)+ .chain(+ self.destination+ .changed_fields(¤t.destination)+ .into_iter()+ .map(ConfigMismatch::Destination),+ )+ .collect();++ if mismatches.is_empty() {+ Ok(())+ } else {+ Err(mismatches)+ }+ }+}++fn sorted_rules<T: Serialize>(items: impl Iterator<Item = T>) -> Result<Vec<Value>, Error> {+ let mut rules = items+ .map(|item| {+ serde_json::to_value(item).map_err(|err| Error::ReshardingFormat(err.to_string()))+ })+ .collect::<Result<Vec<_>, _>>()?;+ rules.sort_by_cached_key(|rule| rule.to_string());+ Ok(rules)+}++/// Compare two rule lists as config values of type `T`. Rules that+/// `T` cannot read are compared as JSON.+fn same_rules<T: DeserializeOwned + PartialEq>(stored: &[Value], current: &[Value]) -> bool {+ let parse = |rules: &[Value]| {+ rules+ .iter()+ .map(T::deserialize)+ .collect::<Result<Vec<T>, _>>()+ };+ match (parse(stored), parse(current)) {+ (Ok(stored), Ok(current)) => stored == current,+ _ => stored == current,+ }+}++#[cfg(test)]+mod test {+ use serde_json::json;++ use super::super::super::RecordMismatch;++ use super::*;++ fn table(name: &str) -> ShardedTableConfig {+ ShardedTableConfig {+ database: "destination".into(),+ name: Some(name.into()),+ column: "id".into(),+ ..Default::default()+ }+ }++ fn snapshot() -> ConfigSnapshot {+ let cluster = |shards| ClusterSnapshot {+ shards,+ sharded_tables: vec![json!({"name": "users"})],+ sharded_mappings: vec![],+ omnisharded_tables: vec![],+ sharded_schemas: vec![],+ };+ ConfigSnapshot {+ source: cluster(1),+ destination: cluster(2),+ }+ }++ fn rules() -> Vec<Value> {+ vec![json!({"name": "orders"})]+ }++ #[test]+ fn sorted_rules_ignore_config_order() {+ let forward = sorted_rules([table("users"), table("orders")].iter()).unwrap();+ let backward = sorted_rules([table("orders"), table("users")].iter()).unwrap();+ assert_eq!(forward, backward);+ assert_eq!(forward.len(), 2);+ }++ #[test]+ fn snapshot_survives_json_round_trip() {+ let left = snapshot();+ let right: ConfigSnapshot =+ serde_json::from_str(&serde_json::to_string(&left).unwrap()).unwrap();+ assert_eq!(left, right);+ assert_eq!(left.validate_equal(&right), Ok(()));+ }++ #[test]+ fn stored_rules_read_missing_fields_as_defaults() {+ let current = sorted_rules([table("users")].iter()).unwrap();+ let mut stored = current.clone();+ stored[0].as_object_mut().unwrap().remove("lookup_query");+ assert!(same_rules::<ShardedTableConfig>(&stored, ¤t));++ let lookup = ShardedTableConfig {+ lookup_query: Some("SELECT shard FROM users".into()),+ ..table("users")+ };+ let current = sorted_rules([lookup].iter()).unwrap();+ assert!(!same_rules::<ShardedTableConfig>(&stored, ¤t));+ }++ #[test]+ fn centroids_read_back_from_json_text_report_no_change() {+ let vector = ShardedTableConfig {+ centroids: (1..=2000)+ .map(|value| vec![value as f32 / 7.0, value as f32 / 3.0].into())+ .collect(),+ ..table("embeddings")+ };+ let current = sorted_rules([vector].iter()).unwrap();+ let stored: Vec<Value> =+ serde_json::from_str(&serde_json::to_string(¤t).unwrap()).unwrap();+ assert!(same_rules::<ShardedTableConfig>(&stored, ¤t));+ }++ #[test]+ fn validate_equal_finds_every_changed_field() {+ let left = snapshot();+ let mut right = snapshot();+ for (cluster, shards) in [(&mut right.source, 2), (&mut right.destination, 3)] {+ cluster.shards = shards;+ cluster.sharded_tables = rules();+ cluster.sharded_mappings = rules();+ cluster.omnisharded_tables = rules();+ cluster.sharded_schemas = rules();+ }++ let fields = [+ ConfigField::Shards,+ ConfigField::ShardedTables,+ ConfigField::ShardedMappings,+ ConfigField::OmnishardedTables,+ ConfigField::ShardedSchemas,+ ];+ assert_eq!(+ left.validate_equal(&right),+ Err(fields+ .map(ConfigMismatch::Source)+ .into_iter()+ .chain(fields.map(ConfigMismatch::Destination))+ .collect())+ );+ }++ #[test]+ fn validate_equal_error_names_the_fields() {+ let left = snapshot();+ let mut right = snapshot();+ right.source.shards = 2;+ right.destination.sharded_tables = vec![];++ let error = RecordMismatch::Config(left.validate_equal(&right).unwrap_err());+ assert_eq!(+ error.to_string(),+ "sharding config changed since the resharding started: source.shards, destination.sharded_tables"+ );+ }+}diff --git a/pgdog/src/backend/replication/logical/run/store.rs b/pgdog/src/backend/replication/logical/run/store.rsnew file mode 100644index 000000000..4f1e275bb--- /dev/null+++ b/pgdog/src/backend/replication/logical/run/store.rs@@ -0,0 +1,604 @@+use std::fmt::Display;+use std::str::FromStr;+use std::time::{Duration, SystemTime, UNIX_EPOCH};++use pgdog_stats::{Databases, InstanceId, ReshardingId};+use serde::de::DeserializeOwned;+use tokio::sync::Mutex;++use super::super::{Error, subscriber::connect_address};+use super::queries::{+ CREATE_TABLE, DeleteRow, Heartbeat, InsertRow, ListRows, LockHolder, Query, ResumeRow,+ ResumeUpdate, SelectRow, SetupSession, StageUpdate, TableExists, TryLock, Unlock, UpdateStage,+};+use super::{FORMAT_VERSION, ReshardingKey, ReshardingRecord, Stage, TaskOwner};+use crate::backend::server::ServerRequest;+use crate::backend::{Error as BackendError, Server, pool::Address};+use crate::net::{DataRow, Format};+use crate::util::instance_id;++const APPLICATION_NAME_PREFIX: &str = "pgdog_resharding:";+const RELOCK_ATTEMPTS: usize = 20;+const RELOCK_DELAY: Duration = Duration::from_millis(250);++#[derive(Debug)]+pub(crate) struct ReshardingStore {+ server: Mutex<Option<Server>>,+ addr: Address,+ key: ReshardingKey,+}++impl ReshardingStore {+ pub(crate) async fn open(addr: &Address, key: ReshardingKey) -> Result<Self, Error> {+ let mut server = connect_session(addr).await?;+ if !try_lock(&mut server, &key).await? {+ return Err(locked_error(&mut server, &key).await);+ }++ Ok(Self {+ server: Mutex::new(Some(server)),+ addr: addr.clone(),+ key,+ })+ }++ pub(crate) async fn reconnect(&self) -> Result<(), Error> {+ let mut server = self.server.lock().await;+ *server = None;+ let mut session = connect_session(&self.addr).await?;+ for _ in 0..RELOCK_ATTEMPTS {+ if try_lock(&mut session, &self.key).await? {+ *server = Some(session);+ return Ok(());+ }+ tokio::time::sleep(RELOCK_DELAY).await;+ }+ Err(locked_error(&mut session, &self.key).await)+ }++ pub(crate) fn key(&self) -> &ReshardingKey {+ &self.key+ }++ pub(crate) fn address(&self) -> &Address {+ &self.addr+ }++ pub(crate) async fn load(&self) -> Result<Option<ReshardingRecord>, Error> {+ Self::read(session(&mut *self.server.lock().await)?, &self.key).await+ }++ pub(crate) async fn insert(&self, row: &ReshardingRecord) -> Result<(), Error> {+ self.execute(InsertRow::request(row)).await+ }++ pub(crate) async fn set_stage(&self, id: ReshardingId, stage: Stage) -> Result<(), Error> {+ self.execute(UpdateStage::request(StageUpdate { id, stage }))+ .await+ }++ pub(crate) async fn resume(+ &self,+ id: ReshardingId,+ owner: &TaskOwner,+ resumed_at: SystemTime,+ ) -> Result<(), Error> {+ self.execute(ResumeRow::request(ResumeUpdate {+ id,+ owner,+ resumed_at,+ }))+ .await+ }++ pub(crate) async fn delete(&self, id: ReshardingId) -> Result<(), Error> {+ self.execute(DeleteRow::request(id)).await+ }++ #[cfg(test)]+ pub(crate) async fn clear(&self) -> Result<(), Error> {+ self.execute(super::queries::DeleteKey::request(&self.key))+ .await+ }++ pub(crate) async fn heartbeat(&self) -> Result<(), Error> {+ self.execute(Heartbeat::request(())).await+ }++ pub(crate) async fn unlock(&self) -> Result<(), Error> {+ self.execute(Unlock::request(&self.key)).await+ }++ async fn execute(&self, request: impl Into<ServerRequest>) -> Result<(), Error> {+ session(&mut *self.server.lock().await)?+ .execute_checked(request)+ .await?;+ Ok(())+ }++ pub(crate) async fn read(+ server: &mut Server,+ key: &ReshardingKey,+ ) -> Result<Option<ReshardingRecord>, Error> {+ let exists = server.fetch_all::<i32>(TableExists::request(())).await?;+ if exists.first() != Some(&1) {+ return Ok(None);+ }++ let rows = server.fetch_all::<DataRow>(SelectRow::request(key)).await?;+ rows.first()+ .map(|row| parse_record(row, key.clone()))+ .transpose()+ }++ pub(super) async fn list(+ server: &mut Server,+ ) -> Result<Vec<Result<ReshardingRecord, UnreadableRecord>>, Error> {+ let exists = server.fetch_all::<i32>(TableExists::request(())).await?;+ if exists.first() != Some(&1) {+ return Ok(vec![]);+ }++ server+ .fetch_all::<DataRow>(ListRows::request(()))+ .await?+ .iter()+ .map(|row| {+ let key = parse_key(row)?;+ Ok(parse_record(row, key.clone()).map_err(|error| UnreadableRecord { key, error }))+ })+ .collect()+ }++ pub(crate) async fn lock_owner(+ server: &mut Server,+ key: &ReshardingKey,+ ) -> Result<Option<String>, Error> {+ let owner = server+ .fetch_all::<String>(LockHolder::request(key))+ .await?+ .into_iter()+ .next()+ .map(|name| match name.strip_prefix(APPLICATION_NAME_PREFIX) {+ Some(instance) => instance.to_owned(),+ None if name.is_empty() => "unknown".into(),+ None => name,+ });+ Ok(owner)+ }+}++fn session(server: &mut Option<Server>) -> Result<&mut Server, Error> {+ server+ .as_mut()+ .ok_or_else(|| Error::Backend(BackendError::NotConnected))+}++async fn try_lock(server: &mut Server, key: &ReshardingKey) -> Result<bool, Error> {+ let locked = server.fetch_all::<i32>(TryLock::request(key)).await?;+ Ok(locked.first() == Some(&1))+}++async fn connect_session(addr: &Address) -> Result<Server, Error> {+ let mut server = connect_address(addr).await?;+ let application_name = format!("{APPLICATION_NAME_PREFIX}{}", instance_id());+ server+ .execute_checked(SetupSession::request(&application_name))+ .await?;+ server.execute_checked(CREATE_TABLE).await?;+ Ok(server)+}++async fn locked_error(server: &mut Server, key: &ReshardingKey) -> Error {+ match ReshardingStore::lock_owner(server, key).await {+ Ok(owner) => Error::ReshardingLocked {+ key: key.to_string(),+ owner: owner.unwrap_or_else(|| "unknown".into()),+ },+ Err(err) => err,+ }+}++#[derive(Debug)]+pub(super) struct UnreadableRecord {+ pub(super) key: ReshardingKey,+ pub(super) error: Error,+}++fn text(row: &DataRow, index: usize) -> Result<String, Error> {+ row.get::<String>(index, Format::Text)+ .ok_or_else(|| Error::MissingData)+}++fn parsed<T>(row: &DataRow, index: usize, field: &str) -> Result<T, Error>+where+ T: FromStr,+ T::Err: Display,+{+ text(row, index)?+ .parse()+ .map_err(|err| Error::ReshardingFormat(format!("{field}: {err}")))+}++fn json<T: DeserializeOwned>(row: &DataRow, index: usize, field: &str) -> Result<T, Error> {+ serde_json::from_str(&text(row, index)?)+ .map_err(|err| Error::ReshardingFormat(format!("{field}: {err}")))+}++fn parse_key(row: &DataRow) -> Result<ReshardingKey, Error> {+ Ok(ReshardingKey {+ databases: Databases {+ source: text(row, 0)?,+ destination: text(row, 1)?,+ },+ publication: text(row, 2)?,+ })+}++fn parse_record(row: &DataRow, key: ReshardingKey) -> Result<ReshardingRecord, Error> {+ let format_version: i32 = parsed(row, 10, "format_version")?;+ if format_version > FORMAT_VERSION {+ return Err(Error::ReshardingFormat(format!(+ "format version {format_version} is newer than {FORMAT_VERSION}"+ )));+ }++ Ok(ReshardingRecord {+ id: parsed(row, 11, "id")?,+ key,+ command: json(row, 3, "command")?,+ slot_name: text(row, 4)?,+ stage: text(row, 5)?.parse()?,+ config: json(row, 6, "config")?,+ owner: TaskOwner {+ instance_id: InstanceId::from(text(row, 7)?),+ run_id: parsed(row, 8, "run_id")?,+ task_id: parsed(row, 9, "task_id")?,+ },+ format_version,+ started_at: millis(row, 12, "started_at")?,+ resumed_at: match text(row, 13)?.as_str() {+ "" => None,+ _ => Some(millis(row, 13, "resumed_at")?),+ },+ })+}++fn millis(row: &DataRow, index: usize, field: &str) -> Result<SystemTime, Error> {+ Ok(UNIX_EPOCH + Duration::from_millis(parsed::<i64>(row, index, field)?.max(0) as u64))+}++#[cfg(test)]+pub(super) mod test {+ use super::super::{ConfigSnapshot, ReshardingCommand};+ use super::*;+ use crate::backend::server::test::test_server;++ async fn backend_pid(store: &ReshardingStore) -> i32 {+ store+ .server+ .lock()+ .await+ .as_mut()+ .unwrap()+ .fetch_all::<i32>("SELECT pg_backend_pid()")+ .await+ .unwrap()[0]+ }++ pub(in super::super) fn key(name: &str) -> ReshardingKey {+ ReshardingKey {+ databases: Databases {+ source: format!("{name}_src"),+ destination: format!("{name}_dst"),+ },+ publication: "pub".into(),+ }+ }++ async fn listed_for(key: &ReshardingKey) -> Vec<Result<ReshardingRecord, UnreadableRecord>> {+ let mut server = test_server().await;+ ReshardingStore::list(&mut server)+ .await+ .unwrap()+ .into_iter()+ .filter(|row| match row {+ Ok(record) => record.key.databases == key.databases,+ Err(unreadable) => unreadable.key.databases == key.databases,+ })+ .collect()+ }++ fn now_millis() -> SystemTime {+ UNIX_EPOCH+ + Duration::from_millis(+ SystemTime::now()+ .duration_since(UNIX_EPOCH)+ .unwrap()+ .as_millis() as u64,+ )+ }++ pub(in super::super) fn row(key: ReshardingKey) -> ReshardingRecord {+ let config: ConfigSnapshot = serde_json::from_str(+ r#"{"source":{"shards":1,"sharded_tables":[],"sharded_mappings":[],"omnisharded_tables":[],"sharded_schemas":[]},+ "destination":{"shards":2,"sharded_tables":[],"sharded_mappings":[],"omnisharded_tables":[],"sharded_schemas":[]}}"#,+ )+ .unwrap();+ ReshardingRecord {+ id: ReshardingId::generate(),+ key,+ command: ReshardingCommand::Reshard {+ skip_schema_sync: false,+ replicate_only: false,+ sync_only: false,+ auto_cutover: true,+ tables_only: false,+ },+ slot_name: "slot".into(),+ stage: Stage::CopyTables,+ config,+ owner: TaskOwner::current(pgdog_stats::TaskId::new(7)),+ format_version: FORMAT_VERSION,+ started_at: now_millis() - Duration::from_secs(3600),+ resumed_at: None,+ }+ }++ async fn open_after_drop(addr: &Address, key: ReshardingKey) -> ReshardingStore {+ for _ in 0..50 {+ match ReshardingStore::open(addr, key.clone()).await {+ Ok(store) => return store,+ Err(Error::ReshardingLocked { .. }) => {+ tokio::time::sleep(std::time::Duration::from_millis(100)).await+ }+ Err(err) => panic!("{err}"),+ }+ }+ panic!("lock of {key} was not released");+ }++ #[tokio::test]+ async fn publications_have_separate_records() {+ let addr = Address::new_test();+ let first_key = key("store_pubs");+ let second_key = ReshardingKey {+ publication: "other_pub".into(),+ ..first_key.clone()+ };+ let first = ReshardingStore::open(&addr, first_key.clone())+ .await+ .unwrap();+ let second = ReshardingStore::open(&addr, second_key.clone())+ .await+ .unwrap();+ for store in [&first, &second] {+ store.clear().await.unwrap();+ }+ let first_row = row(first_key.clone());+ let second_row = row(second_key.clone());+ first.insert(&first_row).await.unwrap();+ second.insert(&second_row).await.unwrap();++ let listed = listed_for(&first_key)+ .await+ .into_iter()+ .map(|row| row.unwrap().key)+ .collect::<Vec<_>>();+ assert_eq!(listed, [second_key, first_key]);++ first.delete(first_row.id).await.unwrap();+ assert!(second.load().await.unwrap().is_some());+ second.delete(second_row.id).await.unwrap();+ }++ #[tokio::test]+ async fn newer_rows_are_reported_alone() {+ let addr = Address::new_test();+ let old_key = key("store_newer");+ let new_key = ReshardingKey {+ publication: "newer_pub".into(),+ ..old_key.clone()+ };+ let old = ReshardingStore::open(&addr, old_key.clone()).await.unwrap();+ let new = ReshardingStore::open(&addr, new_key.clone()).await.unwrap();+ for store in [&old, &new] {+ store.clear().await.unwrap();+ }+ old.insert(&row(old_key.clone())).await.unwrap();+ new.insert(&ReshardingRecord {+ format_version: FORMAT_VERSION + 1,+ ..row(new_key.clone())+ })+ .await+ .unwrap();++ assert!(matches!(new.load().await, Err(Error::ReshardingFormat(_))));++ let listed = listed_for(&old_key).await;+ assert_eq!(listed.len(), 2);+ assert!(+ matches!(&listed[0], Err(UnreadableRecord { key, error: Error::ReshardingFormat(_) }) if *key == new_key)+ );+ assert_eq!(listed[1].as_ref().unwrap().key, old_key);++ old.clear().await.unwrap();+ new.clear().await.unwrap();+ }++ #[tokio::test]+ async fn ddl_is_idempotent() {+ let addr = Address::new_test();+ let store = ReshardingStore::open(&addr, key("store_ddl"))+ .await+ .unwrap();+ drop(store);+ let store = open_after_drop(&addr, key("store_ddl")).await;+ assert_eq!(store.load().await.unwrap(), None);+ }++ #[tokio::test]+ async fn second_owner_is_refused() {+ let addr = Address::new_test();+ let first = ReshardingStore::open(&addr, key("store_owner"))+ .await+ .unwrap();++ let err = ReshardingStore::open(&addr, key("store_owner"))+ .await+ .unwrap_err();+ assert!(+ matches!(&err, Error::ReshardingLocked { owner, .. } if owner == instance_id()),+ "{err}"+ );++ drop(first);+ drop(open_after_drop(&addr, key("store_owner")).await);+ }++ #[tokio::test]+ async fn row_round_trips_and_lists() {+ let addr = Address::new_test();+ let key = key("store_rows");+ let store = ReshardingStore::open(&addr, key.clone()).await.unwrap();+ let row = row(key.clone());+ store.clear().await.unwrap();+ store.insert(&row).await.unwrap();+ assert!(store.insert(&row).await.is_err());++ store.set_stage(row.id, Stage::Replication).await.unwrap();+ let loaded = store.load().await.unwrap().unwrap();+ assert_eq!(loaded.stage, Stage::Replication);+ assert_eq!(+ ReshardingRecord {+ stage: Stage::CopyTables,+ ..loaded+ },+ row+ );++ let mut server = test_server().await;+ let listed = listed_for(&key).await;+ assert_eq!(listed.len(), 1);+ assert_eq!(listed[0].as_ref().unwrap().key, key);+ assert_eq!(+ ReshardingStore::lock_owner(&mut server, &key)+ .await+ .unwrap(),+ Some(instance_id().to_owned())+ );++ store.delete(row.id).await.unwrap();+ assert_eq!(store.load().await.unwrap(), None);+ drop(store);+ drop(open_after_drop(&addr, key).await);+ }++ #[tokio::test]+ async fn writes_with_an_old_id_keep_the_new_row() {+ let addr = Address::new_test();+ let key = key("store_old_id");+ let store = ReshardingStore::open(&addr, key.clone()).await.unwrap();+ store.clear().await.unwrap();+ let old = row(key.clone());+ store.insert(&old).await.unwrap();+ store.delete(old.id).await.unwrap();+ let new = row(key.clone());+ store.insert(&new).await.unwrap();++ store.set_stage(old.id, Stage::Replication).await.unwrap();+ store+ .resume(+ old.id,+ &TaskOwner::current(pgdog_stats::TaskId::new(9)),+ now_millis(),+ )+ .await+ .unwrap();+ store.delete(old.id).await.unwrap();+ assert_eq!(store.load().await.unwrap(), Some(new.clone()));++ store.delete(new.id).await.unwrap();+ assert_eq!(store.load().await.unwrap(), None);+ }++ #[tokio::test]+ async fn writes_fail_after_connection_loss() {+ let addr = Address::new_test();+ let key = key("store_loss");+ let store = ReshardingStore::open(&addr, key.clone()).await.unwrap();+ store.clear().await.unwrap();+ let row = row(key.clone());+ store.insert(&row).await.unwrap();++ let pid = backend_pid(&store).await;+ let mut server = test_server().await;+ server+ .execute_checked(format!("SELECT pg_terminate_backend({pid})"))+ .await+ .unwrap();++ assert!(store.set_stage(row.id, Stage::Replication).await.is_err());+ assert!(store.heartbeat().await.is_err());+ drop(store);++ let store = open_after_drop(&addr, key).await;+ assert_eq!(+ store.load().await.unwrap().unwrap().stage,+ Stage::CopyTables+ );+ store.delete(row.id).await.unwrap();+ }++ #[tokio::test]+ async fn failed_relock_keeps_writes_failing() {+ let addr = Address::new_test();+ let key = key("store_relock");+ let store = ReshardingStore::open(&addr, key.clone()).await.unwrap();+ store.clear().await.unwrap();+ let row = row(key.clone());+ store.insert(&row).await.unwrap();++ let pid = backend_pid(&store).await;+ let mut server = test_server().await;+ server+ .execute_checked(format!("SELECT pg_terminate_backend({pid})"))+ .await+ .unwrap();+ let other = open_after_drop(&addr, key.clone()).await;++ assert!(matches!(+ store.reconnect().await,+ Err(Error::ReshardingLocked { .. })+ ));+ assert!(store.delete(row.id).await.is_err());+ assert!(store.heartbeat().await.is_err());+ assert_eq!(other.load().await.unwrap(), Some(row.clone()));++ other.delete(row.id).await.unwrap();+ }++ #[tokio::test]+ async fn resume_records_the_new_owner_and_time() {+ let addr = Address::new_test();+ let key = key("store_resume");+ let store = ReshardingStore::open(&addr, key.clone()).await.unwrap();+ store.clear().await.unwrap();+ let row = row(key.clone());+ store.insert(&row).await.unwrap();++ let owner = TaskOwner::current(pgdog_stats::TaskId::new(8));+ let resumed_at = now_millis();+ store.resume(row.id, &owner, resumed_at).await.unwrap();+ assert_eq!(+ store.load().await.unwrap().unwrap(),+ ReshardingRecord {+ owner,+ resumed_at: Some(resumed_at),+ ..row+ }+ );++ store.delete(row.id).await.unwrap();+ }+}diff --git a/pgdog/src/backend/replication/logical/run/watchdog.rs b/pgdog/src/backend/replication/logical/run/watchdog.rsnew file mode 100644index 000000000..b0b34a8e4--- /dev/null+++ b/pgdog/src/backend/replication/logical/run/watchdog.rs@@ -0,0 +1,70 @@+use std::time::Duration;++use pgdog_stats::TaskId;+use tokio_util::task::AbortOnDropHandle;+use tracing::warn;++use super::super::resharding_state::ReshardingState;+use crate::api::tasks_storage;+use crate::tasks;+use crate::util::safe_timeout;++/// The heartbeat query checks the state connection.+const HEARTBEAT_INTERVAL: Duration = Duration::from_secs(5);+const HEARTBEAT_TIMEOUT: Duration = Duration::from_secs(10);+const HEARTBEAT_TIMEOUTS: u32 = 5;++/// Checks the state connection that holds the run lock while it lives,+/// and stops the run's root task when the connection, and with it the lock, is lost.+pub(crate) struct LockWatchdog {+ _heartbeat: Option<AbortOnDropHandle<()>>,+}++impl LockWatchdog {+ pub(crate) fn start(state: &ReshardingState, root_task: TaskId) -> Self {+ let heartbeat = state.run_store().map(|store| {+ AbortOnDropHandle::new(tasks::spawn("resharding lock watchdog", async move {+ loop {+ tokio::time::sleep(HEARTBEAT_INTERVAL).await;+ let Some(store) = store.upgrade() else {+ return;+ };+ let mut heartbeat = std::pin::pin!(store.heartbeat());+ let mut timeouts = 0;+ let failure = loop {+ match safe_timeout(HEARTBEAT_TIMEOUT, &mut heartbeat).await {+ Ok(Ok(())) => break None,+ Ok(Err(err)) => break Some(err.to_string()),+ Err(_) => {+ timeouts += 1;+ if timeouts == HEARTBEAT_TIMEOUTS {+ break Some(format!(+ "no reply in {:?}",+ HEARTBEAT_TIMEOUT * HEARTBEAT_TIMEOUTS+ ));+ }+ warn!(+ "resharding {}: heartbeat is slow, no reply in {:?}",+ store.key(),+ HEARTBEAT_TIMEOUT * timeouts+ );+ }+ }+ };+ if let Some(failure) = failure {+ warn!(+ "resharding {}: heartbeat failed ({failure}), stopping task {root_task}",+ store.key()+ );+ tasks_storage().cancel_task(root_task);+ return;+ }+ }+ }))+ });++ Self {+ _heartbeat: heartbeat,+ }+ }+}diff --git a/pgdog/src/backend/replication/logical/subscriber/mod.rs b/pgdog/src/backend/replication/logical/subscriber/mod.rsindex 25ca86cec..ed0fb42d9 100644--- a/pgdog/src/backend/replication/logical/subscriber/mod.rs+++ b/pgdog/src/backend/replication/logical/subscriber/mod.rs@@ -64,7 +64,7 @@ pub(super) async fn connect_primary(shard: &Shard) -> Result<Server, Error> {
}
}
-async fn connect_address(address: &Address) -> Result<Server, Error> {+pub(crate) async fn connect_address(address: &Address) -> Result<Server, Error> {
Ok(Box::pin(Server::connect(
address,
ServerOptions::default(),
diff --git a/pgdog/src/backend/replication/logical/subscriber/pipeline.rs b/pgdog/src/backend/replication/logical/subscriber/pipeline.rsindex bc799458d..fcd7a1164 100644--- a/pgdog/src/backend/replication/logical/subscriber/pipeline.rs+++ b/pgdog/src/backend/replication/logical/subscriber/pipeline.rs@@ -850,7 +850,7 @@ mod test {
let address = Address::new_test();
let origin = origin("test_progress_durable", &address);
origin.drop_origin().await.unwrap();
- origin.create_origin().await.unwrap();+ origin.ensure_exists().await.unwrap();
let mut session = test_server().await;
origin.setup_session(&mut session).await.unwrap();
diff --git a/pgdog/src/backend/replication/logical/subscriber/replication_origin.rs b/pgdog/src/backend/replication/logical/subscriber/replication_origin.rsindex c157f1e88..7bbd9ce32 100644--- a/pgdog/src/backend/replication/logical/subscriber/replication_origin.rs+++ b/pgdog/src/backend/replication/logical/subscriber/replication_origin.rs@@ -43,6 +43,10 @@ pub(crate) struct ReplicationOrigin {
retry_delay: Duration,
}
+pub(crate) fn origin_name(slot: &str, shard: usize) -> String {+ format!("__pgdog_origin_{slot}_{shard}")+}+
#[bon::bon]
impl ReplicationOrigin {
/// Origin for the stream of replication slot `name` on destination
@@ -56,7 +60,7 @@ impl ReplicationOrigin {
#[builder(default = RETRY_DELAY)] retry_delay: Duration,
) -> Self {
Self {
- name: quote_literal(&format!("__pgdog_origin_{name}_{shard}")),+ name: quote_literal(&origin_name(name, shard)),
address: address.clone(),
retry_attempts,
retry_delay,
@@ -68,8 +72,8 @@ impl ReplicationOrigin {
&self.name
}
- /// Create the origin if it does not exist.- pub(crate) async fn create_origin(&self) -> Result<(), Error> {+ /// Create the origin if it does not exist. Keeps the progress of an existing origin.+ pub(crate) async fn ensure_exists(&self) -> Result<(), Error> {
let mut server = connect_address(&self.address).await?;
let created = self.create_if_missing(&mut server).await?;
@@ -92,9 +96,24 @@ impl ReplicationOrigin {
/// Drop the origin and create it again, so the progress of an earlier
/// run is not reused.
- pub(crate) async fn recreate_origin(&self) -> Result<(), Error> {+ pub(crate) async fn recreate(&self) -> Result<(), Error> {
self.drop_if_exists().await?;
- self.create_origin().await+ self.ensure_exists().await+ }++ #[allow(dead_code)]+ pub(crate) async fn verify_exists(&self) -> Result<(), Error> {+ let mut server = connect_address(&self.address).await?;+ let name = &self.name;+ let found: Vec<String> = server+ .fetch_all(format!(+ "SELECT 'true' WHERE pg_replication_origin_oid({name}) IS NOT NULL"+ ))+ .await?;+ if found.is_empty() {+ return Err(Error::MissingReplicationOrigin(name.clone()));+ }+ Ok(())
}
/// Drop the origin. Waits for a short time while a closing session
@@ -298,7 +317,7 @@ mod test {
.retry_attempts(4)
.build();
origin.drop_origin().await.unwrap();
- origin.create_origin().await.unwrap();+ origin.ensure_exists().await.unwrap();
let table = format!("public.test_replication_origin_{name}");
admin
@@ -471,8 +490,8 @@ mod test {
#[tokio::test]
async fn create_and_drop_are_idempotent() {
let mut test = Test::new("lifecycle").await;
- test.origin.create_origin().await.unwrap();- test.origin.create_origin().await.unwrap();+ test.origin.ensure_exists().await.unwrap();+ test.origin.ensure_exists().await.unwrap();
assert_eq!(test.origins_count().await, 1);
assert_eq!(test.progress(true).await, Lsn::default());
@@ -632,7 +651,7 @@ mod test {
.address(&address)
.shard(0)
.build()
- .create_origin()+ .ensure_exists()
.await;
test.admin
diff --git a/pgdog/src/backend/replication/logical/subscriber/tests.rs b/pgdog/src/backend/replication/logical/subscriber/tests.rsindex 460412b61..d727cde06 100644--- a/pgdog/src/backend/replication/logical/subscriber/tests.rs+++ b/pgdog/src/backend/replication/logical/subscriber/tests.rs@@ -355,7 +355,7 @@ pub(super) async fn test_origins(cluster: &Cluster) -> Vec<ReplicationOrigin> {
.address(primary(pools).unwrap().addr())
.shard(shard)
.build();
- origin.recreate_origin().await.unwrap();+ origin.recreate().await.unwrap();
origins.push(origin);
}
origins
diff --git a/pgdog/src/backend/replication/tests.rs b/pgdog/src/backend/replication/tests.rsindex 1150107fe..5417b9457 100644--- a/pgdog/src/backend/replication/tests.rs+++ b/pgdog/src/backend/replication/tests.rs@@ -12,7 +12,7 @@ use super::logical::publisher::replication_progress::ReplicationProgress;
use super::logical::resharding_state::ReshardingState;
use crate::{
api::{
- MigrationError,+ ReshardingError,
copy_data::{CopyDataTask, TableDataSyncTask},
replication::{ReplicationClusterStop, ReplicationClusterTask, ReplicationTask},
resharding::ReshardTask,
@@ -798,7 +798,7 @@ async fn wait_for_reshard_status(
async fn copy_data_task(
schema: &str,
destination: &str,
-) -> Result<(TaskWaiter<(), MigrationError>, ReshardingState), Box<dyn std::error::Error>> {+) -> Result<(TaskWaiter<(), ReshardingError>, ReshardingState), Box<dyn std::error::Error>> {
let state = ReshardingState::builder()
.source(schema)
.destination(destination)
diff --git a/pgdog/src/backend/replication/tests/copy_data.rs b/pgdog/src/backend/replication/tests/copy_data.rsindex 91fec6ae9..9dc8faae5 100644--- a/pgdog/src/backend/replication/tests/copy_data.rs+++ b/pgdog/src/backend/replication/tests/copy_data.rs@@ -43,7 +43,7 @@ async fn failed_table_copy_fails_copy_data_and_cancels_the_other_copies()
})
.await?;
let states = subtree_progress(id);
- state.drop_slots_if_owned().await?;+ state.cancel_run().await?;
Ok::<_, Box<dyn std::error::Error>>((outcome, states))
}
diff --git a/pgdog/src/backend/schema/sync/pg_dump.rs b/pgdog/src/backend/schema/sync/pg_dump.rsindex 8e182a453..fb76c331e 100644--- a/pgdog/src/backend/schema/sync/pg_dump.rs+++ b/pgdog/src/backend/schema/sync/pg_dump.rs@@ -19,6 +19,7 @@ use regex::Regex;
use tracing::{info, trace, warn};
use super::SchemaSyncError;
+use crate::backend::replication::logical::run::RESHARDING_TABLE;
use crate::{
backend::{self, Cluster, pool::Request, replication::publisher::PublicationTable},
config::config,
@@ -125,6 +126,8 @@ fn build_pg_dump_command(
command
.kill_on_drop(true)
.arg("--schema-only")
+ .arg("--exclude-table")+ .arg(table_pattern("pgdog", RESHARDING_TABLE))
.arg("-h")
.arg(&addr.host)
.arg("-p")
diff --git a/pgdog/src/cli.rs b/pgdog/src/cli.rsindex 8b98bf3db..7659a044e 100644--- a/pgdog/src/cli.rs+++ b/pgdog/src/cli.rs@@ -3,17 +3,18 @@ use std::path::PathBuf;
use clap::{Parser, Subcommand};
use tracing::info;
+use crate::api::abort_resharding::AbortReshardingTask;
use crate::api::resharding::ReshardTask;
use crate::api::schema_sync::{SchemaSyncPhase, SchemaSyncTask};
use crate::api::task::TaskError;
-use crate::api::{MigrationError, run_task};+use crate::api::{ReshardingError, run_task};
use crate::backend::databases::databases;
use crate::backend::replication::logical::Error::ReplicationAborted;
use crate::backend::replication::resharding_state::ReshardingState;
use crate::backend::schema::sync::config::ShardConfig;
use crate::frontend::comms::comms;
use crate::frontend::router::cli::RouterCli;
-use pgdog_stats::Databases;+use pgdog_stats::{CommandKind, Databases};
/// PgDog is a PostgreSQL pooler, proxy, load balancer and query router.
#[derive(Parser, Debug)]
@@ -111,6 +112,21 @@ pub(crate) enum Commands {
/// Don't perform pre-data or post-data schema sync.
#[arg(long)]
skip_schema_sync: bool,
++ /// Drop the replication slots that the run created, and all its origins,+ /// when the run stops before it ends. Slots that existed before the run+ /// stay. With `false`, the run keeps them and can be aborted later.+ #[arg(long, default_value_t = true, action = clap::ArgAction::Set)]+ cleanup_on_cancel: bool,++ /// Abort the unfinished resharding from source to destination: drop+ /// every slot with its slot name, also slots that existed before the+ /// run, all its origins and its state.+ #[arg(+ long,+ conflicts_with_all = ["cleanup_on_cancel", "replicate_only", "tables_only", "sync_only", "skip_schema_sync"]+ )]+ abort: bool,
},
/// Copy schema from source to destination cluster.
@@ -241,15 +257,31 @@ pub(crate) async fn data_sync(commands: Commands) -> Result<(), Box<dyn std::err
sync_only,
replication_slot,
skip_schema_sync,
+ cleanup_on_cancel,+ abort,
} = commands
{
let state = ReshardingState::builder()
.source(&from_database)
.destination(&to_database)
.publication(&publication)
- .maybe_replication_slot(replication_slot)+ .maybe_replication_slot(replication_slot.clone())+ .cleanup_on_cancel(cleanup_on_cancel)
.build()?;
+ if abort {+ run_task(+ Box::pin(AbortReshardingTask::checked(+ state,+ CommandKind::Reshard,+ replication_slot,+ ))+ .await?,+ )+ .await?;+ return Ok(());+ }+
let result = run_task(
ReshardTask::builder()
.state(state)
@@ -261,7 +293,7 @@ pub(crate) async fn data_sync(commands: Commands) -> Result<(), Box<dyn std::err
)
.await;
- if let Err(TaskError::Failed(MigrationError::Replication(ReplicationAborted))) = result+ if let Err(TaskError::Failed(ReshardingError::Replication(ReplicationAborted))) = result
&& comms().offline()
{
// pgdog is shutting down, that's why the replication process is aborted.
@@ -384,6 +416,49 @@ mod test {
assert_eq!(tables_only, expected, "{args:?}");
}
}
++ #[test]+ fn test_cleanup_on_cancel_and_abort() {+ for (args, cleanup, abort) in [+ (vec!["--publication", "p"], true, false),+ (+ vec!["--publication", "p", "--cleanup-on-cancel", "false"],+ false,+ false,+ ),+ (+ vec!["--publication", "p", "--cleanup-on-cancel=true"],+ true,+ false,+ ),+ (vec!["--publication", "p", "--abort"], true, true),+ ] {+ let Commands::DataSync {+ cleanup_on_cancel,+ abort: parsed_abort,+ ..+ } = parse(&args).unwrap()+ else {+ panic!("not a data sync command");+ };+ assert_eq!(cleanup_on_cancel, cleanup, "{args:?}");+ assert_eq!(parsed_abort, abort, "{args:?}");+ }++ for args in [+ vec![+ "--publication",+ "p",+ "--abort",+ "--cleanup-on-cancel",+ "false",+ ],+ vec!["--publication", "p", "--abort", "--replicate-only"],+ vec!["--publication", "p", "--abort", "--tables-only"],+ ] {+ assert!(parse(&args).is_err(), "{args:?}");+ }+ }
}
mod schema_sync {