public inbox for [email protected]
help / color / mirror / Atom feed[PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test.
77+ messages / 14 participants
[nested] [flat]
* [PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test.
@ 2018-12-19 03:43 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Kyotaro Horiguchi @ 2018-12-19 03:43 UTC (permalink / raw)
It is convenient that priary_slot_name can be specified on taking a
base backup. This adds a new parameter of the name to the perl
function.
---
src/test/perl/PostgresNode.pm | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 6019f37f91..c7e138c121 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -694,6 +694,10 @@ port = $port
$self->append_conf('postgresql.conf',
"unix_socket_directories = '$host'");
}
+ $self->append_conf('postgresql.conf',
+ qq(primary_slot_name = $params{primary_slot_name}))
+ if (defined $params{primary_slot_name});
+
$self->enable_streaming($root_node) if $params{has_streaming};
$self->enable_restoring($root_node) if $params{has_restoring};
return;
--
2.16.3
----Next_Part(Tue_Jul_30_21_30_45_2019_680)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v15-0004-TAP-test-for-the-slot-limit-feature.patch"
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test.
@ 2018-12-19 03:43 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Kyotaro Horiguchi @ 2018-12-19 03:43 UTC (permalink / raw)
It is convenient that priary_slot_name can be specified on taking a
base backup. This adds a new parameter of the name to the perl
function.
---
src/test/perl/PostgresNode.pm | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 8a2c6fc122..daca2e0085 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -672,11 +672,11 @@ sub init_from_backup
chmod(0700, $data_path);
# Base configuration for this node
- $self->append_conf(
- 'postgresql.conf',
- qq(
-port = $port
-));
+ $self->append_conf('postgresql.conf', qq(port = $port));
+ $self->append_conf('postgresql.conf',
+ qq(primary_slot_name = $params{primary_slot_name}))
+ if (defined $params{primary_slot_name});
+
$self->enable_streaming($root_node) if $params{has_streaming};
$self->enable_restoring($root_node) if $params{has_restoring};
return;
--
2.16.3
----Next_Part(Fri_Feb_22_14_12_28_2019_076)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v13-0004-TAP-test-for-the-slot-limit-feature.patch"
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test.
@ 2018-12-19 03:43 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Kyotaro Horiguchi @ 2018-12-19 03:43 UTC (permalink / raw)
It is convenient that priary_slot_name can be specified on taking a
base backup. This adds a new parameter of the name to the perl
function.
---
src/test/perl/PostgresNode.pm | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 6019f37f91..c7e138c121 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -694,6 +694,10 @@ port = $port
$self->append_conf('postgresql.conf',
"unix_socket_directories = '$host'");
}
+ $self->append_conf('postgresql.conf',
+ qq(primary_slot_name = $params{primary_slot_name}))
+ if (defined $params{primary_slot_name});
+
$self->enable_streaming($root_node) if $params{has_streaming};
$self->enable_restoring($root_node) if $params{has_restoring};
return;
--
2.16.3
----Next_Part(Wed_Jul_31_16_56_16_2019_668)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v16-0004-TAP-test-for-the-slot-limit-feature.patch"
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test.
@ 2018-12-19 03:43 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Kyotaro Horiguchi @ 2018-12-19 03:43 UTC (permalink / raw)
It is convenient that priary_slot_name can be specified on taking a
base backup. This adds a new parameter of the name to the perl
function.
---
src/test/perl/PostgresNode.pm | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 6019f37f91..2c7a894914 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -680,11 +680,7 @@ sub init_from_backup
chmod(0700, $data_path);
# Base configuration for this node
- $self->append_conf(
- 'postgresql.conf',
- qq(
-port = $port
-));
+ $self->append_conf('postgresql.conf', qq(port = $port));
if ($use_tcp)
{
$self->append_conf('postgresql.conf', "listen_addresses = '$host'");
@@ -694,6 +690,11 @@ port = $port
$self->append_conf('postgresql.conf',
"unix_socket_directories = '$host'");
}
+ $self->append_conf('postgresql.conf', qq(port = $port));
+ $self->append_conf('postgresql.conf',
+ qq(primary_slot_name = $params{primary_slot_name}))
+ if defined $params{primary_slot_name};
+
$self->enable_streaming($root_node) if $params{has_streaming};
$self->enable_restoring($root_node) if $params{has_restoring};
return;
--
2.20.1
--MP_/68vU=6Ib.yoYOTdA.M=EVne
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v14-0004-TAP-test-for-the-slot-limit-feature.patch
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v19 1/3] Add primary_slot_name to init_from_backup in TAP test.
@ 2018-12-19 03:43 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Kyotaro Horiguchi @ 2018-12-19 03:43 UTC (permalink / raw)
It is convenient that priary_slot_name can be specified on taking a
base backup. This adds a new parameter of the name to the perl
function.
---
src/test/perl/PostgresNode.pm | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 1d5450758e..b80305b6c5 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -702,6 +702,11 @@ port = $port
$self->append_conf('postgresql.conf',
"unix_socket_directories = '$host'");
}
+ if (defined $params{primary_slot_name})
+ {
+ $self->append_conf('postgresql.conf',
+ "primary_slot_name = $params{primary_slot_name}");
+ }
$self->enable_streaming($root_node) if $params{has_streaming};
$self->enable_restoring($root_node, $params{standby}) if $params{has_restoring};
return;
--
2.20.1
--d6Gm4EdcadzBjdND
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v19-0002-Add-WAL-relief-vent-for-replication-slots.patch"
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v30 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index c344ff0944..364e3a53f7 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3761,6 +3761,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3826,6 +3830,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3861,6 +3866,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3910,6 +3916,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4787,6 +4794,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4997,6 +5043,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5157,19 +5204,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5186,6 +5233,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 8a5a9fe642..6718514d34 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Mon__4_Mar_2024_11_58_46_+0900_UaponF/qQhQrVCFt
Content-Type: text/x-diff;
name="v30-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v30-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v32 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 84494c4b81..0424ab9a29 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3729,6 +3729,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3794,6 +3798,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3829,6 +3834,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3878,6 +3884,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4755,6 +4762,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4965,6 +5011,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5125,19 +5172,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5154,6 +5201,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index cb968d03ec..768d6e1c0b 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Sun__31_Mar_2024_22_59_31_+0900_msknEviJj08_wgqO
Content-Type: text/x-diff;
name="v32-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v32-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v37 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 78 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index b87b4b40d07..49fe531198a 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3819,6 +3819,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3884,6 +3888,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3932,6 +3937,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/* "old" transition table for UPDATE/DELETE, if any */
@@ -3978,6 +3984,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4873,6 +4880,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -5113,6 +5159,7 @@ AfterTriggerBeginXact(void)
afterTriggers.firing_depth = 0;
afterTriggers.batch_callbacks = NIL;
afterTriggers.firing_batch_callbacks = false;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5287,11 +5334,11 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5309,8 +5356,33 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
qs->tables = NIL;
list_free_deep(tables);
- list_free_deep(qs->batch_callbacks);
- qs->batch_callbacks = NIL;
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 1d9869973c0..e5dd915096d 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -269,6 +269,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.43.0
--Multipart=_Fri__29_May_2026_23_14_17_+0900_Te0o73X2VqYK57Gd
Content-Type: text/x-diff;
name="v37-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v37-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v29 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 52177759ab..00b20f4b5b 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3742,6 +3742,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3807,6 +3811,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3842,6 +3847,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3891,6 +3897,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4768,6 +4775,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4978,6 +5024,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5138,19 +5185,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5167,6 +5214,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 430e3ca7dd..48a21c4c51 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Mon__28_Aug_2023_16_05_30_+0900_b1OvQD_3A3ZMTGvj
Content-Type: text/x-diff;
name="v29-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v29-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v39 3/8] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 78 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 401baddbfc6..dc746bccc8e 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3844,6 +3844,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3909,6 +3913,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3957,6 +3962,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/* "old" transition table for UPDATE/DELETE, if any */
@@ -4003,6 +4009,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4898,6 +4905,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -5138,6 +5184,7 @@ AfterTriggerBeginXact(void)
afterTriggers.firing_depth = 0;
afterTriggers.batch_callbacks = NIL;
afterTriggers.firing_batch_callbacks = false;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5312,11 +5359,11 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5334,8 +5381,33 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
qs->tables = NIL;
list_free_deep(tables);
- list_free_deep(qs->batch_callbacks);
- qs->batch_callbacks = NIL;
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 0c3d485abf4..f0f052cc759 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -269,6 +269,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.43.0
--Multipart=_Fri__3_Jul_2026_19_11_16_+0900_CskxSIu_iGcDMEyM
Content-Type: text/x-diff;
name="v39-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v39-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v31 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 84494c4b81..0424ab9a29 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3729,6 +3729,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3794,6 +3798,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3829,6 +3834,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3878,6 +3884,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4755,6 +4762,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4965,6 +5011,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5125,19 +5172,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5154,6 +5201,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index cb968d03ec..768d6e1c0b 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Fri__29_Mar_2024_23_47_00_+0900_KGpmmDOIs1266Ib1
Content-Type: text/x-diff;
name="v31-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v31-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v24 04/15] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 +++++++++++++++++++++++++++++++++-
src/include/commands/trigger.h | 2 +
2 files changed, 80 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index d8890d2c74..c7a2bc3ed8 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3570,6 +3570,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3635,6 +3639,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3670,6 +3675,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
Tuplestorestate *old_tuplestore; /* "old" transition table, if any */
Tuplestorestate *new_tuplestore; /* "new" transition table, if any */
@@ -4448,6 +4454,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4642,6 +4687,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -4802,11 +4848,29 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
if (table->storeslot)
ExecDropSingleTupleTableSlot(table->storeslot);
}
@@ -4818,6 +4882,18 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 9ef7f6d768..0cfb0b3c5c 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -251,6 +251,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.17.1
--Multipart=_Thu__23_Sep_2021_04_57_30_+0900_b5pmgR1N8oMaMz.U
Content-Type: text/x-diff;
name="v24-0005-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v24-0005-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v25 04/15] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 +++++++++++++++++++++++++++++++++-
src/include/commands/trigger.h | 2 +
2 files changed, 80 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 1a9c1ac290..0fcb1aa0b5 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3651,6 +3651,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3716,6 +3720,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3751,6 +3756,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
Tuplestorestate *old_tuplestore; /* "old" transition table, if any */
Tuplestorestate *new_tuplestore; /* "new" transition table, if any */
@@ -4529,6 +4535,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4723,6 +4768,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -4883,11 +4929,29 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
if (table->storeslot)
ExecDropSingleTupleTableSlot(table->storeslot);
}
@@ -4899,6 +4963,18 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index e1271420e5..697a6f6d55 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -255,6 +255,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.17.1
--Multipart=_Fri__4_Feb_2022_01_48_06_+0900_N8BNZpfOR27sWrgY
Content-Type: text/x-diff;
name="v25-0003-Add-new-deptype-option-m-in-pg_depend-system-cat.patch"
Content-Disposition: attachment;
filename="v25-0003-Add-new-deptype-option-m-in-pg_depend-system-cat.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v30 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index c344ff0944..364e3a53f7 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3761,6 +3761,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3826,6 +3830,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3861,6 +3866,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3910,6 +3916,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4787,6 +4794,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4997,6 +5043,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5157,19 +5204,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5186,6 +5233,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 8a5a9fe642..6718514d34 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Mon__4_Mar_2024_11_58_46_+0900_UaponF/qQhQrVCFt
Content-Type: text/x-diff;
name="v30-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v30-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v23 04/15] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 +++++++++++++++++++++++++++++++++-
src/include/commands/trigger.h | 2 +
2 files changed, 80 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index fc0a4b2fa7..049007a484 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3570,6 +3570,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3635,6 +3639,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3670,6 +3675,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
Tuplestorestate *old_tuplestore; /* "old" transition table, if any */
Tuplestorestate *new_tuplestore; /* "new" transition table, if any */
@@ -4448,6 +4454,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4642,6 +4687,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -4802,11 +4848,29 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
if (table->storeslot)
ExecDropSingleTupleTableSlot(table->storeslot);
}
@@ -4818,6 +4882,18 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 9ef7f6d768..0cfb0b3c5c 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -251,6 +251,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.17.1
--Multipart=_Mon__2_Aug_2021_15_28_34_+0900_wlHCjIpnD/FrGAKu
Content-Type: text/x-diff;
name="v23-0005-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v23-0005-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v26 04/10] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 +++++++++++++++++++++++++++++++++-
src/include/commands/trigger.h | 2 +
2 files changed, 80 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index e08bd9a370..adc853a570 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3651,6 +3651,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3716,6 +3720,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3751,6 +3756,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
Tuplestorestate *old_tuplestore; /* "old" transition table, if any */
Tuplestorestate *new_tuplestore; /* "new" transition table, if any */
@@ -4539,6 +4545,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4733,6 +4778,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -4893,11 +4939,29 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
if (table->storeslot)
ExecDropSingleTupleTableSlot(table->storeslot);
}
@@ -4909,6 +4973,18 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index e1271420e5..697a6f6d55 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -255,6 +255,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.17.1
--Multipart=_Mon__14_Mar_2022_19_12_17_+0900_E9HVp8j5QFkIIek.
Content-Type: text/x-diff;
name="v26-0003-Add-new-deptype-option-m-in-pg_depend-system-cat.patch"
Content-Disposition: attachment;
filename="v26-0003-Add-new-deptype-option-m-in-pg_depend-system-cat.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v27 3/9] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 13cb516752..aa1da7b279 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3730,6 +3730,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3795,6 +3799,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3830,6 +3835,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3879,6 +3885,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState state);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4725,6 +4732,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4933,6 +4979,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5093,19 +5140,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
ExecDropSingleTupleTableSlot(table->storeslot);
}
@@ -5117,6 +5164,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 194e8d5bc1..2e9f1ac8c7 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -261,6 +261,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.17.1
--Multipart=_Fri__22_Apr_2022_14_58_01_+0900_MN3L/o2YUVF2g4zw
Content-Type: text/x-diff;
name="v27-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v27-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v29 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 52177759ab..00b20f4b5b 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3742,6 +3742,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3807,6 +3811,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3842,6 +3847,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3891,6 +3897,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4768,6 +4775,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4978,6 +5024,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5138,19 +5185,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5167,6 +5214,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 430e3ca7dd..48a21c4c51 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Mon__28_Aug_2023_16_05_30_+0900_b1OvQD_3A3ZMTGvj
Content-Type: text/x-diff;
name="v29-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v29-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v39 3/8] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 78 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 401baddbfc6..dc746bccc8e 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3844,6 +3844,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3909,6 +3913,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3957,6 +3962,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/* "old" transition table for UPDATE/DELETE, if any */
@@ -4003,6 +4009,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4898,6 +4905,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -5138,6 +5184,7 @@ AfterTriggerBeginXact(void)
afterTriggers.firing_depth = 0;
afterTriggers.batch_callbacks = NIL;
afterTriggers.firing_batch_callbacks = false;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5312,11 +5359,11 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5334,8 +5381,33 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
qs->tables = NIL;
list_free_deep(tables);
- list_free_deep(qs->batch_callbacks);
- qs->batch_callbacks = NIL;
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 0c3d485abf4..f0f052cc759 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -269,6 +269,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.43.0
--Multipart=_Fri__3_Jul_2026_19_11_16_+0900_CskxSIu_iGcDMEyM
Content-Type: text/x-diff;
name="v39-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v39-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v38 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 78 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index b87b4b40d07..49fe531198a 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3819,6 +3819,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3884,6 +3888,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3932,6 +3937,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/* "old" transition table for UPDATE/DELETE, if any */
@@ -3978,6 +3984,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4873,6 +4880,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -5113,6 +5159,7 @@ AfterTriggerBeginXact(void)
afterTriggers.firing_depth = 0;
afterTriggers.batch_callbacks = NIL;
afterTriggers.firing_batch_callbacks = false;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5287,11 +5334,11 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5309,8 +5356,33 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
qs->tables = NIL;
list_free_deep(tables);
- list_free_deep(qs->batch_callbacks);
- qs->batch_callbacks = NIL;
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 0c3d485abf4..f0f052cc759 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -269,6 +269,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.43.0
--Multipart=_Wed__1_Jul_2026_00_04_01_+0900_OVSy2WWK_9aByzDJ
Content-Type: text/x-diff;
name="v38-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v38-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v30 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index c344ff0944..364e3a53f7 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3761,6 +3761,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3826,6 +3830,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3861,6 +3866,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3910,6 +3916,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4787,6 +4794,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4997,6 +5043,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5157,19 +5204,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5186,6 +5233,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 8a5a9fe642..6718514d34 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Mon__4_Mar_2024_11_58_46_+0900_UaponF/qQhQrVCFt
Content-Type: text/x-diff;
name="v30-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v30-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v38 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 78 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index b87b4b40d07..49fe531198a 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3819,6 +3819,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3884,6 +3888,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3932,6 +3937,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/* "old" transition table for UPDATE/DELETE, if any */
@@ -3978,6 +3984,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4873,6 +4880,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -5113,6 +5159,7 @@ AfterTriggerBeginXact(void)
afterTriggers.firing_depth = 0;
afterTriggers.batch_callbacks = NIL;
afterTriggers.firing_batch_callbacks = false;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5287,11 +5334,11 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5309,8 +5356,33 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
qs->tables = NIL;
list_free_deep(tables);
- list_free_deep(qs->batch_callbacks);
- qs->batch_callbacks = NIL;
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 0c3d485abf4..f0f052cc759 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -269,6 +269,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.43.0
--Multipart=_Wed__1_Jul_2026_00_04_01_+0900_OVSy2WWK_9aByzDJ
Content-Type: text/x-diff;
name="v38-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v38-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v38 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 78 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index b87b4b40d07..49fe531198a 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3819,6 +3819,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3884,6 +3888,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3932,6 +3937,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/* "old" transition table for UPDATE/DELETE, if any */
@@ -3978,6 +3984,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4873,6 +4880,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -5113,6 +5159,7 @@ AfterTriggerBeginXact(void)
afterTriggers.firing_depth = 0;
afterTriggers.batch_callbacks = NIL;
afterTriggers.firing_batch_callbacks = false;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5287,11 +5334,11 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5309,8 +5356,33 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
qs->tables = NIL;
list_free_deep(tables);
- list_free_deep(qs->batch_callbacks);
- qs->batch_callbacks = NIL;
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 0c3d485abf4..f0f052cc759 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -269,6 +269,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.43.0
--Multipart=_Wed__1_Jul_2026_00_04_01_+0900_OVSy2WWK_9aByzDJ
Content-Type: text/x-diff;
name="v38-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v38-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v37 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 78 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index b87b4b40d07..49fe531198a 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3819,6 +3819,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3884,6 +3888,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3932,6 +3937,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/* "old" transition table for UPDATE/DELETE, if any */
@@ -3978,6 +3984,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4873,6 +4880,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -5113,6 +5159,7 @@ AfterTriggerBeginXact(void)
afterTriggers.firing_depth = 0;
afterTriggers.batch_callbacks = NIL;
afterTriggers.firing_batch_callbacks = false;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5287,11 +5334,11 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5309,8 +5356,33 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
qs->tables = NIL;
list_free_deep(tables);
- list_free_deep(qs->batch_callbacks);
- qs->batch_callbacks = NIL;
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 1d9869973c0..e5dd915096d 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -269,6 +269,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.43.0
--Multipart=_Fri__29_May_2026_23_14_17_+0900_Te0o73X2VqYK57Gd
Content-Type: text/x-diff;
name="v37-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v37-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v27 3/9] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 13cb516752..aa1da7b279 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3730,6 +3730,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3795,6 +3799,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3830,6 +3835,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3879,6 +3885,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState state);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4725,6 +4732,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4933,6 +4979,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5093,19 +5140,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
ExecDropSingleTupleTableSlot(table->storeslot);
}
@@ -5117,6 +5164,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 194e8d5bc1..2e9f1ac8c7 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -261,6 +261,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.17.1
--Multipart=_Fri__22_Apr_2022_11_29_39_+0900_ZOAC7UMt5e8j1Nvx
Content-Type: text/x-diff;
name="v27-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v27-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v28 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 4b295f8da5..354ea94777 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3742,6 +3742,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3807,6 +3811,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3842,6 +3847,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3891,6 +3897,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4737,6 +4744,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4947,6 +4993,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5107,19 +5154,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5136,6 +5183,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 430e3ca7dd..48a21c4c51 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Thu__1_Jun_2023_23_59_09_+0900_/G5+8nG46.f1T42K
Content-Type: text/x-diff;
name="v28-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v28-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v29 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 52177759ab..00b20f4b5b 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3742,6 +3742,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3807,6 +3811,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3842,6 +3847,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3891,6 +3897,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4768,6 +4775,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4978,6 +5024,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5138,19 +5185,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5167,6 +5214,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 430e3ca7dd..48a21c4c51 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Mon__28_Aug_2023_11_52_52_+0900_hj6L5h176QaSGtg7
Content-Type: text/x-diff;
name="v29-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v29-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v29 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 52177759ab..00b20f4b5b 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3742,6 +3742,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3807,6 +3811,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3842,6 +3847,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3891,6 +3897,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4768,6 +4775,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4978,6 +5024,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5138,19 +5185,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5167,6 +5214,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 430e3ca7dd..48a21c4c51 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Mon__28_Aug_2023_11_52_52_+0900_hj6L5h176QaSGtg7
Content-Type: text/x-diff;
name="v29-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v29-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v24 04/15] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 +++++++++++++++++++++++++++++++++-
src/include/commands/trigger.h | 2 +
2 files changed, 80 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index d8890d2c74..c7a2bc3ed8 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3570,6 +3570,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3635,6 +3639,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3670,6 +3675,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
Tuplestorestate *old_tuplestore; /* "old" transition table, if any */
Tuplestorestate *new_tuplestore; /* "new" transition table, if any */
@@ -4448,6 +4454,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4642,6 +4687,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -4802,11 +4848,29 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ {
+ if (table->prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
+ }
if (table->storeslot)
ExecDropSingleTupleTableSlot(table->storeslot);
}
@@ -4818,6 +4882,18 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 9ef7f6d768..0cfb0b3c5c 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -251,6 +251,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.17.1
--Multipart=_Fri__29_Oct_2021_18_16_28_+0900_jlYRKjywLqhZ7oyk
Content-Type: text/x-diff;
name="v24-0005-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v24-0005-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v37 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 78 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index b87b4b40d07..49fe531198a 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3819,6 +3819,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3884,6 +3888,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3932,6 +3937,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/* "old" transition table for UPDATE/DELETE, if any */
@@ -3978,6 +3984,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4873,6 +4880,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -5113,6 +5159,7 @@ AfterTriggerBeginXact(void)
afterTriggers.firing_depth = 0;
afterTriggers.batch_callbacks = NIL;
afterTriggers.firing_batch_callbacks = false;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5287,11 +5334,11 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5309,8 +5356,33 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
qs->tables = NIL;
list_free_deep(tables);
- list_free_deep(qs->batch_callbacks);
- qs->batch_callbacks = NIL;
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 1d9869973c0..e5dd915096d 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -269,6 +269,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.43.0
--Multipart=_Fri__29_May_2026_23_14_17_+0900_Te0o73X2VqYK57Gd
Content-Type: text/x-diff;
name="v37-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v37-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v30 03/11] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 83 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index c344ff0944..364e3a53f7 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3761,6 +3761,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3826,6 +3830,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3861,6 +3866,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/*
@@ -3910,6 +3916,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4787,6 +4794,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -4997,6 +5043,7 @@ AfterTriggerBeginXact(void)
*/
afterTriggers.firing_counter = (CommandId) 1; /* mustn't be 0 */
afterTriggers.query_depth = -1;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5157,19 +5204,19 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_upd_tuplestore;
table->old_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_upd_tuplestore;
table->new_upd_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->old_del_tuplestore;
table->old_del_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_ins_tuplestore;
table->new_ins_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5186,6 +5233,34 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
*/
qs->tables = NIL;
list_free_deep(tables);
+
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 8a5a9fe642..6718514d34 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -265,6 +265,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.25.1
--Multipart=_Mon__4_Mar_2024_11_58_46_+0900_UaponF/qQhQrVCFt
Content-Type: text/x-diff;
name="v30-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Disposition: attachment;
filename="v30-0004-Add-Incremental-View-Maintenance-support-to-pg_d.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* [PATCH v39 3/8] Allow to prolong life span of transition tables until transaction end
@ 2019-12-20 01:09 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:09 UTC (permalink / raw)
Originally, tuplestores of AFTER trigger's transition tables were
freed for each query depth. For our IVM implementation, we would like
to prolong life of the tuplestores because we have to preserve them
for a whole query assuming that some base tables might be changed
in some trigger functions.
---
src/backend/commands/trigger.c | 80 ++++++++++++++++++++++++++++++++--
src/include/commands/trigger.h | 2 +
2 files changed, 78 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 401baddbfc6..dc746bccc8e 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3844,6 +3844,10 @@ typedef struct AfterTriggerEventList
* end of the list, so it is relatively easy to discard them. The event
* list chunks themselves are stored in event_cxt.
*
+ * prolonged_tuplestored is a list of transition table tuplestores whose
+ * life are prolonged to the end of the outmost query instead of each nested
+ * query.
+ *
* query_depth is the current depth of nested AfterTriggerBeginQuery calls
* (-1 when the stack is empty).
*
@@ -3909,6 +3913,7 @@ typedef struct AfterTriggersData
SetConstraintState state; /* the active S C state */
AfterTriggerEventList events; /* deferred-event list */
MemoryContext event_cxt; /* memory context for events, if any */
+ List *prolonged_tuplestores; /* list of prolonged tuplestores */
/* per-query-level data: */
AfterTriggersQueryData *query_stack; /* array of structs shown below */
@@ -3957,6 +3962,7 @@ struct AfterTriggersTableData
bool closed; /* true when no longer OK to add tuples */
bool before_trig_done; /* did we already queue BS triggers? */
bool after_trig_done; /* did we already queue AS triggers? */
+ bool prolonged; /* are transition tables prolonged? */
AfterTriggerEventList after_trig_events; /* if so, saved list pointer */
/* "old" transition table for UPDATE/DELETE, if any */
@@ -4003,6 +4009,7 @@ static void TransitionTableAddTuple(EState *estate,
TupleTableSlot *original_insert_tuple,
Tuplestorestate *tuplestore);
static void AfterTriggerFreeQuery(AfterTriggersQueryData *qs);
+static void release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged);
static SetConstraintState SetConstraintStateCreate(int numalloc);
static SetConstraintState SetConstraintStateCopy(SetConstraintState origstate);
static SetConstraintState SetConstraintStateAddItem(SetConstraintState state,
@@ -4898,6 +4905,45 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
}
+/*
+ * SetTransitionTablePreserved
+ *
+ * Prolong lifespan of transition tables corresponding specified relid and
+ * command type to the end of the outmost query instead of each nested query.
+ * This enables to use nested AFTER trigger's transition tables from outer
+ * query's triggers. Currently, only immediate incremental view maintenance
+ * uses this.
+ */
+void
+SetTransitionTablePreserved(Oid relid, CmdType cmdType)
+{
+ AfterTriggersTableData *table;
+ AfterTriggersQueryData *qs;
+ bool found = false;
+ ListCell *lc;
+
+ /* Check state, like AfterTriggerSaveEvent. */
+ if (afterTriggers.query_depth < 0)
+ elog(ERROR, "SetTransitionTablePreserved() called outside of query");
+
+ qs = &afterTriggers.query_stack[afterTriggers.query_depth];
+
+ foreach(lc, qs->tables)
+ {
+ table = (AfterTriggersTableData *) lfirst(lc);
+ if (table->relid == relid && table->cmdType == cmdType &&
+ table->closed)
+ {
+ table->prolonged = true;
+ found = true;
+ }
+ }
+
+ if (!found)
+ elog(ERROR,"could not find table with OID %d and command type %d", relid, cmdType);
+}
+
+
/*
* GetAfterTriggersTableData
*
@@ -5138,6 +5184,7 @@ AfterTriggerBeginXact(void)
afterTriggers.firing_depth = 0;
afterTriggers.batch_callbacks = NIL;
afterTriggers.firing_batch_callbacks = false;
+ afterTriggers.prolonged_tuplestores = NIL;
/*
* Verify that there is no leftover state remaining. If these assertions
@@ -5312,11 +5359,11 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
ts = table->old_tuplestore;
table->old_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
ts = table->new_tuplestore;
table->new_tuplestore = NULL;
if (ts)
- tuplestore_end(ts);
+ release_or_prolong_tuplestore(ts, table->prolonged);
if (table->storeslot)
{
TupleTableSlot *slot = table->storeslot;
@@ -5334,8 +5381,33 @@ AfterTriggerFreeQuery(AfterTriggersQueryData *qs)
qs->tables = NIL;
list_free_deep(tables);
- list_free_deep(qs->batch_callbacks);
- qs->batch_callbacks = NIL;
+ /* Release prolonged tuplestores at the end of the outmost query */
+ if (afterTriggers.query_depth == 0)
+ {
+ foreach(lc, afterTriggers.prolonged_tuplestores)
+ {
+ ts = (Tuplestorestate *) lfirst(lc);
+ if (ts)
+ tuplestore_end(ts);
+ }
+ afterTriggers.prolonged_tuplestores = NIL;
+ }
+}
+
+/*
+ * Release the tuplestore, or append it to the prolonged tuplestores list.
+ */
+static void
+release_or_prolong_tuplestore(Tuplestorestate *ts, bool prolonged)
+{
+ if (prolonged && afterTriggers.query_depth > 0)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(CurTransactionContext);
+ afterTriggers.prolonged_tuplestores = lappend(afterTriggers.prolonged_tuplestores, ts);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ tuplestore_end(ts);
}
diff --git a/src/include/commands/trigger.h b/src/include/commands/trigger.h
index 0c3d485abf4..f0f052cc759 100644
--- a/src/include/commands/trigger.h
+++ b/src/include/commands/trigger.h
@@ -269,6 +269,8 @@ extern void AfterTriggerEndSubXact(bool isCommit);
extern void AfterTriggerSetState(ConstraintsSetStmt *stmt);
extern bool AfterTriggerPendingOnRel(Oid relid);
+extern void SetTransitionTablePreserved(Oid relid, CmdType cmdType);
+
/*
* in utils/adt/ri_triggers.c
--
2.43.0
--Multipart=_Fri__3_Jul_2026_19_11_16_+0900_CskxSIu_iGcDMEyM
Content-Type: text/x-diff;
name="v39-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v39-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
@ 2022-03-23 18:50 ` Peter Eisentraut <[email protected]>
6 siblings, 0 replies; 77+ messages in thread
From: Peter Eisentraut @ 2022-03-23 18:50 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; Euler Taveira <[email protected]>; [email protected]
On 18.03.22 23:34, Andrew Dunstan wrote:
> On 3/15/22 09:51, Peter Eisentraut wrote:
>> On 21.02.22 13:09, Euler Taveira wrote:
>>> A new tool called pg_subscriber does this conversion and is tightly
>>> integrated
>>> with Postgres.
>>
>> Are we comfortable with the name pg_subscriber? It seems too general.
>> Are we planning other subscriber-related operations in the future? If
>> so, we should at least make this one use a --create option or
>> something like that.
>
>
> Not really sold on the name (and I didn't much like the name
> pglogical_create_subscriber either, although it's a cool facility and
> I'm happy to see us adopting something like it).
>
> ISTM we should have a name that conveys that we are *converting* a
> replica or equivalent to a subscriber.
The pglogical tool includes the pg_basebackup run, so it actually
"creates" the subscriber from scratch. Whether this tool is also doing
that is still being discussed.
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
@ 2022-03-29 12:34 ` Peter Eisentraut <[email protected]>
2022-08-02 18:56 ` Re: speed up a logical replica setup Jacob Champion <[email protected]>
6 siblings, 1 reply; 77+ messages in thread
From: Peter Eisentraut @ 2022-03-29 12:34 UTC (permalink / raw)
To: Andres Freund <[email protected]>; Euler Taveira <[email protected]>; +Cc: [email protected]
On 22.03.22 02:25, Andres Freund wrote:
> On 2022-02-21 09:09:12 -0300, Euler Taveira wrote:
>> A new tool called pg_subscriber does this conversion and is tightly integrated
>> with Postgres.
>
> Given that this has been submitted just before the last CF and is a patch of
> nontrivial size, has't made significant progress ISTM it should be moved to
> the next CF?
done
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
2022-03-29 12:34 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
@ 2022-08-02 18:56 ` Jacob Champion <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Jacob Champion @ 2022-08-02 18:56 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Euler Taveira <[email protected]>; +Cc: [email protected]
This entry has been waiting on author input for a while (our current
threshold is roughly two weeks), so I've marked it Returned with
Feedback.
Once you think the patchset is ready for review again, you (or any
interested party) can resurrect the patch entry by visiting
https://commitfest.postgresql.org/38/3556/
and changing the status to "Needs Review", and then changing the
status again to "Move to next CF". (Don't forget the second step;
hopefully we will have streamlined this in the near future!)
Thanks,
--Jacob
^ permalink raw reply [nested|flat] 77+ messages in thread
* RE: speed up a logical replica setup
@ 2024-01-08 07:05 ` Hayato Kuroda (Fujitsu) <[email protected]>
6 siblings, 0 replies; 77+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-08 07:05 UTC (permalink / raw)
To: 'Amit Kapila' <[email protected]>; +Cc: Euler Taveira <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>
Dear Amit,
> On Fri, Jan 5, 2024 at 3:36 PM Hayato Kuroda (Fujitsu)
> <[email protected]> wrote:
> >
> > I love your proposal, so I want to join the review. Here are my first comments.
> >
> > 01.
> > Should we restrict that `--subscriber-conninfo` must not have hostname or IP?
> > We want users to execute pg_subscriber on the target, right?
> >
>
> I don't see any harm in users giving those information but we should
> have some checks to ensure that the server is in standby mode and is
> running locally. The other related point is do we need to take input
> for the target cluster directory from the user? Can't we fetch that
> information once we are connected to standby?
I think that functions like inet_client_addr() may be able to use, but it returns
NULL only when the connection is via a Unix-domain socket. Can we restrict
pg_subscriber to use such a socket?
> >
> > 05.
> > I found that the connection string for each subscriptions have a setting
> > "fallback_application_name=pg_subscriber". Can we remove it?
> >
> > ```
> > postgres=# SELECT subconninfo FROM pg_subscription;
> > subconninfo
> >
> ---------------------------------------------------------------------------------
> > user=postgres port=5431 fallback_application_name=pg_subscriber
> dbname=postgres
> > (1 row)
> > ```
>
> Can that help distinguish the pg_subscriber connection on the publisher?
>
Note that this connection string is used between the publisher instance and the
subscriber instance (not pg_subscriber client application). Also, the
fallback_application_name would be replaced to the name of subscriber in
run_apply_worker()->walrcv_connect(). Actually the value would not be used.
See below output on publisher.
```
publisher=# SELECT application_name, backend_type FROM pg_stat_activity where backend_type = 'walsender';
application_name | backend_type
----------------------+--------------
pg_subscriber_5_9411 | walsender
(1 row)
```
Or, if you mean to say that this can distinguish whether the subscription is used
by pg_subscriber or not. I think it is sufficient the current format of name.
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
@ 2024-01-18 09:18 ` Peter Eisentraut <[email protected]>
2024-01-18 09:37 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-31 13:03 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
6 siblings, 2 replies; 77+ messages in thread
From: Peter Eisentraut @ 2024-01-18 09:18 UTC (permalink / raw)
To: Euler Taveira <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>
On 11.01.24 23:15, Euler Taveira wrote:
> A new tool called pg_subscriber can convert a physical replica into a
> logical replica. It runs on the target server and should be able to
> connect to the source server (publisher) and the target server (subscriber).
Can we have a discussion on the name?
I find the name pg_subscriber too general.
The replication/backup/recovery tools in PostgreSQL are usually named
along the lines of "verb - object". (Otherwise, they would all be
called "pg_backup"??) Moreover, "pg_subscriber" also sounds like the
name of the program that runs the subscriber itself, like what the
walreceiver does now.
Very early in this thread, someone mentioned the name
pg_create_subscriber, and of course there is pglogical_create_subscriber
as the historical predecessor. Something along those lines seems better
to me. Maybe there are other ideas.
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
2024-01-18 09:18 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
@ 2024-01-18 09:37 ` Amit Kapila <[email protected]>
2024-01-22 06:55 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
1 sibling, 1 reply; 77+ messages in thread
From: Amit Kapila @ 2024-01-18 09:37 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: Euler Taveira <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>
On Thu, Jan 18, 2024 at 2:49 PM Peter Eisentraut <[email protected]> wrote:
>
> On 11.01.24 23:15, Euler Taveira wrote:
> > A new tool called pg_subscriber can convert a physical replica into a
> > logical replica. It runs on the target server and should be able to
> > connect to the source server (publisher) and the target server (subscriber).
>
> Can we have a discussion on the name?
>
> I find the name pg_subscriber too general.
>
> The replication/backup/recovery tools in PostgreSQL are usually named
> along the lines of "verb - object". (Otherwise, they would all be
> called "pg_backup"??) Moreover, "pg_subscriber" also sounds like the
> name of the program that runs the subscriber itself, like what the
> walreceiver does now.
>
> Very early in this thread, someone mentioned the name
> pg_create_subscriber, and of course there is pglogical_create_subscriber
> as the historical predecessor. Something along those lines seems better
> to me. Maybe there are other ideas.
>
The other option could be pg_createsubscriber on the lines of
pg_verifybackup and pg_combinebackup. Yet other options could be
pg_buildsubscriber, pg_makesubscriber as 'build' or 'make' in the name
sounds like we are doing some work to create the subscriber which I
think is the case here.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
2024-01-18 09:18 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-18 09:37 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
@ 2024-01-22 06:55 ` Peter Eisentraut <[email protected]>
2024-01-22 09:08 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Peter Eisentraut @ 2024-01-22 06:55 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Euler Taveira <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>
On 18.01.24 10:37, Amit Kapila wrote:
> The other option could be pg_createsubscriber on the lines of
> pg_verifybackup and pg_combinebackup.
Yes, that spelling would be more consistent.
> Yet other options could be
> pg_buildsubscriber, pg_makesubscriber as 'build' or 'make' in the name
> sounds like we are doing some work to create the subscriber which I
> think is the case here.
I see your point here. pg_createsubscriber is not like createuser in
that it just runs an SQL command. It does something different than
CREATE SUBSCRIBER. So a different verb would make that clearer. Maybe
something from here: https://www.thesaurus.com/browse/convert
^ permalink raw reply [nested|flat] 77+ messages in thread
* RE: speed up a logical replica setup
2024-01-18 09:18 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-18 09:37 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-22 06:55 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
@ 2024-01-22 09:08 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:22 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-22 09:08 UTC (permalink / raw)
To: 'Peter Eisentraut' <[email protected]>; Amit Kapila <[email protected]>; +Cc: Euler Taveira <[email protected]>; [email protected] <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>
Dear Peter,
>
> > Yet other options could be
> > pg_buildsubscriber, pg_makesubscriber as 'build' or 'make' in the name
> > sounds like we are doing some work to create the subscriber which I
> > think is the case here.
>
> I see your point here. pg_createsubscriber is not like createuser in
> that it just runs an SQL command. It does something different than
> CREATE SUBSCRIBER. So a different verb would make that clearer. Maybe
> something from here: https://www.thesaurus.com/browse/convert
I read the link and found a good verb "switch". So, how about using "pg_switchsubscriber"?
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
2024-01-18 09:18 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-18 09:37 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-22 06:55 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-22 09:08 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
@ 2024-01-22 09:22 ` Amit Kapila <[email protected]>
2024-01-23 23:44 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Amit Kapila @ 2024-01-22 09:22 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Euler Taveira <[email protected]>; [email protected] <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>
On Mon, Jan 22, 2024 at 2:38 PM Hayato Kuroda (Fujitsu)
<[email protected]> wrote:
> >
> > > Yet other options could be
> > > pg_buildsubscriber, pg_makesubscriber as 'build' or 'make' in the name
> > > sounds like we are doing some work to create the subscriber which I
> > > think is the case here.
> >
> > I see your point here. pg_createsubscriber is not like createuser in
> > that it just runs an SQL command. It does something different than
> > CREATE SUBSCRIBER.
Right.
So a different verb would make that clearer. Maybe
> > something from here: https://www.thesaurus.com/browse/convert
>
> I read the link and found a good verb "switch". So, how about using "pg_switchsubscriber"?
>
I also initially thought on these lines and came up with a name like
pg_convertsubscriber but didn't feel strongly about it as that would
have sounded meaningful if we use a name like
pg_convertstandbytosubscriber. Now, that has become too long. Having
said that, I am not opposed to it having a name on those lines. BTW,
another option that occurred to me today is pg_preparesubscriber. We
internally create slots and then wait for wal, etc. which makes me
sound like adding 'prepare' in the name can also explain the purpose.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
2024-01-18 09:18 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-18 09:37 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-22 06:55 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-22 09:08 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:22 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
@ 2024-01-23 23:44 ` Euler Taveira <[email protected]>
2024-01-25 09:27 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Euler Taveira @ 2024-01-23 23:44 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; [email protected] <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; [email protected] <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>
On Mon, Jan 22, 2024, at 6:22 AM, Amit Kapila wrote:
> On Mon, Jan 22, 2024 at 2:38 PM Hayato Kuroda (Fujitsu)
> <[email protected]> wrote:
> > >
> > > > Yet other options could be
> > > > pg_buildsubscriber, pg_makesubscriber as 'build' or 'make' in the name
> > > > sounds like we are doing some work to create the subscriber which I
> > > > think is the case here.
> > >
> > > I see your point here. pg_createsubscriber is not like createuser in
> > > that it just runs an SQL command. It does something different than
> > > CREATE SUBSCRIBER.
>
> Right.
Subscriber has a different meaning of subscription. Subscription is an SQL
object. Subscriber is the server (node in replication terminology) where the
subscription resides. Having said that pg_createsubscriber doesn't seem a bad
name because you are creating a new subscriber. (Indeed, you are transforming /
converting but "create" seems closer and users can infer that it is a tool to
build a new logical replica.
> So a different verb would make that clearer. Maybe
> > > something from here: https://www.thesaurus.com/browse/convert
> >
> > I read the link and found a good verb "switch". So, how about using "pg_switchsubscriber"?
> >
>
> I also initially thought on these lines and came up with a name like
> pg_convertsubscriber but didn't feel strongly about it as that would
> have sounded meaningful if we use a name like
> pg_convertstandbytosubscriber. Now, that has become too long. Having
> said that, I am not opposed to it having a name on those lines. BTW,
> another option that occurred to me today is pg_preparesubscriber. We
> internally create slots and then wait for wal, etc. which makes me
> sound like adding 'prepare' in the name can also explain the purpose.
I think "convert" and "transform" fit for this case. However, "create",
"convert" and "transform" have 6, 7 and 9 characters, respectively. I suggest
that we avoid long names (subscriber already has 10 characters). My preference
is pg_createsubscriber.
--
Euler Taveira
EDB https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
2024-01-18 09:18 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-18 09:37 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-22 06:55 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-22 09:08 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:22 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-23 23:44 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
@ 2024-01-25 09:27 ` Peter Eisentraut <[email protected]>
2024-01-25 10:16 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Peter Eisentraut @ 2024-01-25 09:27 UTC (permalink / raw)
To: Euler Taveira <[email protected]>; Amit Kapila <[email protected]>; [email protected] <[email protected]>; +Cc: [email protected] <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>
On 24.01.24 00:44, Euler Taveira wrote:
> Subscriber has a different meaning of subscription. Subscription is an SQL
> object. Subscriber is the server (node in replication terminology) where the
> subscription resides. Having said that pg_createsubscriber doesn't seem
> a bad
> name because you are creating a new subscriber. (Indeed, you are
> transforming /
> converting but "create" seems closer and users can infer that it is a
> tool to
> build a new logical replica.
That makes sense.
(Also, the problem with "convert" etc. is that "convertsubscriber" would
imply that you are converting an existing subscriber to something else.
It would need to be something like "convertbackup" then, which doesn't
seem helpful.)
> I think "convert" and "transform" fit for this case. However, "create",
> "convert" and "transform" have 6, 7 and 9 characters, respectively. I
> suggest
> that we avoid long names (subscriber already has 10 characters). My
> preference
> is pg_createsubscriber.
That seems best to me.
^ permalink raw reply [nested|flat] 77+ messages in thread
* RE: speed up a logical replica setup
2024-01-18 09:18 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-18 09:37 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-22 06:55 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-22 09:08 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:22 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-23 23:44 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 09:27 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
@ 2024-01-25 10:16 ` Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-25 10:16 UTC (permalink / raw)
To: 'Peter Eisentraut' <[email protected]>; Euler Taveira <[email protected]>; Amit Kapila <[email protected]>; +Cc: [email protected] <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>
Dear Peter,
Thanks for giving your idea!
> > Subscriber has a different meaning of subscription. Subscription is an SQL
> > object. Subscriber is the server (node in replication terminology) where the
> > subscription resides. Having said that pg_createsubscriber doesn't seem
> > a bad
> > name because you are creating a new subscriber. (Indeed, you are
> > transforming /
> > converting but "create" seems closer and users can infer that it is a
> > tool to
> > build a new logical replica.
>
> That makes sense.
>
> (Also, the problem with "convert" etc. is that "convertsubscriber" would
> imply that you are converting an existing subscriber to something else.
> It would need to be something like "convertbackup" then, which doesn't
> seem helpful.)
>
> > I think "convert" and "transform" fit for this case. However, "create",
> > "convert" and "transform" have 6, 7 and 9 characters, respectively. I
> > suggest
> > that we avoid long names (subscriber already has 10 characters). My
> > preference
> > is pg_createsubscriber.
>
> That seems best to me.
Just FYI - I'm ok to change the name to pg_createsubscriber.
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
2024-01-18 09:18 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
@ 2024-01-31 13:03 ` Fabrízio de Royes Mello <[email protected]>
1 sibling, 0 replies; 77+ messages in thread
From: Fabrízio de Royes Mello @ 2024-01-31 13:03 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: Euler Taveira <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>
On Thu, Jan 18, 2024 at 6:19 AM Peter Eisentraut <[email protected]>
wrote:
>
> Very early in this thread, someone mentioned the name
> pg_create_subscriber, and of course there is pglogical_create_subscriber
> as the historical predecessor. Something along those lines seems better
> to me. Maybe there are other ideas.
>
I've mentioned it upthread because of this pet project [1] that is one of
the motivations behind upstream this facility.
[1] https://github.com/fabriziomello/pg_create_subscriber
--
Fabrízio de Royes Mello
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
@ 2024-01-18 09:43 ` Amit Kapila <[email protected]>
6 siblings, 0 replies; 77+ messages in thread
From: Amit Kapila @ 2024-01-18 09:43 UTC (permalink / raw)
To: Shubham Khanna <[email protected]>; +Cc: Euler Taveira <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; [email protected]; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>
On Tue, Jan 16, 2024 at 11:58 AM Shubham Khanna
<[email protected]> wrote:
>
> On Thu, Dec 21, 2023 at 11:47 AM Amit Kapila <[email protected]> wrote:
> >
> >
> > 4. Can we see some numbers with various sizes of databases (cluster)
> > to see how it impacts the time for small to large-size databases as
> > compared to the traditional method? This might help us with giving
> > users advice on when to use this tool. We can do this bit later as
> > well when the patch is closer to being ready for commit.
>
> I have done the Performance testing and attached the results to
> compare the 'Execution Time' between 'logical replication' and
> 'pg_subscriber' for 100MB, 1GB and 5GB data:
> | 100MB | 1GB | 5GB
> Logical rep (2 w) | 1.815s | 14.895s | 75.541s
> Logical rep (4 w) | 1.194s | 9.484s | 46.938s
> Logical rep (8 w) | 0.828s | 6.422s | 31.704s
> Logical rep(10 w)| 0.646s | 3.843s | 18.425s
> pg_subscriber | 3.977s | 9.988s | 12.665s
>
> Here, 'w' stands for 'workers'. I have included the tests to see the
> test result variations with different values for
> 'max_sync_workers_per_subscription' ranging from 2 to 10. I ran the
> tests for different data records; for 100MB I put 3,00,000 Records,
> for 1GB I put 30,00,000 Records and for 5GB I put 1,50,00,000 Records.
> It is observed that 'pg_subscriber' is better when the table size is
> more.
>
Thanks for the tests. IIUC, it shows for smaller data this tool can
take more time. Can we do perf to see if there is something we can do
about reducing the overhead?
> Next I plan to run these tests for 10GB and 20GB to see if this trend
> continues or not.
>
Okay, that makes sense.
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
@ 2024-01-19 08:48 ` Shubham Khanna <[email protected]>
2024-01-25 05:54 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
6 siblings, 1 reply; 77+ messages in thread
From: Shubham Khanna @ 2024-01-19 08:48 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Euler Taveira <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; [email protected]; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>
On Tue, Jan 16, 2024 at 11:58 AM Shubham Khanna
<[email protected]> wrote:
>
> On Thu, Dec 21, 2023 at 11:47 AM Amit Kapila <[email protected]> wrote:
> >
> > On Wed, Dec 6, 2023 at 12:53 PM Euler Taveira <[email protected]> wrote:
> > >
> > > On Thu, Nov 9, 2023, at 8:12 PM, Michael Paquier wrote:
> > >
> > > On Thu, Nov 09, 2023 at 03:41:53PM +0100, Peter Eisentraut wrote:
> > > > On 08.11.23 00:12, Michael Paquier wrote:
> > > >> - Should the subdirectory pg_basebackup be renamed into something more
> > > >> generic at this point? All these things are frontend tools that deal
> > > >> in some way with the replication protocol to do their work. Say
> > > >> a replication_tools?
> > > >
> > > > Seems like unnecessary churn. Nobody has complained about any of the other
> > > > tools in there.
> > >
> > > Not sure. We rename things across releases in the tree from time to
> > > time, and here that's straight-forward.
> > >
> > >
> > > Based on this discussion it seems we have a consensus that this tool should be
> > > in the pg_basebackup directory. (If/when we agree with the directory renaming,
> > > it could be done in a separate patch.) Besides this move, the v3 provides a dry
> > > run mode. It basically executes every routine but skip when should do
> > > modifications. It is an useful option to check if you will be able to run it
> > > without having issues with connectivity, permission, and existing objects
> > > (replication slots, publications, subscriptions). Tests were slightly improved.
> > > Messages were changed to *not* provide INFO messages by default and --verbose
> > > provides INFO messages and --verbose --verbose also provides DEBUG messages. I
> > > also refactored the connect_database() function into which the connection will
> > > always use the logical replication mode. A bug was fixed in the transient
> > > replication slot name. Ashutosh review [1] was included. The code was also indented.
> > >
> > > There are a few suggestions from Ashutosh [2] that I will reply in another
> > > email.
> > >
> > > I'm still planning to work on the following points:
> > >
> > > 1. improve the cleanup routine to point out leftover objects if there is any
> > > connection issue.
> > >
> >
> > I think this is an important part. Shall we try to write to some file
> > the pending objects to be cleaned up? We do something like that during
> > the upgrade.
> >
> > > 2. remove the physical replication slot if the standby is using one
> > > (primary_slot_name).
> > > 3. provide instructions to promote the logical replica into primary, I mean,
> > > stop the replication between the nodes and remove the replication setup
> > > (publications, subscriptions, replication slots). Or even include another
> > > action to do it. We could add both too.
> > >
> > > Point 1 should be done. Points 2 and 3 aren't essential but will provide a nice
> > > UI for users that would like to use it.
> > >
> >
> > Isn't point 2 also essential because how would otherwise such a slot
> > be advanced or removed?
> >
> > A few other points:
> > ==============
> > 1. Previously, I asked whether we need an additional replication slot
> > patch created to get consistent LSN and I see the following comment in
> > the patch:
> >
> > + *
> > + * XXX we should probably use the last created replication slot to get a
> > + * consistent LSN but it should be changed after adding pg_basebackup
> > + * support.
> >
> > Yeah, sure, we may want to do that after backup support and we can
> > keep a comment for the same but I feel as the patch stands today,
> > there is no good reason to keep it. Also, is there a reason that we
> > can't create the slots after backup is complete and before we write
> > recovery parameters
> >
> > 2.
> > + appendPQExpBuffer(str,
> > + "CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
> > + "WITH (create_slot = false, copy_data = false, enabled = false)",
> > + dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
> >
> > Shouldn't we enable two_phase by default for newly created
> > subscriptions? Is there a reason for not doing so?
> >
> > 3. How about sync slots on the physical standby if present? Do we want
> > to retain those as it is or do we need to remove those? We are
> > actively working on the patch [1] for the same.
> >
> > 4. Can we see some numbers with various sizes of databases (cluster)
> > to see how it impacts the time for small to large-size databases as
> > compared to the traditional method? This might help us with giving
> > users advice on when to use this tool. We can do this bit later as
> > well when the patch is closer to being ready for commit.
>
> I have done the Performance testing and attached the results to
> compare the 'Execution Time' between 'logical replication' and
> 'pg_subscriber' for 100MB, 1GB and 5GB data:
> | 100MB | 1GB | 5GB
> Logical rep (2 w) | 1.815s | 14.895s | 75.541s
> Logical rep (4 w) | 1.194s | 9.484s | 46.938s
> Logical rep (8 w) | 0.828s | 6.422s | 31.704s
> Logical rep(10 w)| 0.646s | 3.843s | 18.425s
> pg_subscriber | 3.977s | 9.988s | 12.665s
>
> Here, 'w' stands for 'workers'. I have included the tests to see the
> test result variations with different values for
> 'max_sync_workers_per_subscription' ranging from 2 to 10. I ran the
> tests for different data records; for 100MB I put 3,00,000 Records,
> for 1GB I put 30,00,000 Records and for 5GB I put 1,50,00,000 Records.
> It is observed that 'pg_subscriber' is better when the table size is
> more.
> Next I plan to run these tests for 10GB and 20GB to see if this trend
> continues or not.
I have done the Performance testing and attached the results to
compare the 'Execution Time' between 'logical replication' and
'pg_subscriber' for 10GB and 20GB data:
| 10GB | 20GB
Logical rep (2 w) | 157.131s| 343.191s
Logical rep (4 w) | 116.627s| 240.480s
Logical rep (8 w) | 95.237s | 275.715s
Logical rep(10 w)| 92.792s | 280.538s
pg_subscriber | 22.734s | 25.661s
As expected, we can see that pg_subscriber is very much better in
ideal cases with approximately 7x times better in case of 10GB and 13x
times better in case of 20GB.
I'm attaching the script files which have the details of the test
scripts used and the excel file has the test run details. The
'pg_subscriber.pl' file is for 'Streaming Replication' and the
'logical_replication.pl' file is for 'Logical Replication'.
Note: For 20GB the record count should be changed to 6,00,00,000 and
the 'max_sync_workers_per_subscription' needs to be adjusted for
different logical replication tests with different workers.
Thanks and Regards,
Shubham Khanna.
Attachments:
[application/octet-stream] logical_replication.pl (2.8K, ../../CAHv8RjKYo1Xdkj6WhFZ32Mg4-+9EuNgau=JZH8U7_NPv2gFmGQ@mail.gmail.com/2-logical_replication.pl)
download
[application/octet-stream] pg_subscriber.pl (3.3K, ../../CAHv8RjKYo1Xdkj6WhFZ32Mg4-+9EuNgau=JZH8U7_NPv2gFmGQ@mail.gmail.com/3-pg_subscriber.pl)
download
[application/vnd.openxmlformats-officedocument.spreadsheetml.sheet] time_stamp(comparison).xlsx (11.5K, ../../CAHv8RjKYo1Xdkj6WhFZ32Mg4-+9EuNgau=JZH8U7_NPv2gFmGQ@mail.gmail.com/4-time_stamp%28comparison%29.xlsx)
download
^ permalink raw reply [nested|flat] 77+ messages in thread
* RE: speed up a logical replica setup
2024-01-19 08:48 ` Re: speed up a logical replica setup Shubham Khanna <[email protected]>
@ 2024-01-25 05:54 ` Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-25 05:54 UTC (permalink / raw)
To: 'Shubham Khanna' <[email protected]>; Amit Kapila <[email protected]>; +Cc: Euler Taveira <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>
Dear hackers,
Based on the requirement, I have profiled the performance test. It showed bottlenecks
are in small-data case are mainly two - starting a server and waiting until the
recovery is done.
# Tested source code
V7 patch set was applied atop HEAD(0eb23285). No configure options were specified
when it was built.
# Tested workload
I focused on only 100MB/1GB cases because bigger ones have already had good performance.
(Number of inserted tuples were same as previous tests)
I used bash script instead of tap test framework. See attached. Executed SQLs and
operations were almost the same.
As you can see, I tested only one-db case. Results may be changed if the number
of databases were changed.
# Measurement
Some debug logs which output current time were added (please see diff file).
I picked up some events and done at before/after them. Below bullets showed the measured ones:
* Starting a server
* Stopping a server
* Creating replication slots
* Creating publications
* Waiting until the recovery ended
* Creating subscriptions
# Result
Below table shows the elapsed time for these events. Raw data is also available
by the attached excel file.
|Event category |100MB case [sec]|1GB [sec]|
|Starting a server |1.414 |1.417 |
|Stoping a server |0.506 |0.506 |
|Creating replication slots |0.005 |0.007 |
|Creating publications |0.001 |0.002 |
|Waiting until the recovery ended|1.603 |14.529 |
|Creating subscriptions |0.012 |0.012 |
|Total |3.541 |16.473 |
|actual time |4.37 |17.271 |
As you can see, starting servers and waiting seem slow. We cannot omit these,
but setting smaller shared_buffers will reduce the start time. One approach is
to overwrite the GUC to smaller value, but I think we cannot determine the
appropriate value.
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
[application/octet-stream] run.sh (2.2K, ../../TY3PR01MB9889C1E33C73DC064B3AEA90F57A2@TY3PR01MB9889.jpnprd01.prod.outlook.com/2-run.sh)
download
[application/vnd.openxmlformats-officedocument.spreadsheetml.sheet] perf_result.xlsx (17.3K, ../../TY3PR01MB9889C1E33C73DC064B3AEA90F57A2@TY3PR01MB9889.jpnprd01.prod.outlook.com/3-perf_result.xlsx)
download
[application/octet-stream] add_debug_log.diff (8.2K, ../../TY3PR01MB9889C1E33C73DC064B3AEA90F57A2@TY3PR01MB9889.jpnprd01.prod.outlook.com/4-add_debug_log.diff)
download | inline diff:
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index 355738c..290be7c 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -675,13 +675,66 @@ create_all_logical_replication_slots(PrimaryInfo *primary,
get_subscription_name(perdb->oid, (int) getpid(), replslotname, NAMEDATALEN);
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("creating a replication slot %s start: %s", replslotname, timebuf);
+ }
+
/* Create replication slot on publisher. */
if (create_logical_replication_slot(conn, false, perdb) == NULL && !dry_run)
return false;
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("creating a replication slot %s end: %s", replslotname, timebuf);
+ }
+
+
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("creating a publication start: %s", timebuf);
+ }
+
/* Also create a publication */
create_publication(conn, primary, perdb);
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("creating a publication end: %s", timebuf);
+ }
+
disconnect_database(conn);
}
@@ -839,6 +892,19 @@ wait_for_end_recovery(const char *base_conninfo, const char *dbname)
if (conn == NULL)
exit(1);
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("creating a wait_for start: %s", timebuf);
+ }
+
for (cnt = 0; cnt < wait_seconds * WAITS_PER_SEC; cnt++)
{
bool in_recovery;
@@ -875,6 +941,19 @@ wait_for_end_recovery(const char *base_conninfo, const char *dbname)
pg_usleep(USEC_PER_SEC / WAITS_PER_SEC);
}
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("creating a wait_for stop: %s", timebuf);
+ }
+
disconnect_database(conn);
/*
@@ -1059,6 +1138,19 @@ create_subscription(PGconn *conn, StandbyInfo *standby, char *base_conninfo,
pg_log_debug("command is: %s", str->data);
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("creating a subscription %s start: %s", subname, timebuf);
+ }
+
if (!dry_run)
{
res = PQexec(conn, str->data);
@@ -1071,6 +1163,19 @@ create_subscription(PGconn *conn, StandbyInfo *standby, char *base_conninfo,
}
}
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("creating a subscription %s end: %s", subname, timebuf);
+ }
+
/* for cleanup purposes */
perdb->made_subscription = true;
@@ -1277,8 +1382,35 @@ start_standby_server(StandbyInfo *standby, unsigned short subport,
pg_ctl_cmd = psprintf("\"%s/pg_ctl\" start -D \"%s\" -s -o \"-p %d\" -l \"%s\"",
standby->bindir,
standby->pgdata, subport, server_start_log);
+
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("pg_ctl start begin: %s", timebuf);
+ }
+
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 1);
+
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("pg_ctl start end: %s", timebuf);
+ }
}
static char *
@@ -1723,8 +1855,38 @@ main(int argc, char **argv)
pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
standby.bindir, standby.pgdata);
+
+
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("pg_ctl stop begin: %s", timebuf);
+ }
+
+
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("pg_ctl stop end: %s", timebuf);
+ }
}
/*
@@ -1756,6 +1918,7 @@ main(int argc, char **argv)
consistent_lsn = create_logical_replication_slot(conn, true,
&dbarr.perdb[0]);
+
/*
* Write recovery parameters.
*
@@ -1856,9 +2019,36 @@ main(int argc, char **argv)
pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
standby.bindir, standby.pgdata);
+
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("pg_ctl stop for subscriber begin: %s", timebuf);
+ }
+
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+ pg_log_warning("pg_ctl stop for subscriber end: %s", timebuf);
+ }
+
/*
* Change system identifier.
*/
^ permalink raw reply [nested|flat] 77+ messages in thread
* RE: speed up a logical replica setup
@ 2024-01-22 07:06 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-23 23:58 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
6 siblings, 2 replies; 77+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-22 07:06 UTC (permalink / raw)
To: '[email protected]' <[email protected]>; +Cc: 'Shlok Kyal' <[email protected]>; 'vignesh C' <[email protected]>; 'Michael Paquier' <[email protected]>; 'Peter Eisentraut' <[email protected]>; 'Andres Freund' <[email protected]>; 'Ashutosh Bapat' <[email protected]>; 'Amit Kapila' <[email protected]>; 'Euler Taveira' <[email protected]>
Dear hackers,
>
> 15.
> I found that subscriptions cannot be started if tuples are inserted on publisher
> after creating temp_replslot. After starting a subscriber, I got below output on the
> log.
>
> ```
> ERROR: could not receive data from WAL stream: ERROR: publication
> "pg_subscriber_5" does not exist
> CONTEXT: slot "pg_subscriber_5_3632", output plugin "pgoutput", in the change
> callback, associated LSN 0/30008A8
> LOG: background worker "logical replication apply worker" (PID 3669) exited
> with exit code 1
> ```
>
> But this is strange. I confirmed that the specified publication surely exists.
> Do you know the reason?
>
> ```
> publisher=# SELECT pubname FROM pg_publication;
> pubname
> -----------------
> pg_subscriber_5
> (1 row)
> ```
>
I analyzed and found a reason. This is because publications are invisible for some transactions.
As the first place, below operations were executed in this case.
Tuples were inserted after getting consistent_lsn, but before starting the standby.
After doing the workload, I confirmed again that the publication was created.
1. on primary, logical replication slots were created.
2. on primary, another replication slot was created.
3. ===on primary, some tuples were inserted. ===
4. on standby, a server process was started
5. on standby, the process waited until all changes have come.
6. on primary, publications were created.
7. on standby, subscriptions were created.
8. on standby, a replication progress for each subscriptions was set to given LSN (got at step2).
=====pg_subscriber finished here=====
9. on standby, a server process was started again
10. on standby, subscriptions were enabled. They referred slots created at step1.
11. on primary, decoding was started but ERROR was raised.
In this case, tuples were inserted *before creating publication*.
So I thought that the decoded transaction could not see the publication because
it was committed after insertions.
One solution is to create a publication before creating a consistent slot.
Changes which came before creating the slot were surely replicated to the standby,
so upcoming transactions can see the object. We are planning to patch set to fix
the issue in this approach.
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
@ 2024-01-22 09:30 ` Shlok Kyal <[email protected]>
2024-01-23 02:11 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
1 sibling, 2 replies; 77+ messages in thread
From: Shlok Kyal @ 2024-01-22 09:30 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: [email protected] <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Euler Taveira <[email protected]>
Dear Euler, hackers,
We fixed some of the comments posted in the thread. We have created it
as top-up patch 0002 and 0003.
0002 patch contains the following changes:
* Add a timeout option for the recovery option, per [1]. The code was
basically ported from pg_ctl.c.
* Reject if the target server is not a standby, per [2]
* Raise FATAL error if --subscriber-conninfo specifies non-local server, per [3]
(not sure it is really needed, so feel free reject the part.)
* Add check for max_replication_slots and wal_level; as per [4]
* Add -u and -p options; as per [5]
* Addressed comment except 5 and 8 in [6] and comment in [7]
0003 patch contains fix for bug reported in [8].
Feel free to merge parts of 0002 and 0003 if it looks good to you.
Thanks Kuroda-san to make patch 0003 and a part of patch 0002.
[1]: https://www.postgresql.org/message-id/CANhcyEUCt-g4JLQU3Q3ofFk_Vt-Tqh3ZdXoLcpT8fjz9LY_-ww%40mail.gma...
[2]: https://www.postgresql.org/message-id/CANhcyEUCt-g4JLQU3Q3ofFk_Vt-Tqh3ZdXoLcpT8fjz9LY_-ww%40mail.gma...
[3]: https://www.postgresql.org/message-id/TY3PR01MB98895BA6C1D72CB8582CACC4F5682%40TY3PR01MB9889.jpnprd0...
[4]: https://www.postgresql.org/message-id/CALDaNm098Jkbh%2Bye6zMj9Ro9j1bBe6FfPV80BFbs1%3DpUuTJ07g%40mail...
[5]: https://www.postgresql.org/message-id/CAA4eK1JB_ko7a5JMS3WfAn583RadAKCDhiE9JgmfMA8ZZ5xcQw%40mail.gma...
[6]: https://www.postgresql.org/message-id/TY3PR01MB9889C362FF76102C88FA1C29F56F2%40TY3PR01MB9889.jpnprd0...
[7]: https://www.postgresql.org/message-id/CANhcyEXjGmryoZPACS_i-joqvcz5e6Zb3u4g38SAy_iSTGhShg%40mail.gma...
[8]: https://www.postgresql.org/message-id/TY3PR01MB9889C5D55206DDD978627D07F5752%40TY3PR01MB9889.jpnprd0...
Thanks and regards
Shlok Kyal
Attachments:
[application/octet-stream] v6-0003-Fix-publication-does-not-exist-error.patch (1.6K, ../../CANhcyEXhpfWgc0-FdHmUG6WGD=2cxYqBePr1yxJ8VSZ0gV3XKA@mail.gmail.com/2-v6-0003-Fix-publication-does-not-exist-error.patch)
download | inline diff:
From 5d2b49e55888cdc36a38208d58cf16a5960821dc Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Mon, 22 Jan 2024 12:36:20 +0530
Subject: [PATCH v6 3/3] Fix publication does not exist error.
Fix publication does not exist error.
---
src/bin/pg_basebackup/pg_subscriber.c | 23 +++--------------------
1 file changed, 3 insertions(+), 20 deletions(-)
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index 0dc87e919b..8b1a92b68b 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -677,6 +677,9 @@ create_all_logical_replication_slots(PrimaryInfo *primary,
if (create_logical_replication_slot(conn, false, perdb) == NULL && !dry_run)
return false;
+ /* Also create a publication */
+ create_publication(conn, primary, perdb);
+
disconnect_database(conn);
}
@@ -1792,26 +1795,6 @@ main(int argc, char **argv)
*/
wait_for_end_recovery(standby.base_conninfo, dbarr.perdb[0].dbname);
- /*
- * Create a publication for each database. This step should be executed
- * after promoting the subscriber to avoid replicating unnecessary
- * objects.
- */
- for (i = 0; i < dbarr.ndbs; i++)
- {
- LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
-
- /* Connect to publisher. */
- conn = connect_database(primary.base_conninfo, perdb->dbname);
- if (conn == NULL)
- exit(1);
-
- /* Also create a publication */
- create_publication(conn, &primary, perdb);
-
- disconnect_database(conn);
- }
-
/*
* Create a subscription for each database.
*/
--
2.34.1
[application/octet-stream] v6-0002-Address-some-comments-proposed-on-hackers.patch (58.3K, ../../CANhcyEXhpfWgc0-FdHmUG6WGD=2cxYqBePr1yxJ8VSZ0gV3XKA@mail.gmail.com/3-v6-0002-Address-some-comments-proposed-on-hackers.patch)
download | inline diff:
From ad645b61dad1a8ce03ab0ad28a0c44d0a943cc3e Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Mon, 22 Jan 2024 12:42:34 +0530
Subject: [PATCH v6 2/3] Address some comments proposed on -hackers
The patch has following changes:
* Some comments reported on the thread
* Add a timeout option for the recovery option
* Reject if the target server is not a standby
* Reject when the --subscriber-conninfo specifies non-local server
* Add -u and -p options
* Check wal_level and max_replication_slot parameters
---
doc/src/sgml/ref/pg_subscriber.sgml | 21 +-
src/bin/pg_basebackup/pg_subscriber.c | 911 +++++++++++-------
src/bin/pg_basebackup/t/040_pg_subscriber.pl | 9 +-
.../t/041_pg_subscriber_standby.pl | 8 +-
4 files changed, 600 insertions(+), 349 deletions(-)
diff --git a/doc/src/sgml/ref/pg_subscriber.sgml b/doc/src/sgml/ref/pg_subscriber.sgml
index 553185c35f..eaabfc7053 100644
--- a/doc/src/sgml/ref/pg_subscriber.sgml
+++ b/doc/src/sgml/ref/pg_subscriber.sgml
@@ -16,12 +16,18 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_subscriber</refname>
- <refpurpose>create a new logical replica from a standby server</refpurpose>
+ <refpurpose>Convert a standby replica to a logical replica</refpurpose>
</refnamediv>
<refsynopsisdiv>
<cmdsynopsis>
<command>pg_subscriber</command>
+ <arg choice="plain"><option>-D</option></arg>
+ <arg choice="plain"><replaceable>datadir</replaceable></arg>
+ <arg choice="plain"><option>-P</option>
+ <replaceable>publisher-conninfo</replaceable></arg>
+ <arg choice="plain"><option>-S</option></arg>
+ <arg choice="plain"><replaceable>subscriber-conninfo</replaceable></arg>
<arg rep="repeat"><replaceable>option</replaceable></arg>
</cmdsynopsis>
</refsynopsisdiv>
@@ -29,17 +35,18 @@ PostgreSQL documentation
<refsect1>
<title>Description</title>
<para>
- <application>pg_subscriber</application> takes the publisher and subscriber
- connection strings, a cluster directory from a standby server and a list of
- database names and it sets up a new logical replica using the physical
- recovery process.
+ pg_subscriber creates a new <link
+ linkend="logical-replication-subscription">subscriber</link> from a physical
+ standby server. This allows users to quickly set up logical replication
+ system.
</para>
<para>
- The <application>pg_subscriber</application> should be run at the target
+ The <application>pg_subscriber</application> has to be run at the target
server. The source server (known as publisher server) should accept logical
replication connections from the target server (known as subscriber server).
- The target server should accept local logical replication connection.
+ The target server should accept logical replication connection from
+ localhost.
</para>
</refsect1>
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index e998c29f9e..0dc87e919b 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -1,12 +1,12 @@
/*-------------------------------------------------------------------------
*
* pg_subscriber.c
- * Create a new logical replica from a standby server
+ * Convert a standby replica to a logical replica
*
* Copyright (C) 2024, PostgreSQL Global Development Group
*
* IDENTIFICATION
- * src/bin/pg_subscriber/pg_subscriber.c
+ * src/bin/pg_basebackup/pg_subscriber.c
*
*-------------------------------------------------------------------------
*/
@@ -32,81 +32,122 @@
#define PGS_OUTPUT_DIR "pg_subscriber_output.d"
-typedef struct LogicalRepInfo
+typedef struct LogicalRepPerdbInfo
{
- Oid oid; /* database OID */
- char *dbname; /* database name */
- char *pubconninfo; /* publication connection string for logical
- * replication */
- char *subconninfo; /* subscription connection string for logical
- * replication */
- char *pubname; /* publication name */
- char *subname; /* subscription name (also replication slot
- * name) */
-
- bool made_replslot; /* replication slot was created */
- bool made_publication; /* publication was created */
- bool made_subscription; /* subscription was created */
-} LogicalRepInfo;
+ Oid oid;
+ char *dbname;
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepPerdbInfo;
+
+typedef struct
+{
+ LogicalRepPerdbInfo *perdb; /* array of db infos */
+ int ndbs; /* number of db infos */
+} LogicalRepPerdbInfoArr;
+
+typedef struct PrimaryInfo
+{
+ char *base_conninfo;
+ uint64 sysid;
+} PrimaryInfo;
+
+typedef struct StandbyInfo
+{
+ char *base_conninfo;
+ char *bindir;
+ char *pgdata;
+ char *primary_slot_name;
+ uint64 sysid;
+} StandbyInfo;
static void cleanup_objects_atexit(void);
static void usage();
-static char *get_base_conninfo(char *conninfo, char *dbname,
- const char *noderole);
-static bool get_exec_path(const char *path);
+static char *get_base_conninfo(char *conninfo, char *dbname);
+static bool get_exec_base_path(const char *path);
static bool check_data_directory(const char *datadir);
+static void store_db_names(LogicalRepPerdbInfo **perdb, int ndbs);
+static void get_sysid_for_primary(PrimaryInfo *primary, char *dbname);
+static void get_control_for_standby(StandbyInfo *standby);
static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
-static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
-static PGconn *connect_database(const char *conninfo);
+static PGconn *connect_database(const char *base_conninfo, const char*dbname);
static void disconnect_database(PGconn *conn);
-static uint64 get_sysid_from_conn(const char *conninfo);
-static uint64 get_control_from_datadir(const char *datadir);
-static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
-static char *use_primary_slot_name(void);
-static bool create_all_logical_replication_slots(LogicalRepInfo *dbinfo);
-static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
- char *slot_name);
-static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static char *use_primary_slot_name(PrimaryInfo *primary, StandbyInfo *standby,
+ LogicalRepPerdbInfo *perdb);
+static bool create_all_logical_replication_slots(PrimaryInfo *primary,
+ LogicalRepPerdbInfoArr *dbarr);
+static char *create_logical_replication_slot(PGconn *conn, bool temporary,
+ LogicalRepPerdbInfo *perdb);
+static void modify_sysid(const char *bindir, const char *datadir);
+static void drop_replication_slot(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *slot_name);
static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
-static void wait_for_end_recovery(const char *conninfo);
-static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
-static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
-static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
-static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void wait_for_end_recovery(const char *base_conninfo,
+ const char *dbname);
+static void create_publication(PGconn *conn, PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb);
+static void drop_publication(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void create_subscription(PGconn *conn, StandbyInfo *standby,
+ char *base_conninfo,
+ LogicalRepPerdbInfo *perdb);
+static void drop_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void set_replication_progress(PGconn *conn, LogicalRepPerdbInfo *perdb, const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void start_standby_server(StandbyInfo *standby, unsigned short subport,
+ char *server_start_log);
+static char *construct_sub_conninfo(char *username, unsigned short subport);
#define USEC_PER_SEC 1000000
-#define WAIT_INTERVAL 1 /* 1 second */
+#define DEFAULT_WAIT 60
+#define WAITS_PER_SEC 10 /* should divide USEC_PER_SEC evenly */
+#define DEF_PGSPORT 50111
/* Options */
-static const char *progname;
-
-static char *subscriber_dir = NULL;
static char *pub_conninfo_str = NULL;
-static char *sub_conninfo_str = NULL;
static SimpleStringList database_names = {NULL, NULL};
-static char *primary_slot_name = NULL;
+static int wait_seconds = DEFAULT_WAIT;
+static bool retain = false;
static bool dry_run = false;
static bool success = false;
+static const char *progname;
+static LogicalRepPerdbInfoArr dbarr;
+static PrimaryInfo primary;
+static StandbyInfo standby;
-static char *pg_ctl_path = NULL;
-static char *pg_resetwal_path = NULL;
+enum PGSWaitPMResult
+{
+ PGS_POSTMASTER_READY,
+ PGS_POSTMASTER_STANDBY,
+ PGS_POSTMASTER_STILL_STARTING,
+ PGS_POSTMASTER_FAILED
+};
-static LogicalRepInfo *dbinfo;
-static int num_dbs = 0;
-static char temp_replslot[NAMEDATALEN] = {0};
-static bool made_transient_replslot = false;
+/*
+ * Build the replication slot and subscription name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 36 characters
+ * (14 + 10 + 1 + 10 + '\0'). System identifier is included to reduce the
+ * probability of collision. By default, subscription name is used as
+ * replication slot name.
+ */
+static inline void
+get_subscription_name(Oid oid, int pid, char *subname, Size szsub)
+{
+ snprintf(subname, szsub, "pg_subscriber_%u_%d", oid, pid);
+}
-enum WaitPMResult
+/*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 35 characters (14 + 10 +
+ * '\0').
+ */
+static inline void
+get_publication_name(Oid oid, char *pubname, Size szpub)
{
- POSTMASTER_READY,
- POSTMASTER_STANDBY,
- POSTMASTER_STILL_STARTING,
- POSTMASTER_FAILED
-};
+ snprintf(pubname, szpub, "pg_subscriber_%u", oid);
+}
/*
@@ -125,41 +166,39 @@ cleanup_objects_atexit(void)
if (success)
return;
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
- if (dbinfo[i].made_subscription)
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
+
+ if (perdb->made_subscription)
{
- conn = connect_database(dbinfo[i].subconninfo);
+ conn = connect_database(standby.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- drop_subscription(conn, &dbinfo[i]);
+ drop_subscription(conn, perdb);
disconnect_database(conn);
}
}
- if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ if (perdb->made_publication || perdb->made_replslot)
{
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- if (dbinfo[i].made_publication)
- drop_publication(conn, &dbinfo[i]);
- if (dbinfo[i].made_replslot)
- drop_replication_slot(conn, &dbinfo[i], NULL);
+ if (perdb->made_publication)
+ drop_publication(conn, perdb);
+ if (perdb->made_replslot)
+ {
+ char replslotname[NAMEDATALEN];
+
+ get_subscription_name(perdb->oid, (int) getpid(),
+ replslotname, NAMEDATALEN);
+ drop_replication_slot(conn, perdb, replslotname);
+ }
disconnect_database(conn);
}
}
}
-
- if (made_transient_replslot)
- {
- conn = connect_database(dbinfo[0].pubconninfo);
- if (conn != NULL)
- {
- drop_replication_slot(conn, &dbinfo[0], temp_replslot);
- disconnect_database(conn);
- }
- }
}
static void
@@ -184,17 +223,16 @@ usage(void)
/*
* Validate a connection string. Returns a base connection string that is a
- * connection string without a database name plus a fallback application name.
- * Since we might process multiple databases, each database name will be
- * appended to this base connection string to provide a final connection string.
- * If the second argument (dbname) is not null, returns dbname if the provided
- * connection string contains it. If option --database is not provided, uses
- * dbname as the only database to setup the logical replica.
- * It is the caller's responsibility to free the returned connection string and
- * dbname.
+ * connection string without a database name. Since we might process multiple
+ * databases, each database name will be appended to this base connection
+ * string to provide a final connection string. If the second argument (dbname)
+ * is not null, returns dbname if the provided connection string contains it.
+ * If option --database is not provided, uses dbname as the only database to
+ * setup the logical replica. It is the caller's responsibility to free the
+ * returned connection string and dbname.
*/
static char *
-get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+get_base_conninfo(char *conninfo, char *dbname)
{
PQExpBuffer buf = createPQExpBuffer();
PQconninfoOption *conn_opts = NULL;
@@ -203,7 +241,7 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
char *ret;
int i;
- pg_log_info("validating connection string on %s", noderole);
+ pg_log_info("validating connection string on publisher");
conn_opts = PQconninfoParse(conninfo, &errmsg);
if (conn_opts == NULL)
@@ -231,10 +269,6 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
}
}
- if (i > 0)
- appendPQExpBufferChar(buf, ' ');
- appendPQExpBuffer(buf, "fallback_application_name=%s", progname);
-
ret = pg_strdup(buf->data);
destroyPQExpBuffer(buf);
@@ -244,15 +278,16 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
}
/*
- * Get the absolute path from other PostgreSQL binaries (pg_ctl and
- * pg_resetwal) that is used by it.
+ * Get the absolute binary path from another PostgreSQL binary (pg_ctl) and set
+ * to StandbyInfo.
*/
static bool
-get_exec_path(const char *path)
+get_exec_base_path(const char *path)
{
int rc;
+ char pg_ctl_path[MAXPGPATH];
+ char *p;
- pg_ctl_path = pg_malloc(MAXPGPATH);
rc = find_other_exec(path, "pg_ctl",
"pg_ctl (PostgreSQL) " PG_VERSION "\n",
pg_ctl_path);
@@ -277,30 +312,10 @@ get_exec_path(const char *path)
pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
- pg_resetwal_path = pg_malloc(MAXPGPATH);
- rc = find_other_exec(path, "pg_resetwal",
- "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
- pg_resetwal_path);
- if (rc < 0)
- {
- char full_path[MAXPGPATH];
-
- if (find_my_exec(path, full_path) < 0)
- strlcpy(full_path, progname, sizeof(full_path));
- if (rc == -1)
- pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
- "same directory as \"%s\".\n"
- "Check your installation.",
- "pg_resetwal", progname, full_path);
- else
- pg_log_error("The program \"%s\" was found by \"%s\"\n"
- "but was not the same version as %s.\n"
- "Check your installation.",
- "pg_resetwal", full_path, progname);
- return false;
- }
-
- pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+ /* Extract the directory part from the path */
+ Assert(p = strrchr(pg_ctl_path, 'p'));
+ *p = '\0';
+ standby.bindir = pg_strdup(pg_ctl_path);
return true;
}
@@ -364,49 +379,36 @@ concat_conninfo_dbname(const char *conninfo, const char *dbname)
}
/*
- * Store publication and subscription information.
+ * Initialize per-db structure and store the name of databases
*/
-static LogicalRepInfo *
-store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+static void
+store_db_names(LogicalRepPerdbInfo **perdb, int ndbs)
{
- LogicalRepInfo *dbinfo;
SimpleStringListCell *cell;
int i = 0;
- dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+ *perdb = (LogicalRepPerdbInfo *) pg_malloc0(sizeof(LogicalRepPerdbInfo) *
+ ndbs);
for (cell = database_names.head; cell; cell = cell->next)
{
- char *conninfo;
-
- /* Publisher. */
- conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
- dbinfo[i].pubconninfo = conninfo;
- dbinfo[i].dbname = cell->val;
- dbinfo[i].made_replslot = false;
- dbinfo[i].made_publication = false;
- dbinfo[i].made_subscription = false;
- /* other struct fields will be filled later. */
-
- /* Subscriber. */
- conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
- dbinfo[i].subconninfo = conninfo;
-
+ (*perdb)[i].dbname = pg_strdup(cell->val);
i++;
}
-
- return dbinfo;
}
static PGconn *
-connect_database(const char *conninfo)
+connect_database(const char *base_conninfo, const char*dbname)
{
PGconn *conn;
PGresult *res;
- const char *rconninfo;
+
+ char *rconninfo;
+ char *concat_conninfo = concat_conninfo_dbname(base_conninfo,
+ dbname);
/* logical replication mode */
- rconninfo = psprintf("%s replication=database", conninfo);
+ rconninfo = psprintf("%s replication=database", concat_conninfo);
conn = PQconnectdb(rconninfo);
if (PQstatus(conn) != CONNECTION_OK)
@@ -424,6 +426,9 @@ connect_database(const char *conninfo)
}
PQclear(res);
+ pfree(rconninfo);
+ pfree(concat_conninfo);
+
return conn;
}
@@ -436,19 +441,18 @@ disconnect_database(PGconn *conn)
}
/*
- * Obtain the system identifier using the provided connection. It will be used
- * to compare if a data directory is a clone of another one.
+ * Obtain the system identifier from the primary server. It will be used to
+ * compare if a data directory is a clone of another one.
*/
-static uint64
-get_sysid_from_conn(const char *conninfo)
+static void
+get_sysid_for_primary(PrimaryInfo *primary, char *dbname)
{
PGconn *conn;
PGresult *res;
- uint64 sysid;
pg_log_info("getting system identifier from publisher");
- conn = connect_database(conninfo);
+ conn = connect_database(primary->base_conninfo, dbname);
if (conn == NULL)
exit(1);
@@ -471,43 +475,39 @@ get_sysid_from_conn(const char *conninfo)
exit(1);
}
- sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+ primary->sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
- pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+ pg_log_info("system identifier is %llu on publisher",
+ (unsigned long long) primary->sysid);
disconnect_database(conn);
-
- return sysid;
}
/*
- * Obtain the system identifier from control file. It will be used to compare
- * if a data directory is a clone of another one. This routine is used locally
- * and avoids a replication connection.
+ * Obtain the system identifier from a standby server. It will be used to
+ * compare if a data directory is a clone of another one. This routine is used
+ * locally and avoids a replication connection.
*/
-static uint64
-get_control_from_datadir(const char *datadir)
+static void
+get_control_for_standby(StandbyInfo *standby)
{
ControlFileData *cf;
bool crc_ok;
- uint64 sysid;
pg_log_info("getting system identifier from subscriber");
- cf = get_controlfile(datadir, &crc_ok);
+ cf = get_controlfile(standby->pgdata, &crc_ok);
if (!crc_ok)
{
pg_log_error("control file appears to be corrupt");
exit(1);
}
- sysid = cf->system_identifier;
+ standby->sysid = cf->system_identifier;
- pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) standby->sysid);
pfree(cf);
-
- return sysid;
}
/*
@@ -516,7 +516,7 @@ get_control_from_datadir(const char *datadir)
* files from one of the systems might be used in the other one.
*/
static void
-modify_sysid(const char *pg_resetwal_path, const char *datadir)
+modify_sysid(const char *bindir, const char *datadir)
{
ControlFileData *cf;
bool crc_ok;
@@ -551,7 +551,7 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
pg_log_info("running pg_resetwal on the subscriber");
- cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+ cmd_str = psprintf("\"%s/pg_resetwal\" -D \"%s\"", bindir, datadir);
pg_log_debug("command is: %s", cmd_str);
@@ -571,14 +571,15 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
* Return a palloc'd slot name if the replication is using one.
*/
static char *
-use_primary_slot_name(void)
+use_primary_slot_name(PrimaryInfo *primary, StandbyInfo *standby,
+ LogicalRepPerdbInfo *perdb)
{
PGconn *conn;
PGresult *res;
PQExpBuffer str = createPQExpBuffer();
char *slot_name;
- conn = connect_database(dbinfo[0].subconninfo);
+ conn = connect_database(standby->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -604,7 +605,7 @@ use_primary_slot_name(void)
disconnect_database(conn);
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -634,17 +635,19 @@ use_primary_slot_name(void)
}
static bool
-create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
+create_all_logical_replication_slots(PrimaryInfo *primary,
+ LogicalRepPerdbInfoArr *dbarr)
{
int i;
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr->ndbs; i++)
{
PGconn *conn;
PGresult *res;
char replslotname[NAMEDATALEN];
+ LogicalRepPerdbInfo *perdb = &dbarr->perdb[i];
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -664,27 +667,14 @@ create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
}
/* Remember database OID. */
- dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ perdb->oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
PQclear(res);
- /*
- * Build the replication slot name. The name must not exceed
- * NAMEDATALEN - 1. This current schema uses a maximum of 36
- * characters (14 + 10 + 1 + 10 + '\0'). System identifier is included
- * to reduce the probability of collision. By default, subscription
- * name is used as replication slot name.
- */
- snprintf(replslotname, sizeof(replslotname),
- "pg_subscriber_%u_%d",
- dbinfo[i].oid,
- (int) getpid());
- dbinfo[i].subname = pg_strdup(replslotname);
+ get_subscription_name(perdb->oid, (int) getpid(), replslotname, NAMEDATALEN);
/* Create replication slot on publisher. */
- if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
- pg_log_info("create replication slot \"%s\" on publisher", replslotname);
- else
+ if (create_logical_replication_slot(conn, false, perdb) == NULL && !dry_run)
return false;
disconnect_database(conn);
@@ -701,30 +691,36 @@ create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
* result set that contains the consistent LSN.
*/
static char *
-create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
- char *slot_name)
+create_logical_replication_slot(PGconn *conn, bool temporary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res = NULL;
char *lsn = NULL;
- bool transient_replslot = false;
+ char slot_name[NAMEDATALEN];
Assert(conn != NULL);
/*
- * If no slot name is informed, it is a transient replication slot used
- * only for catch up purposes.
+ * Construct a name of logical replication slot. The formatting is
+ * different depends on its persistency.
+ *
+ * For persistent slots: the name must be same as the subscription.
+ * For temporary slots: OID is not needed, but another string is added.
*/
- if (slot_name[0] == '\0')
- {
+ if (!temporary)
+ get_subscription_name(perdb->oid, (int) getpid(), slot_name, NAMEDATALEN);
+ else
snprintf(slot_name, NAMEDATALEN, "pg_subscriber_%d_startpoint",
(int) getpid());
- transient_replslot = true;
- }
- pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, perdb->dbname);
appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
+
+ if(temporary)
+ appendPQExpBufferStr(str, " TEMPORARY");
+
appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
pg_log_debug("command is: %s", str->data);
@@ -734,17 +730,14 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
- pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
- PQresultErrorMessage(res));
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s",
+ slot_name, perdb->dbname, PQresultErrorMessage(res));
return lsn;
}
}
/* for cleanup purposes */
- if (transient_replslot)
- made_transient_replslot = true;
- else
- dbinfo->made_replslot = true;
+ perdb->made_replslot = true;
if (!dry_run)
{
@@ -758,14 +751,15 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
}
static void
-drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+drop_replication_slot(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *slot_name)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
Assert(conn != NULL);
- pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, perdb->dbname);
appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
@@ -775,7 +769,7 @@ drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_nam
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, perdb->dbname,
PQerrorMessage(conn));
PQclear(res);
@@ -825,19 +819,22 @@ pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
* Returns after the server finishes the recovery process.
*/
static void
-wait_for_end_recovery(const char *conninfo)
+wait_for_end_recovery(const char *base_conninfo, const char *dbname)
{
PGconn *conn;
PGresult *res;
- int status = POSTMASTER_STILL_STARTING;
+ int status = PGS_POSTMASTER_STILL_STARTING;
+ int cnt;
+ int rc;
+ char *pg_ctl_cmd;
pg_log_info("waiting the postmaster to reach the consistent state");
- conn = connect_database(conninfo);
+ conn = connect_database(base_conninfo, dbname);
if (conn == NULL)
exit(1);
- for (;;)
+ for (cnt = 0; cnt < wait_seconds * WAITS_PER_SEC; cnt++)
{
bool in_recovery;
@@ -865,17 +862,32 @@ wait_for_end_recovery(const char *conninfo)
*/
if (!in_recovery || dry_run)
{
- status = POSTMASTER_READY;
+ status = PGS_POSTMASTER_READY;
break;
}
/* Keep waiting. */
- pg_usleep(WAIT_INTERVAL * USEC_PER_SEC);
+ pg_usleep(USEC_PER_SEC / WAITS_PER_SEC);
}
disconnect_database(conn);
- if (status == POSTMASTER_STILL_STARTING)
+ /*
+ * If timeout is reached exit the pg_subscriber and stop the standby node.
+ */
+ if (cnt >= wait_seconds * WAITS_PER_SEC)
+ {
+ pg_log_error("recovery timed out");
+
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
+ standby.bindir, standby.pgdata);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+ exit(1);
+ }
+
+ if (status == PGS_POSTMASTER_STILL_STARTING)
{
pg_log_error("server did not end recovery");
exit(1);
@@ -888,17 +900,21 @@ wait_for_end_recovery(const char *conninfo)
* Create a publication that includes all tables in the database.
*/
static void
-create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+create_publication(PGconn *conn, PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
/* Check if the publication needs to be created. */
appendPQExpBuffer(str,
"SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
- dbinfo->pubname);
+ pubname);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
@@ -918,7 +934,7 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
*/
if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
{
- pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ pg_log_info("publication \"%s\" already exists", pubname);
return;
}
else
@@ -931,7 +947,7 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* database oid in which puballtables is false.
*/
pg_log_error("publication \"%s\" does not replicate changes for all tables",
- dbinfo->pubname);
+ pubname);
pg_log_error_hint("Consider renaming this publication.");
PQclear(res);
PQfinish(conn);
@@ -942,9 +958,9 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
PQclear(res);
resetPQExpBuffer(str);
- pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+ pg_log_info("creating publication \"%s\" on database \"%s\"", pubname, perdb->dbname);
- appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", pubname);
pg_log_debug("command is: %s", str->data);
@@ -954,14 +970,14 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
- dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ pubname, perdb->dbname, PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
}
/* for cleanup purposes */
- dbinfo->made_publication = true;
+ perdb->made_publication = true;
if (!dry_run)
PQclear(res);
@@ -973,16 +989,19 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* Remove publication if it couldn't finish all steps.
*/
static void
-drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+drop_publication(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
+ pg_log_info("dropping publication \"%s\" on database \"%s\"", pubname, perdb->dbname);
- appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", pubname);
pg_log_debug("command is: %s", str->data);
@@ -990,7 +1009,7 @@ drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", pubname, perdb->dbname, PQerrorMessage(conn));
PQclear(res);
}
@@ -1011,19 +1030,27 @@ drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* initial location.
*/
static void
-create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+create_subscription(PGconn *conn, StandbyInfo *standby, char *base_conninfo,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", subname,
+ perdb->dbname);
appendPQExpBuffer(str,
"CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
"WITH (create_slot = false, copy_data = false, enabled = false)",
- dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+ subname, concat_conninfo_dbname(base_conninfo, perdb->dbname), pubname);
pg_log_debug("command is: %s", str->data);
@@ -1033,14 +1060,14 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
- dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ subname, perdb->dbname, PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
}
/* for cleanup purposes */
- dbinfo->made_subscription = true;
+ perdb->made_subscription = true;
if (!dry_run)
PQclear(res);
@@ -1052,16 +1079,19 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* Remove subscription if it couldn't finish all steps.
*/
static void
-drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+drop_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
- appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"", subname, perdb->dbname);
+
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", subname);
pg_log_debug("command is: %s", str->data);
@@ -1069,7 +1099,7 @@ drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", subname, perdb->dbname, PQerrorMessage(conn));
PQclear(res);
}
@@ -1088,18 +1118,21 @@ drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* printing purposes.
*/
static void
-set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+set_replication_progress(PGconn *conn, LogicalRepPerdbInfo *perdb, const char *lsn)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
Oid suboid;
char originname[NAMEDATALEN];
char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
appendPQExpBuffer(str,
- "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", subname);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
@@ -1140,7 +1173,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
PQclear(res);
pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
- originname, lsnstr, dbinfo->dbname);
+ originname, lsnstr, perdb->dbname);
resetPQExpBuffer(str);
appendPQExpBuffer(str,
@@ -1154,7 +1187,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
pg_log_error("could not set replication progress for the subscription \"%s\": %s",
- dbinfo->subname, PQresultErrorMessage(res));
+ subname, PQresultErrorMessage(res));
PQfinish(conn);
exit(1);
}
@@ -1173,16 +1206,20 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
* of this setup.
*/
static void
-enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+enable_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
- appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", subname,
+ perdb->dbname);
+
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", subname);
pg_log_debug("command is: %s", str->data);
@@ -1191,7 +1228,7 @@ enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
- pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ pg_log_error("could not enable subscription \"%s\": %s", subname,
PQerrorMessage(conn));
PQfinish(conn);
exit(1);
@@ -1203,6 +1240,61 @@ enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
destroyPQExpBuffer(str);
}
+static void
+start_standby_server(StandbyInfo *standby, unsigned short subport,
+ char *server_start_log)
+{
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+ int rc;
+ char *pg_ctl_cmd;
+
+ if (server_start_log[0] == '\0')
+ {
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ len = snprintf(server_start_log, MAXPGPATH,
+ "%s/%s/server_start_%s.log", standby->pgdata,
+ PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+ }
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" start -D \"%s\" -s -o \"-p %d\" -l \"%s\"",
+ standby->bindir,
+ standby->pgdata, subport, server_start_log);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+}
+
+static char *
+construct_sub_conninfo(char *username, unsigned short subport)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *ret;
+
+ if (username)
+ appendPQExpBuffer(buf, "user=%s ", username);
+
+ appendPQExpBuffer(buf, "port=%d fallback_application_name=%s",
+ subport, progname);
+
+ ret = pg_strdup(buf->data);
+
+ destroyPQExpBuffer(buf);
+
+ return ret;
+}
+
int
main(int argc, char **argv)
{
@@ -1214,6 +1306,10 @@ main(int argc, char **argv)
{"publisher-conninfo", required_argument, NULL, 'P'},
{"subscriber-conninfo", required_argument, NULL, 'S'},
{"database", required_argument, NULL, 'd'},
+ {"timeout", required_argument, NULL, 't'},
+ {"username", required_argument, NULL, 'u'},
+ {"port", required_argument, NULL, 'p'},
+ {"retain", no_argument, NULL, 'r'},
{"dry-run", no_argument, NULL, 'n'},
{"verbose", no_argument, NULL, 'v'},
{NULL, 0, NULL, 0}
@@ -1225,20 +1321,15 @@ main(int argc, char **argv)
char *pg_ctl_cmd;
- char *base_dir;
- char *server_start_log;
-
- char timebuf[128];
- struct timeval time;
- time_t tt;
+ char base_dir[MAXPGPATH];
+ char server_start_log[MAXPGPATH] = {0};
int len;
- char *pub_base_conninfo = NULL;
- char *sub_base_conninfo = NULL;
char *dbname_conninfo = NULL;
- uint64 pub_sysid;
- uint64 sub_sysid;
+ unsigned short subport = DEF_PGSPORT;
+ char *username = NULL;
+
struct stat statbuf;
PGconn *conn;
@@ -1250,6 +1341,13 @@ main(int argc, char **argv)
int i;
+ PGresult *res;
+
+ char *wal_level;
+ int max_replication_slots;
+ int nslots_old;
+ int nslots_new;
+
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
progname = get_progname(argv[0]);
@@ -1286,28 +1384,40 @@ main(int argc, char **argv)
}
#endif
- while ((c = getopt_long(argc, argv, "D:P:S:d:nv",
+ while ((c = getopt_long(argc, argv, "D:P:S:d:t:u:p:rnv",
long_options, &option_index)) != -1)
{
switch (c)
{
case 'D':
- subscriber_dir = pg_strdup(optarg);
+ standby.pgdata = pg_strdup(optarg);
+ canonicalize_path(standby.pgdata);
break;
case 'P':
pub_conninfo_str = pg_strdup(optarg);
break;
- case 'S':
- sub_conninfo_str = pg_strdup(optarg);
- break;
case 'd':
/* Ignore duplicated database names. */
if (!simple_string_list_member(&database_names, optarg))
{
simple_string_list_append(&database_names, optarg);
- num_dbs++;
+ dbarr.ndbs++;
}
break;
+ case 't':
+ wait_seconds = atoi(optarg);
+ break;
+ case 'u':
+ pfree(username);
+ username = pg_strdup(optarg);
+ break;
+ case 'p':
+ if ((subport = atoi(optarg)) <= 0)
+ pg_fatal("invalid old port number");
+ break;
+ case 'r':
+ retain = true;
+ break;
case 'n':
dry_run = true;
break;
@@ -1335,7 +1445,7 @@ main(int argc, char **argv)
/*
* Required arguments
*/
- if (subscriber_dir == NULL)
+ if (standby.pgdata == NULL)
{
pg_log_error("no subscriber data directory specified");
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1358,21 +1468,14 @@ main(int argc, char **argv)
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
exit(1);
}
- pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
- "publisher");
- if (pub_base_conninfo == NULL)
- exit(1);
- if (sub_conninfo_str == NULL)
- {
- pg_log_error("no subscriber connection string specified");
- pg_log_error_hint("Try \"%s --help\" for more information.", progname);
- exit(1);
- }
- sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
- if (sub_base_conninfo == NULL)
+ primary.base_conninfo = get_base_conninfo(pub_conninfo_str,
+ dbname_conninfo);
+ if (primary.base_conninfo == NULL)
exit(1);
+ standby.base_conninfo = construct_sub_conninfo(username, subport);
+
if (database_names.head == NULL)
{
pg_log_info("no database was specified");
@@ -1385,7 +1488,7 @@ main(int argc, char **argv)
if (dbname_conninfo)
{
simple_string_list_append(&database_names, dbname_conninfo);
- num_dbs++;
+ dbarr.ndbs++;
pg_log_info("database \"%s\" was extracted from the publisher connection string",
dbname_conninfo);
@@ -1399,25 +1502,25 @@ main(int argc, char **argv)
}
/*
- * Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
+ * Get the absolute path of binaries on the subscriber.
*/
- if (!get_exec_path(argv[0]))
+ if (!get_exec_base_path(argv[0]))
exit(1);
/* rudimentary check for a data directory. */
- if (!check_data_directory(subscriber_dir))
+ if (!check_data_directory(standby.pgdata))
exit(1);
- /* Store database information for publisher and subscriber. */
- dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+ /* Store database information to dbarr */
+ store_db_names(&dbarr.perdb, dbarr.ndbs);
/*
* Check if the subscriber data directory has the same system identifier
* than the publisher data directory.
*/
- pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
- sub_sysid = get_control_from_datadir(subscriber_dir);
- if (pub_sysid != sub_sysid)
+ get_sysid_for_primary(&primary, dbarr.perdb[0].dbname);
+ get_control_for_standby(&standby);
+ if (primary.sysid != standby.sysid)
{
pg_log_error("subscriber data directory is not a copy of the source database cluster");
exit(1);
@@ -1426,8 +1529,8 @@ main(int argc, char **argv)
/*
* Create the output directory to store any data generated by this tool.
*/
- base_dir = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s",
+ standby.pgdata, PGS_OUTPUT_DIR);
if (len >= MAXPGPATH)
{
pg_log_error("directory path for subscriber is too long");
@@ -1441,7 +1544,153 @@ main(int argc, char **argv)
}
/* subscriber PID file. */
- snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid",
+ standby.pgdata);
+
+ /* Start the standby server anyway */
+ start_standby_server(&standby, subport, server_start_log);
+
+ /*
+ * Check wal_level in publisher and the max_replication_slots of publisher
+ */
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "SELECT count(*) from pg_replication_slots;");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain number of replication slots");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not determine parameter settings on publisher");
+ exit(1);
+ }
+
+ nslots_old = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ res = PQexec(conn, "SELECT setting FROM pg_settings "
+ "WHERE name IN ('wal_level', 'max_replication_slots') "
+ "ORDER BY name DESC;");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain guc parameters on publisher");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 2)
+ {
+ pg_log_error("could not determine parameter settings on publisher");
+ exit(1);
+ }
+
+ wal_level = PQgetvalue(res, 0, 0);
+
+ if (strcmp(wal_level, "logical") != 0)
+ {
+ pg_log_error("wal_level must be \"logical\", but is set to \"%s\"", wal_level);
+ exit(1);
+ }
+
+ max_replication_slots = atoi(PQgetvalue(res, 1, 0));
+ nslots_new = nslots_old + dbarr.ndbs + 1;
+
+ if (nslots_new > max_replication_slots)
+ {
+ pg_log_error("max_replication_slots (%d) must be greater than or equal to "
+ "the number of replication slots required (%d)", max_replication_slots, nslots_new);
+ exit(1);
+ }
+
+ PQclear(res);
+ disconnect_database(conn);
+
+ conn = connect_database(standby.base_conninfo, dbarr.perdb[0].dbname);
+ if (conn == NULL)
+ exit(1);
+
+ /*
+ * Check the max_replication_slots in subscriber
+ */
+ res = PQexec(conn, "SELECT count(*) from pg_replication_slots;");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain number of replication slots on subscriber");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not determine parameter settings on subscriber");
+ exit(1);
+ }
+
+ nslots_old = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ res = PQexec(conn, "SELECT setting FROM pg_settings "
+ "WHERE name = 'max_replication_slots';");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain guc parameters");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not determine parameter settings on publisher");
+ exit(1);
+ }
+
+ max_replication_slots = atoi(PQgetvalue(res, 0, 0));
+ nslots_new = nslots_old + dbarr.ndbs;
+
+ if (nslots_new > max_replication_slots)
+ {
+ pg_log_error("max_replication_slots (%d) must be greater than or equal to "
+ "the number of replication slots required (%d)", max_replication_slots, nslots_new);
+ exit(1);
+ }
+
+ PQclear(res);
+
+ /*
+ * Exit the pg_subscriber if the node is not a standby server.
+ */
+ res = PQexec(conn, "SELECT pg_catalog.pg_is_in_recovery()");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain recovery progress");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("unexpected result from pg_is_in_recovery function");
+ exit(1);
+ }
+
+ /* Check if the server is in recovery */
+ if (strcmp(PQgetvalue(res, 0, 0), "t") != 0)
+ {
+ pg_log_error("pg_subscriber is supported only on standby server");
+ exit(1);
+ }
+
+ PQclear(res);
+ disconnect_database(conn);
+
+ /* subscriber PID file. */
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", standby.pgdata);
/*
* Stop the subscriber if it is a standby server. Before executing the
@@ -1457,14 +1706,18 @@ main(int argc, char **argv)
* replication slot has no use after the transformation, hence, it
* will be removed at the end of this process.
*/
- primary_slot_name = use_primary_slot_name();
- if (primary_slot_name != NULL)
- pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+ standby.primary_slot_name = use_primary_slot_name(&primary,
+ &standby,
+ &dbarr.perdb[0]);
+ if (standby.primary_slot_name != NULL)
+ pg_log_info("primary has replication slot \"%s\"",
+ standby.primary_slot_name);
pg_log_info("subscriber is up and running");
pg_log_info("stopping the server to start the transformation steps");
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
+ standby.bindir, standby.pgdata);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
}
@@ -1472,7 +1725,7 @@ main(int argc, char **argv)
/*
* Create a replication slot for each database on the publisher.
*/
- if (!create_all_logical_replication_slots(dbinfo))
+ if (!create_all_logical_replication_slots(&primary, &dbarr))
exit(1);
/*
@@ -1492,11 +1745,11 @@ main(int argc, char **argv)
* replication connection open (depending when base backup was taken, the
* connection should be open for a few hours).
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
if (conn == NULL)
exit(1);
- consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
- temp_replslot);
+ consistent_lsn = create_logical_replication_slot(conn, true,
+ &dbarr.perdb[0]);
/*
* Write recovery parameters.
@@ -1522,7 +1775,7 @@ main(int argc, char **argv)
{
appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
consistent_lsn);
- WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ WriteRecoveryConfig(conn, standby.pgdata, recoveryconfcontents);
}
disconnect_database(conn);
@@ -1532,54 +1785,29 @@ main(int argc, char **argv)
* Start subscriber and wait until accepting connections.
*/
pg_log_info("starting the subscriber");
-
- /* append timestamp with ISO 8601 format. */
- gettimeofday(&time, NULL);
- tt = (time_t) time.tv_sec;
- strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
- snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
- ".%03d", (int) (time.tv_usec / 1000));
-
- server_start_log = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(server_start_log, MAXPGPATH, "%s/%s/server_start_%s.log", subscriber_dir, PGS_OUTPUT_DIR, timebuf);
- if (len >= MAXPGPATH)
- {
- pg_log_error("log file path is too long");
- exit(1);
- }
-
- pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, subscriber_dir, server_start_log);
- rc = system(pg_ctl_cmd);
- pg_ctl_status(pg_ctl_cmd, rc, 1);
+ start_standby_server(&standby, subport, server_start_log);
/*
* Waiting the subscriber to be promoted.
*/
- wait_for_end_recovery(dbinfo[0].subconninfo);
+ wait_for_end_recovery(standby.base_conninfo, dbarr.perdb[0].dbname);
/*
* Create a publication for each database. This step should be executed
* after promoting the subscriber to avoid replicating unnecessary
* objects.
*/
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
- char pubname[NAMEDATALEN];
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
/* Connect to publisher. */
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary.base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
- /*
- * Build the publication name. The name must not exceed NAMEDATALEN -
- * 1. This current schema uses a maximum of 35 characters (14 + 10 +
- * '\0').
- */
- snprintf(pubname, sizeof(pubname), "pg_subscriber_%u", dbinfo[i].oid);
- dbinfo[i].pubname = pg_strdup(pubname);
-
- create_publication(conn, &dbinfo[i]);
+ /* Also create a publication */
+ create_publication(conn, &primary, perdb);
disconnect_database(conn);
}
@@ -1587,20 +1815,25 @@ main(int argc, char **argv)
/*
* Create a subscription for each database.
*/
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
+
/* Connect to subscriber. */
- conn = connect_database(dbinfo[i].subconninfo);
+ conn = connect_database(standby.base_conninfo, perdb->dbname);
+
if (conn == NULL)
exit(1);
- create_subscription(conn, &dbinfo[i]);
+ create_subscription(conn, &standby, primary.base_conninfo, perdb);
/* Set the replication progress to the correct LSN. */
- set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+ set_replication_progress(conn, perdb, consistent_lsn);
/* Enable subscription. */
- enable_subscription(conn, &dbinfo[i]);
+ enable_subscription(conn, perdb);
+
+ drop_publication(conn, perdb);
disconnect_database(conn);
}
@@ -1613,19 +1846,21 @@ main(int argc, char **argv)
* XXX we might not fail here. Instead, we provide a warning so the user
* eventually drops the replication slot later.
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
if (conn == NULL)
{
- pg_log_warning("could not drop transient replication slot \"%s\" on publisher", temp_replslot);
- pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
+ char *primary_slot_name = standby.primary_slot_name;
+
if (primary_slot_name != NULL)
pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
}
else
{
- drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[0];
+ char *primary_slot_name = standby.primary_slot_name;
+
if (primary_slot_name != NULL)
- drop_replication_slot(conn, &dbinfo[0], primary_slot_name);
+ drop_replication_slot(conn, perdb, primary_slot_name);
disconnect_database(conn);
}
@@ -1634,20 +1869,22 @@ main(int argc, char **argv)
*/
pg_log_info("stopping the subscriber");
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
+ standby.bindir, standby.pgdata);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
/*
* Change system identifier.
*/
- modify_sysid(pg_resetwal_path, subscriber_dir);
+ modify_sysid(standby.bindir, standby.pgdata);
/*
* Remove log file generated by this tool, if it runs successfully.
* Otherwise, file is kept that may provide useful debugging information.
*/
- unlink(server_start_log);
+ if (!retain)
+ unlink(server_start_log);
success = true;
diff --git a/src/bin/pg_basebackup/t/040_pg_subscriber.pl b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
index 4ebff76b2d..9915b8cb3c 100644
--- a/src/bin/pg_basebackup/t/040_pg_subscriber.pl
+++ b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
@@ -37,8 +37,13 @@ command_fails(
'--verbose',
'--pgdata', $datadir,
'--publisher-conninfo', 'dbname=postgres',
- '--subscriber-conninfo', 'dbname=postgres'
],
'no database name specified');
-
+command_fails(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres',
+ ],
+ 'subscriber connection string specnfied non-local server');
done_testing();
diff --git a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
index fbcd0fc82b..4e26607611 100644
--- a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
+++ b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
@@ -51,25 +51,27 @@ $node_s->start;
$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('second row')");
$node_p->wait_for_replay_catchup($node_s);
+$node_f->stop;
+
# Run pg_subscriber on about-to-fail node F
command_fails(
[
'pg_subscriber', '--verbose',
'--pgdata', $node_f->data_dir,
'--publisher-conninfo', $node_p->connstr('pg1'),
- '--subscriber-conninfo', $node_f->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
],
'subscriber data directory is not a copy of the source database cluster');
+$node_s->stop;
+
# dry run mode on node S
command_ok(
[
'pg_subscriber', '--verbose', '--dry-run',
'--pgdata', $node_s->data_dir,
'--publisher-conninfo', $node_p->connstr('pg1'),
- '--subscriber-conninfo', $node_s->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
],
@@ -82,6 +84,7 @@ $node_s->start;
# Check if node S is still a standby
is($node_s->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
't', 'standby is in recovery');
+$node_s->stop;
# Run pg_subscriber on node S
command_ok(
@@ -89,7 +92,6 @@ command_ok(
'pg_subscriber', '--verbose',
'--pgdata', $node_s->data_dir,
'--publisher-conninfo', $node_p->connstr('pg1'),
- '--subscriber-conninfo', $node_s->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
],
--
2.34.1
[application/octet-stream] v6-0001-Creates-a-new-logical-replica-from-a-standby-serv.patch (69.2K, ../../CANhcyEXhpfWgc0-FdHmUG6WGD=2cxYqBePr1yxJ8VSZ0gV3XKA@mail.gmail.com/4-v6-0001-Creates-a-new-logical-replica-from-a-standby-serv.patch)
download | inline diff:
From 7b808c5a927e3abf98b1e3bb62ec64dd5b80b013 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email protected]>
Date: Mon, 5 Jun 2023 14:39:40 -0400
Subject: [PATCH v6 1/3] Creates a new logical replica from a standby server
A new tool called pg_subscriber can convert a physical replica into a
logical replica. It runs on the target server and should be able to
connect to the source server (publisher) and the target server
(subscriber).
The conversion requires a few steps. Check if the target data directory
has the same system identifier than the source data directory. Stop the
target server if it is running as a standby server. Create one
replication slot per specified database on the source server. One
additional replication slot is created at the end to get the consistent
LSN (This consistent LSN will be used as (a) a stopping point for the
recovery process and (b) a starting point for the subscriptions). Write
recovery parameters into the target data directory and start the target
server (Wait until the target server is promoted). Create one
publication (FOR ALL TABLES) per specified database on the source
server. Create one subscription per specified database on the target
server (Use replication slot and publication created in a previous step.
Don't enable the subscriptions yet). Sets the replication progress to
the consistent LSN that was got in a previous step. Enable the
subscription for each specified database on the target server. Remove
the additional replication slot that was used to get the consistent LSN.
Stop the target server. Change the system identifier from the target
server.
Depending on your workload and database size, creating a logical replica
couldn't be an option due to resource constraints (WAL backlog should be
available until all table data is synchronized). The initial data copy
and the replication progress tends to be faster on a physical replica.
The purpose of this tool is to speed up a logical replica setup.
---
doc/src/sgml/ref/allfiles.sgml | 1 +
doc/src/sgml/ref/pg_subscriber.sgml | 284 +++
doc/src/sgml/reference.sgml | 1 +
src/bin/pg_basebackup/.gitignore | 1 +
src/bin/pg_basebackup/Makefile | 8 +-
src/bin/pg_basebackup/meson.build | 19 +
src/bin/pg_basebackup/pg_subscriber.c | 1657 +++++++++++++++++
src/bin/pg_basebackup/t/040_pg_subscriber.pl | 44 +
.../t/041_pg_subscriber_standby.pl | 139 ++
9 files changed, 2153 insertions(+), 1 deletion(-)
create mode 100644 doc/src/sgml/ref/pg_subscriber.sgml
create mode 100644 src/bin/pg_basebackup/pg_subscriber.c
create mode 100644 src/bin/pg_basebackup/t/040_pg_subscriber.pl
create mode 100644 src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index 4a42999b18..3862c976d7 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -214,6 +214,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY pgResetwal SYSTEM "pg_resetwal.sgml">
<!ENTITY pgRestore SYSTEM "pg_restore.sgml">
<!ENTITY pgRewind SYSTEM "pg_rewind.sgml">
+<!ENTITY pgSubscriber SYSTEM "pg_subscriber.sgml">
<!ENTITY pgVerifyBackup SYSTEM "pg_verifybackup.sgml">
<!ENTITY pgtestfsync SYSTEM "pgtestfsync.sgml">
<!ENTITY pgtesttiming SYSTEM "pgtesttiming.sgml">
diff --git a/doc/src/sgml/ref/pg_subscriber.sgml b/doc/src/sgml/ref/pg_subscriber.sgml
new file mode 100644
index 0000000000..553185c35f
--- /dev/null
+++ b/doc/src/sgml/ref/pg_subscriber.sgml
@@ -0,0 +1,284 @@
+<!--
+doc/src/sgml/ref/pg_subscriber.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="app-pgsubscriber">
+ <indexterm zone="app-pgsubscriber">
+ <primary>pg_subscriber</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle><application>pg_subscriber</application></refentrytitle>
+ <manvolnum>1</manvolnum>
+ <refmiscinfo>Application</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>pg_subscriber</refname>
+ <refpurpose>create a new logical replica from a standby server</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_subscriber</command>
+ <arg rep="repeat"><replaceable>option</replaceable></arg>
+ </cmdsynopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+ <para>
+ <application>pg_subscriber</application> takes the publisher and subscriber
+ connection strings, a cluster directory from a standby server and a list of
+ database names and it sets up a new logical replica using the physical
+ recovery process.
+ </para>
+
+ <para>
+ The <application>pg_subscriber</application> should be run at the target
+ server. The source server (known as publisher server) should accept logical
+ replication connections from the target server (known as subscriber server).
+ The target server should accept local logical replication connection.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Options</title>
+
+ <para>
+ <application>pg_subscriber</application> accepts the following
+ command-line arguments:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-D <replaceable class="parameter">directory</replaceable></option></term>
+ <term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term>
+ <listitem>
+ <para>
+ The target directory that contains a cluster directory from a standby
+ server.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-P <replaceable class="parameter">conninfo</replaceable></option></term>
+ <term><option>--publisher-conninfo=<replaceable class="parameter">conninfo</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the publisher. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-S <replaceable class="parameter">conninfo</replaceable></option></term>
+ <term><option>--subscriber-conninfo=<replaceable class="parameter">conninfo</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the subscriber. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-d <replaceable class="parameter">dbname</replaceable></option></term>
+ <term><option>--database=<replaceable class="parameter">dbname</replaceable></option></term>
+ <listitem>
+ <para>
+ The database name to create the subscription. Multiple databases can be
+ selected by writing multiple <option>-d</option> switches.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
+ <listitem>
+ <para>
+ Do everything except actually modifying the target directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-v</option></term>
+ <term><option>--verbose</option></term>
+ <listitem>
+ <para>
+ Enables verbose mode. This will cause
+ <application>pg_subscriber</application> to output progress messages
+ and detailed information about each step.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ Other options are also available:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-V</option></term>
+ <term><option>--version</option></term>
+ <listitem>
+ <para>
+ Print the <application>pg_subscriber</application> version and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-?</option></term>
+ <term><option>--help</option></term>
+ <listitem>
+ <para>
+ Show help about <application>pg_subscriber</application> command
+ line arguments, and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Notes</title>
+
+ <para>
+ The transformation proceeds in the following steps:
+ </para>
+
+ <procedure>
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the given target data
+ directory has the same system identifier than the source data directory.
+ Since it uses the recovery process as one of the steps, it starts the
+ target server as a replica from the source server. If the system
+ identifier is not the same, <application>pg_subscriber</application> will
+ terminate with an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the target data
+ directory is used by a standby server. Stop the standby server if it is
+ running. One of the next steps is to add some recovery parameters that
+ requires a server start. This step avoids an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one replication slot for
+ each specified database on the source server. The replication slot name
+ contains a <literal>pg_subscriber</literal> prefix. These replication
+ slots will be used by the subscriptions in a future step. Another
+ replication slot is used to get a consistent start location. This
+ consistent LSN will be used as a stopping point in the <xref
+ linkend="guc-recovery-target-lsn"/> parameter and by the
+ subscriptions as a replication starting point. It guarantees that no
+ transaction will be lost.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> writes recovery parameters into
+ the target data directory and start the target server. It specifies a LSN
+ (consistent LSN that was obtained in the previous step) of write-ahead
+ log location up to which recovery will proceed. It also specifies
+ <literal>promote</literal> as the action that the server should take once
+ the recovery target is reached. This step finishes once the server ends
+ standby mode and is accepting read-write operations.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Next, <application>pg_subscriber</application> creates one publication
+ for each specified database on the source server. Each publication
+ replicates changes for all tables in the database. The publication name
+ contains a <literal>pg_subscriber</literal> prefix. These publication
+ will be used by a corresponding subscription in a next step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one subscription for
+ each specified database on the target server. Each subscription name
+ contains a <literal>pg_subscriber</literal> prefix. The replication slot
+ name is identical to the subscription name. It does not copy existing data
+ from the source server. It does not create a replication slot. Instead, it
+ uses the replication slot that was created in a previous step. The
+ subscription is created but it is not enabled yet. The reason is the
+ replication progress must be set to the consistent LSN but replication
+ origin name contains the subscription oid in its name. Hence, the
+ subscription will be enabled in a separate step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> sets the replication progress to
+ the consistent LSN that was obtained in a previous step. When the target
+ server started the recovery process, it caught up to the consistent LSN.
+ This is the exact LSN to be used as a initial location for each
+ subscription.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Finally, <application>pg_subscriber</application> enables the subscription
+ for each specified database on the target server. The subscription starts
+ streaming from the consistent LSN.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> removes the additional replication
+ slot that was used to get the consistent LSN on the source server.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> stops the target server to change
+ its system identifier.
+ </para>
+ </step>
+ </procedure>
+ </refsect1>
+
+ <refsect1>
+ <title>Examples</title>
+
+ <para>
+ To create a logical replica for databases <literal>hr</literal> and
+ <literal>finance</literal> from a standby server at <literal>foo</literal>:
+<screen>
+<prompt>$</prompt> <userinput>pg_subscriber -D /usr/local/pgsql/data -P "host=foo" -S "host=localhost" -d hr -d finance</userinput>
+</screen>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="app-pgbasebackup"/></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index aa94f6adf6..266f4e515a 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -285,6 +285,7 @@
&pgCtl;
&pgResetwal;
&pgRewind;
+ &pgSubscriber;
&pgtestfsync;
&pgtesttiming;
&pgupgrade;
diff --git a/src/bin/pg_basebackup/.gitignore b/src/bin/pg_basebackup/.gitignore
index 26048bdbd8..0e5384a1d5 100644
--- a/src/bin/pg_basebackup/.gitignore
+++ b/src/bin/pg_basebackup/.gitignore
@@ -1,5 +1,6 @@
/pg_basebackup
/pg_receivewal
/pg_recvlogical
+/pg_subscriber
/tmp_check/
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index abfb6440ec..f6281b7676 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -44,7 +44,7 @@ BBOBJS = \
bbstreamer_tar.o \
bbstreamer_zstd.o
-all: pg_basebackup pg_receivewal pg_recvlogical
+all: pg_basebackup pg_receivewal pg_recvlogical pg_subscriber
pg_basebackup: $(BBOBJS) $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) $(BBOBJS) $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
@@ -55,10 +55,14 @@ pg_receivewal: pg_receivewal.o $(OBJS) | submake-libpq submake-libpgport submake
pg_recvlogical: pg_recvlogical.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_recvlogical.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_subscriber: $(WIN32RES) pg_subscriber.o | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
install: all installdirs
$(INSTALL_PROGRAM) pg_basebackup$(X) '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
$(INSTALL_PROGRAM) pg_receivewal$(X) '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
$(INSTALL_PROGRAM) pg_recvlogical$(X) '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ $(INSTALL_PROGRAM) pg_subscriber$(X) '$(DESTDIR)$(bindir)/pg_subscriber$(X)'
installdirs:
$(MKDIR_P) '$(DESTDIR)$(bindir)'
@@ -67,10 +71,12 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ rm -f '$(DESTDIR)$(bindir)/pg_subscriber$(X)'
clean distclean:
rm -f pg_basebackup$(X) pg_receivewal$(X) pg_recvlogical$(X) \
$(BBOBJS) pg_receivewal.o pg_recvlogical.o \
+ pg_subscriber$(X) pg_subscriber.o \
$(OBJS)
rm -rf tmp_check
diff --git a/src/bin/pg_basebackup/meson.build b/src/bin/pg_basebackup/meson.build
index f7e60e6670..ccfd7bb7a5 100644
--- a/src/bin/pg_basebackup/meson.build
+++ b/src/bin/pg_basebackup/meson.build
@@ -75,6 +75,23 @@ pg_recvlogical = executable('pg_recvlogical',
)
bin_targets += pg_recvlogical
+pg_subscriber_sources = files(
+ 'pg_subscriber.c'
+)
+
+if host_system == 'windows'
+ pg_subscriber_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'pg_subscriber',
+ '--FILEDESC', 'pg_subscriber - create a new logical replica from a standby server',])
+endif
+
+pg_subscriber = executable('pg_subscriber',
+ pg_subscriber_sources,
+ dependencies: [frontend_code, libpq],
+ kwargs: default_bin_args,
+)
+bin_targets += pg_subscriber
+
tests += {
'name': 'pg_basebackup',
'sd': meson.current_source_dir(),
@@ -89,6 +106,8 @@ tests += {
't/011_in_place_tablespace.pl',
't/020_pg_receivewal.pl',
't/030_pg_recvlogical.pl',
+ 't/040_pg_subscriber.pl',
+ 't/041_pg_subscriber_standby.pl',
],
},
}
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
new file mode 100644
index 0000000000..e998c29f9e
--- /dev/null
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -0,0 +1,1657 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_subscriber.c
+ * Create a new logical replica from a standby server
+ *
+ * Copyright (C) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_subscriber/pg_subscriber.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include <signal.h>
+#include <sys/stat.h>
+#include <sys/time.h>
+#include <sys/wait.h>
+#include <time.h>
+
+#include "access/xlogdefs.h"
+#include "catalog/pg_control.h"
+#include "common/connect.h"
+#include "common/controldata_utils.h"
+#include "common/file_perm.h"
+#include "common/file_utils.h"
+#include "common/logging.h"
+#include "fe_utils/recovery_gen.h"
+#include "fe_utils/simple_list.h"
+#include "getopt_long.h"
+#include "utils/pidfile.h"
+
+#define PGS_OUTPUT_DIR "pg_subscriber_output.d"
+
+typedef struct LogicalRepInfo
+{
+ Oid oid; /* database OID */
+ char *dbname; /* database name */
+ char *pubconninfo; /* publication connection string for logical
+ * replication */
+ char *subconninfo; /* subscription connection string for logical
+ * replication */
+ char *pubname; /* publication name */
+ char *subname; /* subscription name (also replication slot
+ * name) */
+
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepInfo;
+
+static void cleanup_objects_atexit(void);
+static void usage();
+static char *get_base_conninfo(char *conninfo, char *dbname,
+ const char *noderole);
+static bool get_exec_path(const char *path);
+static bool check_data_directory(const char *datadir);
+static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
+static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
+static PGconn *connect_database(const char *conninfo);
+static void disconnect_database(PGconn *conn);
+static uint64 get_sysid_from_conn(const char *conninfo);
+static uint64 get_control_from_datadir(const char *datadir);
+static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
+static char *use_primary_slot_name(void);
+static bool create_all_logical_replication_slots(LogicalRepInfo *dbinfo);
+static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name);
+static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
+static void wait_for_end_recovery(const char *conninfo);
+static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+
+#define USEC_PER_SEC 1000000
+#define WAIT_INTERVAL 1 /* 1 second */
+
+/* Options */
+static const char *progname;
+
+static char *subscriber_dir = NULL;
+static char *pub_conninfo_str = NULL;
+static char *sub_conninfo_str = NULL;
+static SimpleStringList database_names = {NULL, NULL};
+static char *primary_slot_name = NULL;
+static bool dry_run = false;
+
+static bool success = false;
+
+static char *pg_ctl_path = NULL;
+static char *pg_resetwal_path = NULL;
+
+static LogicalRepInfo *dbinfo;
+static int num_dbs = 0;
+
+static char temp_replslot[NAMEDATALEN] = {0};
+static bool made_transient_replslot = false;
+
+enum WaitPMResult
+{
+ POSTMASTER_READY,
+ POSTMASTER_STANDBY,
+ POSTMASTER_STILL_STARTING,
+ POSTMASTER_FAILED
+};
+
+
+/*
+ * Cleanup objects that were created by pg_subscriber if there is an error.
+ *
+ * Replication slots, publications and subscriptions are created. Depending on
+ * the step it failed, it should remove the already created objects if it is
+ * possible (sometimes it won't work due to a connection issue).
+ */
+static void
+cleanup_objects_atexit(void)
+{
+ PGconn *conn;
+ int i;
+
+ if (success)
+ return;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ if (dbinfo[i].made_subscription)
+ {
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn != NULL)
+ {
+ drop_subscription(conn, &dbinfo[i]);
+ disconnect_database(conn);
+ }
+ }
+
+ if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ {
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn != NULL)
+ {
+ if (dbinfo[i].made_publication)
+ drop_publication(conn, &dbinfo[i]);
+ if (dbinfo[i].made_replslot)
+ drop_replication_slot(conn, &dbinfo[i], NULL);
+ disconnect_database(conn);
+ }
+ }
+ }
+
+ if (made_transient_replslot)
+ {
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn != NULL)
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ disconnect_database(conn);
+ }
+ }
+}
+
+static void
+usage(void)
+{
+ printf(_("%s creates a new logical replica from a standby server.\n\n"),
+ progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION]...\n"), progname);
+ printf(_("\nOptions:\n"));
+ printf(_(" -D, --pgdata=DATADIR location for the subscriber data directory\n"));
+ printf(_(" -P, --publisher-conninfo=CONNINFO publisher connection string\n"));
+ printf(_(" -S, --subscriber-conninfo=CONNINFO subscriber connection string\n"));
+ printf(_(" -d, --database=DBNAME database to create a subscription\n"));
+ printf(_(" -n, --dry-run stop before modifying anything\n"));
+ printf(_(" -v, --verbose output verbose messages\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
+ printf(_("\nReport bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+ printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
+
+/*
+ * Validate a connection string. Returns a base connection string that is a
+ * connection string without a database name plus a fallback application name.
+ * Since we might process multiple databases, each database name will be
+ * appended to this base connection string to provide a final connection string.
+ * If the second argument (dbname) is not null, returns dbname if the provided
+ * connection string contains it. If option --database is not provided, uses
+ * dbname as the only database to setup the logical replica.
+ * It is the caller's responsibility to free the returned connection string and
+ * dbname.
+ */
+static char *
+get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ PQconninfoOption *conn_opts = NULL;
+ PQconninfoOption *conn_opt;
+ char *errmsg = NULL;
+ char *ret;
+ int i;
+
+ pg_log_info("validating connection string on %s", noderole);
+
+ conn_opts = PQconninfoParse(conninfo, &errmsg);
+ if (conn_opts == NULL)
+ {
+ pg_log_error("could not parse connection string: %s", errmsg);
+ return NULL;
+ }
+
+ i = 0;
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (strcmp(conn_opt->keyword, "dbname") == 0 && conn_opt->val != NULL)
+ {
+ if (dbname)
+ dbname = pg_strdup(conn_opt->val);
+ continue;
+ }
+
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0')
+ {
+ if (i > 0)
+ appendPQExpBufferChar(buf, ' ');
+ appendPQExpBuffer(buf, "%s=%s", conn_opt->keyword, conn_opt->val);
+ i++;
+ }
+ }
+
+ if (i > 0)
+ appendPQExpBufferChar(buf, ' ');
+ appendPQExpBuffer(buf, "fallback_application_name=%s", progname);
+
+ ret = pg_strdup(buf->data);
+
+ destroyPQExpBuffer(buf);
+ PQconninfoFree(conn_opts);
+
+ return ret;
+}
+
+/*
+ * Get the absolute path from other PostgreSQL binaries (pg_ctl and
+ * pg_resetwal) that is used by it.
+ */
+static bool
+get_exec_path(const char *path)
+{
+ int rc;
+
+ pg_ctl_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_ctl",
+ "pg_ctl (PostgreSQL) " PG_VERSION "\n",
+ pg_ctl_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_ctl", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_ctl", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
+
+ pg_resetwal_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_resetwal",
+ "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
+ pg_resetwal_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_resetwal", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_resetwal", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+
+ return true;
+}
+
+/*
+ * Is it a cluster directory? These are preliminary checks. It is far from
+ * making an accurate check. If it is not a clone from the publisher, it will
+ * eventually fail in a future step.
+ */
+static bool
+check_data_directory(const char *datadir)
+{
+ struct stat statbuf;
+ char versionfile[MAXPGPATH];
+
+ pg_log_info("checking if directory \"%s\" is a cluster data directory",
+ datadir);
+
+ if (stat(datadir, &statbuf) != 0)
+ {
+ if (errno == ENOENT)
+ pg_log_error("data directory \"%s\" does not exist", datadir);
+ else
+ pg_log_error("could not access directory \"%s\": %s", datadir, strerror(errno));
+
+ return false;
+ }
+
+ snprintf(versionfile, MAXPGPATH, "%s/PG_VERSION", datadir);
+ if (stat(versionfile, &statbuf) != 0 && errno == ENOENT)
+ {
+ pg_log_error("directory \"%s\" is not a database cluster directory", datadir);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Append database name into a base connection string.
+ *
+ * dbname is the only parameter that changes so it is not included in the base
+ * connection string. This function concatenates dbname to build a "real"
+ * connection string.
+ */
+static char *
+concat_conninfo_dbname(const char *conninfo, const char *dbname)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *ret;
+
+ Assert(conninfo != NULL);
+
+ appendPQExpBufferStr(buf, conninfo);
+ appendPQExpBuffer(buf, " dbname=%s", dbname);
+
+ ret = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+
+ return ret;
+}
+
+/*
+ * Store publication and subscription information.
+ */
+static LogicalRepInfo *
+store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+{
+ LogicalRepInfo *dbinfo;
+ SimpleStringListCell *cell;
+ int i = 0;
+
+ dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+
+ for (cell = database_names.head; cell; cell = cell->next)
+ {
+ char *conninfo;
+
+ /* Publisher. */
+ conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
+ dbinfo[i].pubconninfo = conninfo;
+ dbinfo[i].dbname = cell->val;
+ dbinfo[i].made_replslot = false;
+ dbinfo[i].made_publication = false;
+ dbinfo[i].made_subscription = false;
+ /* other struct fields will be filled later. */
+
+ /* Subscriber. */
+ conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
+ dbinfo[i].subconninfo = conninfo;
+
+ i++;
+ }
+
+ return dbinfo;
+}
+
+static PGconn *
+connect_database(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ const char *rconninfo;
+
+ /* logical replication mode */
+ rconninfo = psprintf("%s replication=database", conninfo);
+
+ conn = PQconnectdb(rconninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ pg_log_error("connection to database failed: %s", PQerrorMessage(conn));
+ return NULL;
+ }
+
+ /* secure search_path */
+ res = PQexec(conn, ALWAYS_SECURE_SEARCH_PATH_SQL);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not clear search_path: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+ PQclear(res);
+
+ return conn;
+}
+
+static void
+disconnect_database(PGconn *conn)
+{
+ Assert(conn != NULL);
+
+ PQfinish(conn);
+}
+
+/*
+ * Obtain the system identifier using the provided connection. It will be used
+ * to compare if a data directory is a clone of another one.
+ */
+static uint64
+get_sysid_from_conn(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from publisher");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "IDENTIFY_SYSTEM");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not send replication command \"%s\": %s",
+ "IDENTIFY_SYSTEM", PQresultErrorMessage(res));
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+ if (PQntuples(res) != 1 || PQnfields(res) < 3)
+ {
+ pg_log_error("could not identify system: got %d rows and %d fields, expected %d rows and %d or more fields",
+ PQntuples(res), PQnfields(res), 1, 3);
+
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+
+ sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+
+ pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+
+ disconnect_database(conn);
+
+ return sysid;
+}
+
+/*
+ * Obtain the system identifier from control file. It will be used to compare
+ * if a data directory is a clone of another one. This routine is used locally
+ * and avoids a replication connection.
+ */
+static uint64
+get_control_from_datadir(const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ sysid = cf->system_identifier;
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+
+ pfree(cf);
+
+ return sysid;
+}
+
+/*
+ * Modify the system identifier. Since a standby server preserves the system
+ * identifier, it makes sense to change it to avoid situations in which WAL
+ * files from one of the systems might be used in the other one.
+ */
+static void
+modify_sysid(const char *pg_resetwal_path, const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ struct timeval tv;
+
+ char *cmd_str;
+ int rc;
+
+ pg_log_info("modifying system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ /*
+ * Select a new system identifier.
+ *
+ * XXX this code was extracted from BootStrapXLOG().
+ */
+ gettimeofday(&tv, NULL);
+ cf->system_identifier = ((uint64) tv.tv_sec) << 32;
+ cf->system_identifier |= ((uint64) tv.tv_usec) << 12;
+ cf->system_identifier |= getpid() & 0xFFF;
+
+ if (!dry_run)
+ update_controlfile(datadir, cf, true);
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) cf->system_identifier);
+
+ pg_log_info("running pg_resetwal on the subscriber");
+
+ cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+
+ pg_log_debug("command is: %s", cmd_str);
+
+ if (!dry_run)
+ {
+ rc = system(cmd_str);
+ if (rc == 0)
+ pg_log_info("subscriber successfully changed the system identifier");
+ else
+ pg_log_error("subscriber failed to change system identifier: exit code: %d", rc);
+ }
+
+ pfree(cf);
+}
+
+/*
+ * Return a palloc'd slot name if the replication is using one.
+ */
+static char *
+use_primary_slot_name(void)
+{
+ PGconn *conn;
+ PGresult *res;
+ PQExpBuffer str = createPQExpBuffer();
+ char *slot_name;
+
+ conn = connect_database(dbinfo[0].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "SELECT setting FROM pg_settings WHERE name = 'primary_slot_name'");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain parameter information: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+
+ /*
+ * If primary_slot_name is an empty string, the current replication
+ * connection is not using a replication slot, bail out.
+ */
+ if (strcmp(PQgetvalue(res, 0, 0), "") == 0)
+ {
+ PQclear(res);
+ return NULL;
+ }
+
+ slot_name = pg_strdup(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ disconnect_database(conn);
+
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ appendPQExpBuffer(str,
+ "SELECT 1 FROM pg_replication_slots r INNER JOIN pg_stat_activity a ON (r.active_pid = a.pid) WHERE slot_name = '%s'", slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain replication slot information: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
+ PQntuples(res), 1);
+ return NULL;
+ }
+
+ PQclear(res);
+ disconnect_database(conn);
+
+ return slot_name;
+}
+
+static bool
+create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
+{
+ int i;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ PGconn *conn;
+ PGresult *res;
+ char replslotname[NAMEDATALEN];
+
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT oid FROM pg_catalog.pg_database WHERE datname = current_database()");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain database OID: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain database OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ return false;
+ }
+
+ /* Remember database OID. */
+ dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+
+ PQclear(res);
+
+ /*
+ * Build the replication slot name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 36
+ * characters (14 + 10 + 1 + 10 + '\0'). System identifier is included
+ * to reduce the probability of collision. By default, subscription
+ * name is used as replication slot name.
+ */
+ snprintf(replslotname, sizeof(replslotname),
+ "pg_subscriber_%u_%d",
+ dbinfo[i].oid,
+ (int) getpid());
+ dbinfo[i].subname = pg_strdup(replslotname);
+
+ /* Create replication slot on publisher. */
+ if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
+ pg_log_info("create replication slot \"%s\" on publisher", replslotname);
+ else
+ return false;
+
+ disconnect_database(conn);
+ }
+
+ return true;
+}
+
+/*
+ * Create a logical replication slot and returns a consistent LSN. The returned
+ * LSN might be used to catch up the subscriber up to the required point.
+ *
+ * CreateReplicationSlot() is not used because it does not provide the one-row
+ * result set that contains the consistent LSN.
+ */
+static char *
+create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res = NULL;
+ char *lsn = NULL;
+ bool transient_replslot = false;
+
+ Assert(conn != NULL);
+
+ /*
+ * If no slot name is informed, it is a transient replication slot used
+ * only for catch up purposes.
+ */
+ if (slot_name[0] == '\0')
+ {
+ snprintf(slot_name, NAMEDATALEN, "pg_subscriber_%d_startpoint",
+ (int) getpid());
+ transient_replslot = true;
+ }
+
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
+ appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQresultErrorMessage(res));
+ return lsn;
+ }
+ }
+
+ /* for cleanup purposes */
+ if (transient_replslot)
+ made_transient_replslot = true;
+ else
+ dbinfo->made_replslot = true;
+
+ if (!dry_run)
+ {
+ lsn = pg_strdup(PQgetvalue(res, 0, 1));
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+
+ return lsn;
+}
+
+static void
+drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Reports a suitable message if pg_ctl fails.
+ */
+static void
+pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
+{
+ if (rc != 0)
+ {
+ if (WIFEXITED(rc))
+ {
+ pg_log_error("pg_ctl failed with exit code %d", WEXITSTATUS(rc));
+ }
+ else if (WIFSIGNALED(rc))
+ {
+#if defined(WIN32)
+ pg_log_error("pg_ctl was terminated by exception 0x%X", WTERMSIG(rc));
+ pg_log_error_detail("See C include file \"ntstatus.h\" for a description of the hexadecimal value.");
+#else
+ pg_log_error("pg_ctl was terminated by signal %d: %s",
+ WTERMSIG(rc), pg_strsignal(WTERMSIG(rc)));
+#endif
+ }
+ else
+ {
+ pg_log_error("pg_ctl exited with unrecognized status %d", rc);
+ }
+
+ pg_log_error_detail("The failed command was: %s", pg_ctl_cmd);
+ exit(1);
+ }
+
+ if (action)
+ pg_log_info("postmaster was started");
+ else
+ pg_log_info("postmaster was stopped");
+}
+
+/*
+ * Returns after the server finishes the recovery process.
+ */
+static void
+wait_for_end_recovery(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ int status = POSTMASTER_STILL_STARTING;
+
+ pg_log_info("waiting the postmaster to reach the consistent state");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ for (;;)
+ {
+ bool in_recovery;
+
+ res = PQexec(conn, "SELECT pg_catalog.pg_is_in_recovery()");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain recovery progress");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("unexpected result from pg_is_in_recovery function");
+ exit(1);
+ }
+
+ in_recovery = (strcmp(PQgetvalue(res, 0, 0), "t") == 0);
+
+ PQclear(res);
+
+ /*
+ * Does the recovery process finish? In dry run mode, there is no
+ * recovery mode. Bail out as the recovery process has ended.
+ */
+ if (!in_recovery || dry_run)
+ {
+ status = POSTMASTER_READY;
+ break;
+ }
+
+ /* Keep waiting. */
+ pg_usleep(WAIT_INTERVAL * USEC_PER_SEC);
+ }
+
+ disconnect_database(conn);
+
+ if (status == POSTMASTER_STILL_STARTING)
+ {
+ pg_log_error("server did not end recovery");
+ exit(1);
+ }
+
+ pg_log_info("postmaster reached the consistent state");
+}
+
+/*
+ * Create a publication that includes all tables in the database.
+ */
+static void
+create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ /* Check if the publication needs to be created. */
+ appendPQExpBuffer(str,
+ "SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
+ dbinfo->pubname);
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publication information: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) == 1)
+ {
+ /*
+ * If publication name already exists and puballtables is true, let's
+ * use it. A previous run of pg_subscriber must have created this
+ * publication. Bail out.
+ */
+ if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
+ {
+ pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ return;
+ }
+ else
+ {
+ /*
+ * Unfortunately, if it reaches this code path, it will always
+ * fail (unless you decide to change the existing publication
+ * name). That's bad but it is very unlikely that the user will
+ * choose a name with pg_subscriber_ prefix followed by the exact
+ * database oid in which puballtables is false.
+ */
+ pg_log_error("publication \"%s\" does not replicate changes for all tables",
+ dbinfo->pubname);
+ pg_log_error_hint("Consider renaming this publication.");
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(str);
+
+ pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
+ dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_publication = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove publication if it couldn't finish all steps.
+ */
+static void
+drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Create a subscription with some predefined options.
+ *
+ * A replication slot was already created in a previous step. Let's use it. By
+ * default, the subscription name is used as replication slot name. It is
+ * not required to copy data. The subscription will be created but it will not
+ * be enabled now. That's because the replication progress must be set and the
+ * replication origin name (one of the function arguments) contains the
+ * subscription OID in its name. Once the subscription is created,
+ * set_replication_progress() can obtain the chosen origin name and set up its
+ * initial location.
+ */
+static void
+create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str,
+ "CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
+ "WITH (create_slot = false, copy_data = false, enabled = false)",
+ dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
+ dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_subscription = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove subscription if it couldn't finish all steps.
+ */
+static void
+drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Sets the replication progress to the consistent LSN.
+ *
+ * The subscriber caught up to the consistent LSN provided by the temporary
+ * replication slot. The goal is to set up the initial location for the logical
+ * replication that is the exact LSN that the subscriber was promoted. Once the
+ * subscription is enabled it will start streaming from that location onwards.
+ * In dry run mode, the subscription OID and LSN are set to invalid values for
+ * printing purposes.
+ */
+static void
+set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+ Oid suboid;
+ char originname[NAMEDATALEN];
+ char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+
+ Assert(conn != NULL);
+
+ appendPQExpBuffer(str,
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscription OID: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1 && !dry_run)
+ {
+ pg_log_error("could not obtain subscription OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (dry_run)
+ {
+ suboid = InvalidOid;
+ snprintf(lsnstr, sizeof(lsnstr), "%X/%X", LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ suboid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ snprintf(lsnstr, sizeof(lsnstr), "%s", lsn);
+ }
+
+ /*
+ * The origin name is defined as pg_%u. %u is the subscription OID. See
+ * ApplyWorkerMain().
+ */
+ snprintf(originname, sizeof(originname), "pg_%u", suboid);
+
+ PQclear(res);
+
+ pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
+ originname, lsnstr, dbinfo->dbname);
+
+ resetPQExpBuffer(str);
+ appendPQExpBuffer(str,
+ "SELECT pg_catalog.pg_replication_origin_advance('%s', '%s')", originname, lsnstr);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not set replication progress for the subscription \"%s\": %s",
+ dbinfo->subname, PQresultErrorMessage(res));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Enables the subscription.
+ *
+ * The subscription was created in a previous step but it was disabled. After
+ * adjusting the initial location, enabling the subscription is the last step
+ * of this setup.
+ */
+static void
+enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+int
+main(int argc, char **argv)
+{
+ static struct option long_options[] =
+ {
+ {"help", no_argument, NULL, '?'},
+ {"version", no_argument, NULL, 'V'},
+ {"pgdata", required_argument, NULL, 'D'},
+ {"publisher-conninfo", required_argument, NULL, 'P'},
+ {"subscriber-conninfo", required_argument, NULL, 'S'},
+ {"database", required_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"verbose", no_argument, NULL, 'v'},
+ {NULL, 0, NULL, 0}
+ };
+
+ int c;
+ int option_index;
+ int rc;
+
+ char *pg_ctl_cmd;
+
+ char *base_dir;
+ char *server_start_log;
+
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+
+ char *pub_base_conninfo = NULL;
+ char *sub_base_conninfo = NULL;
+ char *dbname_conninfo = NULL;
+
+ uint64 pub_sysid;
+ uint64 sub_sysid;
+ struct stat statbuf;
+
+ PGconn *conn;
+ char *consistent_lsn;
+
+ PQExpBuffer recoveryconfcontents = NULL;
+
+ char pidfile[MAXPGPATH];
+
+ int i;
+
+ pg_logging_init(argv[0]);
+ pg_logging_set_level(PG_LOG_WARNING);
+ progname = get_progname(argv[0]);
+ set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_subscriber"));
+
+ if (argc > 1)
+ {
+ if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+ {
+ usage();
+ exit(0);
+ }
+ else if (strcmp(argv[1], "-V") == 0
+ || strcmp(argv[1], "--version") == 0)
+ {
+ puts("pg_subscriber (PostgreSQL) " PG_VERSION);
+ exit(0);
+ }
+ }
+
+ atexit(cleanup_objects_atexit);
+
+ /*
+ * Don't allow it to be run as root. It uses pg_ctl which does not allow
+ * it either.
+ */
+#ifndef WIN32
+ if (geteuid() == 0)
+ {
+ pg_log_error("cannot be executed by \"root\"");
+ pg_log_error_hint("You must run %s as the PostgreSQL superuser.",
+ progname);
+ exit(1);
+ }
+#endif
+
+ while ((c = getopt_long(argc, argv, "D:P:S:d:nv",
+ long_options, &option_index)) != -1)
+ {
+ switch (c)
+ {
+ case 'D':
+ subscriber_dir = pg_strdup(optarg);
+ break;
+ case 'P':
+ pub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'S':
+ sub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'd':
+ /* Ignore duplicated database names. */
+ if (!simple_string_list_member(&database_names, optarg))
+ {
+ simple_string_list_append(&database_names, optarg);
+ num_dbs++;
+ }
+ break;
+ case 'n':
+ dry_run = true;
+ break;
+ case 'v':
+ pg_logging_increase_verbosity();
+ break;
+ default:
+ /* getopt_long already emitted a complaint */
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Any non-option arguments?
+ */
+ if (optind < argc)
+ {
+ pg_log_error("too many command-line arguments (first is \"%s\")",
+ argv[optind]);
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Required arguments
+ */
+ if (subscriber_dir == NULL)
+ {
+ pg_log_error("no subscriber data directory specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Parse connection string. Build a base connection string that might be
+ * reused by multiple databases.
+ */
+ if (pub_conninfo_str == NULL)
+ {
+ /*
+ * TODO use primary_conninfo (if available) from subscriber and
+ * extract publisher connection string. Assume that there are
+ * identical entries for physical and logical replication. If there is
+ * not, we would fail anyway.
+ */
+ pg_log_error("no publisher connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
+ "publisher");
+ if (pub_base_conninfo == NULL)
+ exit(1);
+
+ if (sub_conninfo_str == NULL)
+ {
+ pg_log_error("no subscriber connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
+ if (sub_base_conninfo == NULL)
+ exit(1);
+
+ if (database_names.head == NULL)
+ {
+ pg_log_info("no database was specified");
+
+ /*
+ * If --database option is not provided, try to obtain the dbname from
+ * the publisher conninfo. If dbname parameter is not available, error
+ * out.
+ */
+ if (dbname_conninfo)
+ {
+ simple_string_list_append(&database_names, dbname_conninfo);
+ num_dbs++;
+
+ pg_log_info("database \"%s\" was extracted from the publisher connection string",
+ dbname_conninfo);
+ }
+ else
+ {
+ pg_log_error("no database name specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
+ */
+ if (!get_exec_path(argv[0]))
+ exit(1);
+
+ /* rudimentary check for a data directory. */
+ if (!check_data_directory(subscriber_dir))
+ exit(1);
+
+ /* Store database information for publisher and subscriber. */
+ dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+
+ /*
+ * Check if the subscriber data directory has the same system identifier
+ * than the publisher data directory.
+ */
+ pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
+ sub_sysid = get_control_from_datadir(subscriber_dir);
+ if (pub_sysid != sub_sysid)
+ {
+ pg_log_error("subscriber data directory is not a copy of the source database cluster");
+ exit(1);
+ }
+
+ /*
+ * Create the output directory to store any data generated by this tool.
+ */
+ base_dir = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("directory path for subscriber is too long");
+ exit(1);
+ }
+
+ if (mkdir(base_dir, pg_dir_create_mode) < 0 && errno != EEXIST)
+ {
+ pg_log_error("could not create directory \"%s\": %m", base_dir);
+ exit(1);
+ }
+
+ /* subscriber PID file. */
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+
+ /*
+ * Stop the subscriber if it is a standby server. Before executing the
+ * transformation steps, make sure the subscriber is not running because
+ * one of the steps is to modify some recovery parameters that require a
+ * restart.
+ */
+ if (stat(pidfile, &statbuf) == 0)
+ {
+ /*
+ * Since the standby server is running, check if it is using an
+ * existing replication slot for WAL retention purposes. This
+ * replication slot has no use after the transformation, hence, it
+ * will be removed at the end of this process.
+ */
+ primary_slot_name = use_primary_slot_name();
+ if (primary_slot_name != NULL)
+ pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+
+ pg_log_info("subscriber is up and running");
+ pg_log_info("stopping the server to start the transformation steps");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+ }
+
+ /*
+ * Create a replication slot for each database on the publisher.
+ */
+ if (!create_all_logical_replication_slots(dbinfo))
+ exit(1);
+
+ /*
+ * Create a logical replication slot to get a consistent LSN.
+ *
+ * This consistent LSN will be used later to advanced the recently created
+ * replication slots. We cannot use the last created replication slot
+ * because the consistent LSN should be obtained *after* the base backup
+ * finishes (and the base backup should include the logical replication
+ * slots).
+ *
+ * XXX we should probably use the last created replication slot to get a
+ * consistent LSN but it should be changed after adding pg_basebackup
+ * support.
+ *
+ * A temporary replication slot is not used here to avoid keeping a
+ * replication connection open (depending when base backup was taken, the
+ * connection should be open for a few hours).
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+ consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
+ temp_replslot);
+
+ /*
+ * Write recovery parameters.
+ *
+ * Despite of the recovery parameters will be written to the subscriber,
+ * use a publisher connection for the follwing recovery functions. The
+ * connection is only used to check the current server version (physical
+ * replica, same server version). The subscriber is not running yet. In
+ * dry run mode, the recovery parameters *won't* be written. An invalid
+ * LSN is used for printing purposes.
+ */
+ recoveryconfcontents = GenerateRecoveryConfig(conn, NULL);
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_inclusive = true\n");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_action = promote\n");
+
+ if (dry_run)
+ {
+ appendPQExpBuffer(recoveryconfcontents, "# dry run mode");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%X/%X'\n",
+ LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
+ consistent_lsn);
+ WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ }
+ disconnect_database(conn);
+
+ pg_log_debug("recovery parameters:\n%s", recoveryconfcontents->data);
+
+ /*
+ * Start subscriber and wait until accepting connections.
+ */
+ pg_log_info("starting the subscriber");
+
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ server_start_log = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(server_start_log, MAXPGPATH, "%s/%s/server_start_%s.log", subscriber_dir, PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+
+ pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, subscriber_dir, server_start_log);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+
+ /*
+ * Waiting the subscriber to be promoted.
+ */
+ wait_for_end_recovery(dbinfo[0].subconninfo);
+
+ /*
+ * Create a publication for each database. This step should be executed
+ * after promoting the subscriber to avoid replicating unnecessary
+ * objects.
+ */
+ for (i = 0; i < num_dbs; i++)
+ {
+ char pubname[NAMEDATALEN];
+
+ /* Connect to publisher. */
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ /*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 35 characters (14 + 10 +
+ * '\0').
+ */
+ snprintf(pubname, sizeof(pubname), "pg_subscriber_%u", dbinfo[i].oid);
+ dbinfo[i].pubname = pg_strdup(pubname);
+
+ create_publication(conn, &dbinfo[i]);
+
+ disconnect_database(conn);
+ }
+
+ /*
+ * Create a subscription for each database.
+ */
+ for (i = 0; i < num_dbs; i++)
+ {
+ /* Connect to subscriber. */
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ create_subscription(conn, &dbinfo[i]);
+
+ /* Set the replication progress to the correct LSN. */
+ set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+
+ /* Enable subscription. */
+ enable_subscription(conn, &dbinfo[i]);
+
+ disconnect_database(conn);
+ }
+
+ /*
+ * The transient replication slot is no longer required. Drop it.
+ *
+ * If the physical replication slot exists, drop it.
+ *
+ * XXX we might not fail here. Instead, we provide a warning so the user
+ * eventually drops the replication slot later.
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ {
+ pg_log_warning("could not drop transient replication slot \"%s\" on publisher", temp_replslot);
+ pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
+ if (primary_slot_name != NULL)
+ pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
+ }
+ else
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ if (primary_slot_name != NULL)
+ drop_replication_slot(conn, &dbinfo[0], primary_slot_name);
+ disconnect_database(conn);
+ }
+
+ /*
+ * Stop the subscriber.
+ */
+ pg_log_info("stopping the subscriber");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+ /*
+ * Change system identifier.
+ */
+ modify_sysid(pg_resetwal_path, subscriber_dir);
+
+ /*
+ * Remove log file generated by this tool, if it runs successfully.
+ * Otherwise, file is kept that may provide useful debugging information.
+ */
+ unlink(server_start_log);
+
+ success = true;
+
+ pg_log_info("Done!");
+
+ return 0;
+}
diff --git a/src/bin/pg_basebackup/t/040_pg_subscriber.pl b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
new file mode 100644
index 0000000000..4ebff76b2d
--- /dev/null
+++ b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
@@ -0,0 +1,44 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test checking options of pg_subscriber.
+#
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+program_help_ok('pg_subscriber');
+program_version_ok('pg_subscriber');
+program_options_handling_ok('pg_subscriber');
+
+my $datadir = PostgreSQL::Test::Utils::tempdir;
+
+command_fails(['pg_subscriber'],
+ 'no subscriber data directory specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--pgdata', $datadir
+ ],
+ 'no publisher connection string specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--dry-run',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres'
+ ],
+ 'no subscriber connection string specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--verbose',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres',
+ '--subscriber-conninfo', 'dbname=postgres'
+ ],
+ 'no database name specified');
+
+done_testing();
diff --git a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
new file mode 100644
index 0000000000..fbcd0fc82b
--- /dev/null
+++ b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
@@ -0,0 +1,139 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test using a standby server as the subscriber.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node_p;
+my $node_f;
+my $node_s;
+my $result;
+
+# Set up node P as primary
+$node_p = PostgreSQL::Test::Cluster->new('node_p');
+$node_p->init(allows_streaming => 'logical');
+$node_p->start;
+
+# Set up node F as about-to-fail node
+# The extra option forces it to initialize a new cluster instead of copying a
+# previously initdb's cluster.
+$node_f = PostgreSQL::Test::Cluster->new('node_f');
+$node_f->init(allows_streaming => 'logical', extra => [ '--no-instructions' ]);
+$node_f->start;
+
+# On node P
+# - create databases
+# - create test tables
+# - insert a row
+$node_p->safe_psql(
+ 'postgres', q(
+ CREATE DATABASE pg1;
+ CREATE DATABASE pg2;
+));
+$node_p->safe_psql('pg1', 'CREATE TABLE tbl1 (a text)');
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('first row')");
+$node_p->safe_psql('pg2', 'CREATE TABLE tbl2 (a text)');
+
+# Set up node S as standby linking to node P
+$node_p->backup('backup_1');
+$node_s = PostgreSQL::Test::Cluster->new('node_s');
+$node_s->init_from_backup($node_p, 'backup_1', has_streaming => 1);
+$node_s->append_conf('postgresql.conf', 'log_min_messages = debug2');
+$node_s->set_standby_mode();
+$node_s->start;
+
+# Insert another row on node P and wait node S to catch up
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('second row')");
+$node_p->wait_for_replay_catchup($node_s);
+
+# Run pg_subscriber on about-to-fail node F
+command_fails(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $node_f->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_f->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'subscriber data directory is not a copy of the source database cluster');
+
+# dry run mode on node S
+command_ok(
+ [
+ 'pg_subscriber', '--verbose', '--dry-run',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_subscriber --dry-run on node S');
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+# Check if node S is still a standby
+is($node_s->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
+ 't', 'standby is in recovery');
+
+# Run pg_subscriber on node S
+command_ok(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_subscriber on node S');
+
+# Insert rows on P
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('third row')");
+$node_p->safe_psql('pg2', "INSERT INTO tbl2 VALUES('row 1')");
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+
+# Get subscription names
+$result = $node_s->safe_psql(
+ 'postgres', qq(
+ SELECT subname FROM pg_subscription WHERE subname ~ '^pg_subscriber_'
+));
+my @subnames = split("\n", $result);
+
+# Wait subscriber to catch up
+$node_s->wait_for_subscription_sync($node_p, $subnames[0]);
+$node_s->wait_for_subscription_sync($node_p, $subnames[1]);
+
+# Check result on database pg1
+$result = $node_s->safe_psql('pg1', 'SELECT * FROM tbl1');
+is( $result, qq(first row
+second row
+third row),
+ 'logical replication works on database pg1');
+
+# Check result on database pg2
+$result = $node_s->safe_psql('pg2', 'SELECT * FROM tbl2');
+is( $result, qq(row 1),
+ 'logical replication works on database pg2');
+
+# Different system identifier?
+my $sysid_p = $node_p->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+my $sysid_s = $node_s->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+ok($sysid_p != $sysid_s, 'system identifier was changed');
+
+# clean up
+$node_p->teardown_node;
+$node_s->teardown_node;
+
+done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 77+ messages in thread
* RE: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
@ 2024-01-23 02:11 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-24 06:11 ` Re: speed up a logical replica setup Shubham Khanna <[email protected]>
1 sibling, 1 reply; 77+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-23 02:11 UTC (permalink / raw)
To: 'Shlok Kyal' <[email protected]>; [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Euler Taveira <[email protected]>
Dear hackers,
> We fixed some of the comments posted in the thread. We have created it
> as top-up patch 0002 and 0003.
I found that the CFbot raised an ERROR. Also, it may not work well in case
of production build.
PSA the fixed patch set.
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
[application/octet-stream] v7-0001-Creates-a-new-logical-replica-from-a-standby-serv.patch (67.0K, ../../TY3PR01MB98890AB22AECA2F503EB7858F5742@TY3PR01MB9889.jpnprd01.prod.outlook.com/2-v7-0001-Creates-a-new-logical-replica-from-a-standby-serv.patch)
download | inline diff:
From 773b6d187892a7e0aea0edf1547b86ad2b2c8e2f Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email protected]>
Date: Mon, 5 Jun 2023 14:39:40 -0400
Subject: [PATCH v7 1/3] Creates a new logical replica from a standby server
A new tool called pg_subscriber can convert a physical replica into a
logical replica. It runs on the target server and should be able to
connect to the source server (publisher) and the target server
(subscriber).
The conversion requires a few steps. Check if the target data directory
has the same system identifier than the source data directory. Stop the
target server if it is running as a standby server. Create one
replication slot per specified database on the source server. One
additional replication slot is created at the end to get the consistent
LSN (This consistent LSN will be used as (a) a stopping point for the
recovery process and (b) a starting point for the subscriptions). Write
recovery parameters into the target data directory and start the target
server (Wait until the target server is promoted). Create one
publication (FOR ALL TABLES) per specified database on the source
server. Create one subscription per specified database on the target
server (Use replication slot and publication created in a previous step.
Don't enable the subscriptions yet). Sets the replication progress to
the consistent LSN that was got in a previous step. Enable the
subscription for each specified database on the target server. Remove
the additional replication slot that was used to get the consistent LSN.
Stop the target server. Change the system identifier from the target
server.
Depending on your workload and database size, creating a logical replica
couldn't be an option due to resource constraints (WAL backlog should be
available until all table data is synchronized). The initial data copy
and the replication progress tends to be faster on a physical replica.
The purpose of this tool is to speed up a logical replica setup.
---
doc/src/sgml/ref/allfiles.sgml | 1 +
doc/src/sgml/ref/pg_subscriber.sgml | 284 +++
doc/src/sgml/reference.sgml | 1 +
src/bin/pg_basebackup/.gitignore | 1 +
src/bin/pg_basebackup/Makefile | 8 +-
src/bin/pg_basebackup/meson.build | 19 +
src/bin/pg_basebackup/pg_subscriber.c | 1657 +++++++++++++++++
src/bin/pg_basebackup/t/040_pg_subscriber.pl | 44 +
.../t/041_pg_subscriber_standby.pl | 139 ++
9 files changed, 2153 insertions(+), 1 deletion(-)
create mode 100644 doc/src/sgml/ref/pg_subscriber.sgml
create mode 100644 src/bin/pg_basebackup/pg_subscriber.c
create mode 100644 src/bin/pg_basebackup/t/040_pg_subscriber.pl
create mode 100644 src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index 4a42999b18..3862c976d7 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -214,6 +214,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY pgResetwal SYSTEM "pg_resetwal.sgml">
<!ENTITY pgRestore SYSTEM "pg_restore.sgml">
<!ENTITY pgRewind SYSTEM "pg_rewind.sgml">
+<!ENTITY pgSubscriber SYSTEM "pg_subscriber.sgml">
<!ENTITY pgVerifyBackup SYSTEM "pg_verifybackup.sgml">
<!ENTITY pgtestfsync SYSTEM "pgtestfsync.sgml">
<!ENTITY pgtesttiming SYSTEM "pgtesttiming.sgml">
diff --git a/doc/src/sgml/ref/pg_subscriber.sgml b/doc/src/sgml/ref/pg_subscriber.sgml
new file mode 100644
index 0000000000..553185c35f
--- /dev/null
+++ b/doc/src/sgml/ref/pg_subscriber.sgml
@@ -0,0 +1,284 @@
+<!--
+doc/src/sgml/ref/pg_subscriber.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="app-pgsubscriber">
+ <indexterm zone="app-pgsubscriber">
+ <primary>pg_subscriber</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle><application>pg_subscriber</application></refentrytitle>
+ <manvolnum>1</manvolnum>
+ <refmiscinfo>Application</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>pg_subscriber</refname>
+ <refpurpose>create a new logical replica from a standby server</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_subscriber</command>
+ <arg rep="repeat"><replaceable>option</replaceable></arg>
+ </cmdsynopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+ <para>
+ <application>pg_subscriber</application> takes the publisher and subscriber
+ connection strings, a cluster directory from a standby server and a list of
+ database names and it sets up a new logical replica using the physical
+ recovery process.
+ </para>
+
+ <para>
+ The <application>pg_subscriber</application> should be run at the target
+ server. The source server (known as publisher server) should accept logical
+ replication connections from the target server (known as subscriber server).
+ The target server should accept local logical replication connection.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Options</title>
+
+ <para>
+ <application>pg_subscriber</application> accepts the following
+ command-line arguments:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-D <replaceable class="parameter">directory</replaceable></option></term>
+ <term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term>
+ <listitem>
+ <para>
+ The target directory that contains a cluster directory from a standby
+ server.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-P <replaceable class="parameter">conninfo</replaceable></option></term>
+ <term><option>--publisher-conninfo=<replaceable class="parameter">conninfo</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the publisher. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-S <replaceable class="parameter">conninfo</replaceable></option></term>
+ <term><option>--subscriber-conninfo=<replaceable class="parameter">conninfo</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the subscriber. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-d <replaceable class="parameter">dbname</replaceable></option></term>
+ <term><option>--database=<replaceable class="parameter">dbname</replaceable></option></term>
+ <listitem>
+ <para>
+ The database name to create the subscription. Multiple databases can be
+ selected by writing multiple <option>-d</option> switches.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
+ <listitem>
+ <para>
+ Do everything except actually modifying the target directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-v</option></term>
+ <term><option>--verbose</option></term>
+ <listitem>
+ <para>
+ Enables verbose mode. This will cause
+ <application>pg_subscriber</application> to output progress messages
+ and detailed information about each step.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ Other options are also available:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-V</option></term>
+ <term><option>--version</option></term>
+ <listitem>
+ <para>
+ Print the <application>pg_subscriber</application> version and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-?</option></term>
+ <term><option>--help</option></term>
+ <listitem>
+ <para>
+ Show help about <application>pg_subscriber</application> command
+ line arguments, and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Notes</title>
+
+ <para>
+ The transformation proceeds in the following steps:
+ </para>
+
+ <procedure>
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the given target data
+ directory has the same system identifier than the source data directory.
+ Since it uses the recovery process as one of the steps, it starts the
+ target server as a replica from the source server. If the system
+ identifier is not the same, <application>pg_subscriber</application> will
+ terminate with an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the target data
+ directory is used by a standby server. Stop the standby server if it is
+ running. One of the next steps is to add some recovery parameters that
+ requires a server start. This step avoids an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one replication slot for
+ each specified database on the source server. The replication slot name
+ contains a <literal>pg_subscriber</literal> prefix. These replication
+ slots will be used by the subscriptions in a future step. Another
+ replication slot is used to get a consistent start location. This
+ consistent LSN will be used as a stopping point in the <xref
+ linkend="guc-recovery-target-lsn"/> parameter and by the
+ subscriptions as a replication starting point. It guarantees that no
+ transaction will be lost.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> writes recovery parameters into
+ the target data directory and start the target server. It specifies a LSN
+ (consistent LSN that was obtained in the previous step) of write-ahead
+ log location up to which recovery will proceed. It also specifies
+ <literal>promote</literal> as the action that the server should take once
+ the recovery target is reached. This step finishes once the server ends
+ standby mode and is accepting read-write operations.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Next, <application>pg_subscriber</application> creates one publication
+ for each specified database on the source server. Each publication
+ replicates changes for all tables in the database. The publication name
+ contains a <literal>pg_subscriber</literal> prefix. These publication
+ will be used by a corresponding subscription in a next step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one subscription for
+ each specified database on the target server. Each subscription name
+ contains a <literal>pg_subscriber</literal> prefix. The replication slot
+ name is identical to the subscription name. It does not copy existing data
+ from the source server. It does not create a replication slot. Instead, it
+ uses the replication slot that was created in a previous step. The
+ subscription is created but it is not enabled yet. The reason is the
+ replication progress must be set to the consistent LSN but replication
+ origin name contains the subscription oid in its name. Hence, the
+ subscription will be enabled in a separate step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> sets the replication progress to
+ the consistent LSN that was obtained in a previous step. When the target
+ server started the recovery process, it caught up to the consistent LSN.
+ This is the exact LSN to be used as a initial location for each
+ subscription.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Finally, <application>pg_subscriber</application> enables the subscription
+ for each specified database on the target server. The subscription starts
+ streaming from the consistent LSN.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> removes the additional replication
+ slot that was used to get the consistent LSN on the source server.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> stops the target server to change
+ its system identifier.
+ </para>
+ </step>
+ </procedure>
+ </refsect1>
+
+ <refsect1>
+ <title>Examples</title>
+
+ <para>
+ To create a logical replica for databases <literal>hr</literal> and
+ <literal>finance</literal> from a standby server at <literal>foo</literal>:
+<screen>
+<prompt>$</prompt> <userinput>pg_subscriber -D /usr/local/pgsql/data -P "host=foo" -S "host=localhost" -d hr -d finance</userinput>
+</screen>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="app-pgbasebackup"/></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index aa94f6adf6..266f4e515a 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -285,6 +285,7 @@
&pgCtl;
&pgResetwal;
&pgRewind;
+ &pgSubscriber;
&pgtestfsync;
&pgtesttiming;
&pgupgrade;
diff --git a/src/bin/pg_basebackup/.gitignore b/src/bin/pg_basebackup/.gitignore
index 26048bdbd8..0e5384a1d5 100644
--- a/src/bin/pg_basebackup/.gitignore
+++ b/src/bin/pg_basebackup/.gitignore
@@ -1,5 +1,6 @@
/pg_basebackup
/pg_receivewal
/pg_recvlogical
+/pg_subscriber
/tmp_check/
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index abfb6440ec..f6281b7676 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -44,7 +44,7 @@ BBOBJS = \
bbstreamer_tar.o \
bbstreamer_zstd.o
-all: pg_basebackup pg_receivewal pg_recvlogical
+all: pg_basebackup pg_receivewal pg_recvlogical pg_subscriber
pg_basebackup: $(BBOBJS) $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) $(BBOBJS) $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
@@ -55,10 +55,14 @@ pg_receivewal: pg_receivewal.o $(OBJS) | submake-libpq submake-libpgport submake
pg_recvlogical: pg_recvlogical.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_recvlogical.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_subscriber: $(WIN32RES) pg_subscriber.o | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
install: all installdirs
$(INSTALL_PROGRAM) pg_basebackup$(X) '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
$(INSTALL_PROGRAM) pg_receivewal$(X) '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
$(INSTALL_PROGRAM) pg_recvlogical$(X) '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ $(INSTALL_PROGRAM) pg_subscriber$(X) '$(DESTDIR)$(bindir)/pg_subscriber$(X)'
installdirs:
$(MKDIR_P) '$(DESTDIR)$(bindir)'
@@ -67,10 +71,12 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ rm -f '$(DESTDIR)$(bindir)/pg_subscriber$(X)'
clean distclean:
rm -f pg_basebackup$(X) pg_receivewal$(X) pg_recvlogical$(X) \
$(BBOBJS) pg_receivewal.o pg_recvlogical.o \
+ pg_subscriber$(X) pg_subscriber.o \
$(OBJS)
rm -rf tmp_check
diff --git a/src/bin/pg_basebackup/meson.build b/src/bin/pg_basebackup/meson.build
index f7e60e6670..ccfd7bb7a5 100644
--- a/src/bin/pg_basebackup/meson.build
+++ b/src/bin/pg_basebackup/meson.build
@@ -75,6 +75,23 @@ pg_recvlogical = executable('pg_recvlogical',
)
bin_targets += pg_recvlogical
+pg_subscriber_sources = files(
+ 'pg_subscriber.c'
+)
+
+if host_system == 'windows'
+ pg_subscriber_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'pg_subscriber',
+ '--FILEDESC', 'pg_subscriber - create a new logical replica from a standby server',])
+endif
+
+pg_subscriber = executable('pg_subscriber',
+ pg_subscriber_sources,
+ dependencies: [frontend_code, libpq],
+ kwargs: default_bin_args,
+)
+bin_targets += pg_subscriber
+
tests += {
'name': 'pg_basebackup',
'sd': meson.current_source_dir(),
@@ -89,6 +106,8 @@ tests += {
't/011_in_place_tablespace.pl',
't/020_pg_receivewal.pl',
't/030_pg_recvlogical.pl',
+ 't/040_pg_subscriber.pl',
+ 't/041_pg_subscriber_standby.pl',
],
},
}
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
new file mode 100644
index 0000000000..e998c29f9e
--- /dev/null
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -0,0 +1,1657 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_subscriber.c
+ * Create a new logical replica from a standby server
+ *
+ * Copyright (C) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_subscriber/pg_subscriber.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include <signal.h>
+#include <sys/stat.h>
+#include <sys/time.h>
+#include <sys/wait.h>
+#include <time.h>
+
+#include "access/xlogdefs.h"
+#include "catalog/pg_control.h"
+#include "common/connect.h"
+#include "common/controldata_utils.h"
+#include "common/file_perm.h"
+#include "common/file_utils.h"
+#include "common/logging.h"
+#include "fe_utils/recovery_gen.h"
+#include "fe_utils/simple_list.h"
+#include "getopt_long.h"
+#include "utils/pidfile.h"
+
+#define PGS_OUTPUT_DIR "pg_subscriber_output.d"
+
+typedef struct LogicalRepInfo
+{
+ Oid oid; /* database OID */
+ char *dbname; /* database name */
+ char *pubconninfo; /* publication connection string for logical
+ * replication */
+ char *subconninfo; /* subscription connection string for logical
+ * replication */
+ char *pubname; /* publication name */
+ char *subname; /* subscription name (also replication slot
+ * name) */
+
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepInfo;
+
+static void cleanup_objects_atexit(void);
+static void usage();
+static char *get_base_conninfo(char *conninfo, char *dbname,
+ const char *noderole);
+static bool get_exec_path(const char *path);
+static bool check_data_directory(const char *datadir);
+static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
+static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
+static PGconn *connect_database(const char *conninfo);
+static void disconnect_database(PGconn *conn);
+static uint64 get_sysid_from_conn(const char *conninfo);
+static uint64 get_control_from_datadir(const char *datadir);
+static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
+static char *use_primary_slot_name(void);
+static bool create_all_logical_replication_slots(LogicalRepInfo *dbinfo);
+static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name);
+static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
+static void wait_for_end_recovery(const char *conninfo);
+static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+
+#define USEC_PER_SEC 1000000
+#define WAIT_INTERVAL 1 /* 1 second */
+
+/* Options */
+static const char *progname;
+
+static char *subscriber_dir = NULL;
+static char *pub_conninfo_str = NULL;
+static char *sub_conninfo_str = NULL;
+static SimpleStringList database_names = {NULL, NULL};
+static char *primary_slot_name = NULL;
+static bool dry_run = false;
+
+static bool success = false;
+
+static char *pg_ctl_path = NULL;
+static char *pg_resetwal_path = NULL;
+
+static LogicalRepInfo *dbinfo;
+static int num_dbs = 0;
+
+static char temp_replslot[NAMEDATALEN] = {0};
+static bool made_transient_replslot = false;
+
+enum WaitPMResult
+{
+ POSTMASTER_READY,
+ POSTMASTER_STANDBY,
+ POSTMASTER_STILL_STARTING,
+ POSTMASTER_FAILED
+};
+
+
+/*
+ * Cleanup objects that were created by pg_subscriber if there is an error.
+ *
+ * Replication slots, publications and subscriptions are created. Depending on
+ * the step it failed, it should remove the already created objects if it is
+ * possible (sometimes it won't work due to a connection issue).
+ */
+static void
+cleanup_objects_atexit(void)
+{
+ PGconn *conn;
+ int i;
+
+ if (success)
+ return;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ if (dbinfo[i].made_subscription)
+ {
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn != NULL)
+ {
+ drop_subscription(conn, &dbinfo[i]);
+ disconnect_database(conn);
+ }
+ }
+
+ if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ {
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn != NULL)
+ {
+ if (dbinfo[i].made_publication)
+ drop_publication(conn, &dbinfo[i]);
+ if (dbinfo[i].made_replslot)
+ drop_replication_slot(conn, &dbinfo[i], NULL);
+ disconnect_database(conn);
+ }
+ }
+ }
+
+ if (made_transient_replslot)
+ {
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn != NULL)
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ disconnect_database(conn);
+ }
+ }
+}
+
+static void
+usage(void)
+{
+ printf(_("%s creates a new logical replica from a standby server.\n\n"),
+ progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION]...\n"), progname);
+ printf(_("\nOptions:\n"));
+ printf(_(" -D, --pgdata=DATADIR location for the subscriber data directory\n"));
+ printf(_(" -P, --publisher-conninfo=CONNINFO publisher connection string\n"));
+ printf(_(" -S, --subscriber-conninfo=CONNINFO subscriber connection string\n"));
+ printf(_(" -d, --database=DBNAME database to create a subscription\n"));
+ printf(_(" -n, --dry-run stop before modifying anything\n"));
+ printf(_(" -v, --verbose output verbose messages\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
+ printf(_("\nReport bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+ printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
+
+/*
+ * Validate a connection string. Returns a base connection string that is a
+ * connection string without a database name plus a fallback application name.
+ * Since we might process multiple databases, each database name will be
+ * appended to this base connection string to provide a final connection string.
+ * If the second argument (dbname) is not null, returns dbname if the provided
+ * connection string contains it. If option --database is not provided, uses
+ * dbname as the only database to setup the logical replica.
+ * It is the caller's responsibility to free the returned connection string and
+ * dbname.
+ */
+static char *
+get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ PQconninfoOption *conn_opts = NULL;
+ PQconninfoOption *conn_opt;
+ char *errmsg = NULL;
+ char *ret;
+ int i;
+
+ pg_log_info("validating connection string on %s", noderole);
+
+ conn_opts = PQconninfoParse(conninfo, &errmsg);
+ if (conn_opts == NULL)
+ {
+ pg_log_error("could not parse connection string: %s", errmsg);
+ return NULL;
+ }
+
+ i = 0;
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (strcmp(conn_opt->keyword, "dbname") == 0 && conn_opt->val != NULL)
+ {
+ if (dbname)
+ dbname = pg_strdup(conn_opt->val);
+ continue;
+ }
+
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0')
+ {
+ if (i > 0)
+ appendPQExpBufferChar(buf, ' ');
+ appendPQExpBuffer(buf, "%s=%s", conn_opt->keyword, conn_opt->val);
+ i++;
+ }
+ }
+
+ if (i > 0)
+ appendPQExpBufferChar(buf, ' ');
+ appendPQExpBuffer(buf, "fallback_application_name=%s", progname);
+
+ ret = pg_strdup(buf->data);
+
+ destroyPQExpBuffer(buf);
+ PQconninfoFree(conn_opts);
+
+ return ret;
+}
+
+/*
+ * Get the absolute path from other PostgreSQL binaries (pg_ctl and
+ * pg_resetwal) that is used by it.
+ */
+static bool
+get_exec_path(const char *path)
+{
+ int rc;
+
+ pg_ctl_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_ctl",
+ "pg_ctl (PostgreSQL) " PG_VERSION "\n",
+ pg_ctl_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_ctl", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_ctl", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
+
+ pg_resetwal_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_resetwal",
+ "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
+ pg_resetwal_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_resetwal", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_resetwal", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+
+ return true;
+}
+
+/*
+ * Is it a cluster directory? These are preliminary checks. It is far from
+ * making an accurate check. If it is not a clone from the publisher, it will
+ * eventually fail in a future step.
+ */
+static bool
+check_data_directory(const char *datadir)
+{
+ struct stat statbuf;
+ char versionfile[MAXPGPATH];
+
+ pg_log_info("checking if directory \"%s\" is a cluster data directory",
+ datadir);
+
+ if (stat(datadir, &statbuf) != 0)
+ {
+ if (errno == ENOENT)
+ pg_log_error("data directory \"%s\" does not exist", datadir);
+ else
+ pg_log_error("could not access directory \"%s\": %s", datadir, strerror(errno));
+
+ return false;
+ }
+
+ snprintf(versionfile, MAXPGPATH, "%s/PG_VERSION", datadir);
+ if (stat(versionfile, &statbuf) != 0 && errno == ENOENT)
+ {
+ pg_log_error("directory \"%s\" is not a database cluster directory", datadir);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Append database name into a base connection string.
+ *
+ * dbname is the only parameter that changes so it is not included in the base
+ * connection string. This function concatenates dbname to build a "real"
+ * connection string.
+ */
+static char *
+concat_conninfo_dbname(const char *conninfo, const char *dbname)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *ret;
+
+ Assert(conninfo != NULL);
+
+ appendPQExpBufferStr(buf, conninfo);
+ appendPQExpBuffer(buf, " dbname=%s", dbname);
+
+ ret = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+
+ return ret;
+}
+
+/*
+ * Store publication and subscription information.
+ */
+static LogicalRepInfo *
+store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+{
+ LogicalRepInfo *dbinfo;
+ SimpleStringListCell *cell;
+ int i = 0;
+
+ dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+
+ for (cell = database_names.head; cell; cell = cell->next)
+ {
+ char *conninfo;
+
+ /* Publisher. */
+ conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
+ dbinfo[i].pubconninfo = conninfo;
+ dbinfo[i].dbname = cell->val;
+ dbinfo[i].made_replslot = false;
+ dbinfo[i].made_publication = false;
+ dbinfo[i].made_subscription = false;
+ /* other struct fields will be filled later. */
+
+ /* Subscriber. */
+ conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
+ dbinfo[i].subconninfo = conninfo;
+
+ i++;
+ }
+
+ return dbinfo;
+}
+
+static PGconn *
+connect_database(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ const char *rconninfo;
+
+ /* logical replication mode */
+ rconninfo = psprintf("%s replication=database", conninfo);
+
+ conn = PQconnectdb(rconninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ pg_log_error("connection to database failed: %s", PQerrorMessage(conn));
+ return NULL;
+ }
+
+ /* secure search_path */
+ res = PQexec(conn, ALWAYS_SECURE_SEARCH_PATH_SQL);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not clear search_path: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+ PQclear(res);
+
+ return conn;
+}
+
+static void
+disconnect_database(PGconn *conn)
+{
+ Assert(conn != NULL);
+
+ PQfinish(conn);
+}
+
+/*
+ * Obtain the system identifier using the provided connection. It will be used
+ * to compare if a data directory is a clone of another one.
+ */
+static uint64
+get_sysid_from_conn(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from publisher");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "IDENTIFY_SYSTEM");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not send replication command \"%s\": %s",
+ "IDENTIFY_SYSTEM", PQresultErrorMessage(res));
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+ if (PQntuples(res) != 1 || PQnfields(res) < 3)
+ {
+ pg_log_error("could not identify system: got %d rows and %d fields, expected %d rows and %d or more fields",
+ PQntuples(res), PQnfields(res), 1, 3);
+
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+
+ sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+
+ pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+
+ disconnect_database(conn);
+
+ return sysid;
+}
+
+/*
+ * Obtain the system identifier from control file. It will be used to compare
+ * if a data directory is a clone of another one. This routine is used locally
+ * and avoids a replication connection.
+ */
+static uint64
+get_control_from_datadir(const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ sysid = cf->system_identifier;
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+
+ pfree(cf);
+
+ return sysid;
+}
+
+/*
+ * Modify the system identifier. Since a standby server preserves the system
+ * identifier, it makes sense to change it to avoid situations in which WAL
+ * files from one of the systems might be used in the other one.
+ */
+static void
+modify_sysid(const char *pg_resetwal_path, const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ struct timeval tv;
+
+ char *cmd_str;
+ int rc;
+
+ pg_log_info("modifying system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ /*
+ * Select a new system identifier.
+ *
+ * XXX this code was extracted from BootStrapXLOG().
+ */
+ gettimeofday(&tv, NULL);
+ cf->system_identifier = ((uint64) tv.tv_sec) << 32;
+ cf->system_identifier |= ((uint64) tv.tv_usec) << 12;
+ cf->system_identifier |= getpid() & 0xFFF;
+
+ if (!dry_run)
+ update_controlfile(datadir, cf, true);
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) cf->system_identifier);
+
+ pg_log_info("running pg_resetwal on the subscriber");
+
+ cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+
+ pg_log_debug("command is: %s", cmd_str);
+
+ if (!dry_run)
+ {
+ rc = system(cmd_str);
+ if (rc == 0)
+ pg_log_info("subscriber successfully changed the system identifier");
+ else
+ pg_log_error("subscriber failed to change system identifier: exit code: %d", rc);
+ }
+
+ pfree(cf);
+}
+
+/*
+ * Return a palloc'd slot name if the replication is using one.
+ */
+static char *
+use_primary_slot_name(void)
+{
+ PGconn *conn;
+ PGresult *res;
+ PQExpBuffer str = createPQExpBuffer();
+ char *slot_name;
+
+ conn = connect_database(dbinfo[0].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "SELECT setting FROM pg_settings WHERE name = 'primary_slot_name'");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain parameter information: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+
+ /*
+ * If primary_slot_name is an empty string, the current replication
+ * connection is not using a replication slot, bail out.
+ */
+ if (strcmp(PQgetvalue(res, 0, 0), "") == 0)
+ {
+ PQclear(res);
+ return NULL;
+ }
+
+ slot_name = pg_strdup(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ disconnect_database(conn);
+
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ appendPQExpBuffer(str,
+ "SELECT 1 FROM pg_replication_slots r INNER JOIN pg_stat_activity a ON (r.active_pid = a.pid) WHERE slot_name = '%s'", slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain replication slot information: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
+ PQntuples(res), 1);
+ return NULL;
+ }
+
+ PQclear(res);
+ disconnect_database(conn);
+
+ return slot_name;
+}
+
+static bool
+create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
+{
+ int i;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ PGconn *conn;
+ PGresult *res;
+ char replslotname[NAMEDATALEN];
+
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT oid FROM pg_catalog.pg_database WHERE datname = current_database()");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain database OID: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain database OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ return false;
+ }
+
+ /* Remember database OID. */
+ dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+
+ PQclear(res);
+
+ /*
+ * Build the replication slot name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 36
+ * characters (14 + 10 + 1 + 10 + '\0'). System identifier is included
+ * to reduce the probability of collision. By default, subscription
+ * name is used as replication slot name.
+ */
+ snprintf(replslotname, sizeof(replslotname),
+ "pg_subscriber_%u_%d",
+ dbinfo[i].oid,
+ (int) getpid());
+ dbinfo[i].subname = pg_strdup(replslotname);
+
+ /* Create replication slot on publisher. */
+ if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
+ pg_log_info("create replication slot \"%s\" on publisher", replslotname);
+ else
+ return false;
+
+ disconnect_database(conn);
+ }
+
+ return true;
+}
+
+/*
+ * Create a logical replication slot and returns a consistent LSN. The returned
+ * LSN might be used to catch up the subscriber up to the required point.
+ *
+ * CreateReplicationSlot() is not used because it does not provide the one-row
+ * result set that contains the consistent LSN.
+ */
+static char *
+create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res = NULL;
+ char *lsn = NULL;
+ bool transient_replslot = false;
+
+ Assert(conn != NULL);
+
+ /*
+ * If no slot name is informed, it is a transient replication slot used
+ * only for catch up purposes.
+ */
+ if (slot_name[0] == '\0')
+ {
+ snprintf(slot_name, NAMEDATALEN, "pg_subscriber_%d_startpoint",
+ (int) getpid());
+ transient_replslot = true;
+ }
+
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
+ appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQresultErrorMessage(res));
+ return lsn;
+ }
+ }
+
+ /* for cleanup purposes */
+ if (transient_replslot)
+ made_transient_replslot = true;
+ else
+ dbinfo->made_replslot = true;
+
+ if (!dry_run)
+ {
+ lsn = pg_strdup(PQgetvalue(res, 0, 1));
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+
+ return lsn;
+}
+
+static void
+drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Reports a suitable message if pg_ctl fails.
+ */
+static void
+pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
+{
+ if (rc != 0)
+ {
+ if (WIFEXITED(rc))
+ {
+ pg_log_error("pg_ctl failed with exit code %d", WEXITSTATUS(rc));
+ }
+ else if (WIFSIGNALED(rc))
+ {
+#if defined(WIN32)
+ pg_log_error("pg_ctl was terminated by exception 0x%X", WTERMSIG(rc));
+ pg_log_error_detail("See C include file \"ntstatus.h\" for a description of the hexadecimal value.");
+#else
+ pg_log_error("pg_ctl was terminated by signal %d: %s",
+ WTERMSIG(rc), pg_strsignal(WTERMSIG(rc)));
+#endif
+ }
+ else
+ {
+ pg_log_error("pg_ctl exited with unrecognized status %d", rc);
+ }
+
+ pg_log_error_detail("The failed command was: %s", pg_ctl_cmd);
+ exit(1);
+ }
+
+ if (action)
+ pg_log_info("postmaster was started");
+ else
+ pg_log_info("postmaster was stopped");
+}
+
+/*
+ * Returns after the server finishes the recovery process.
+ */
+static void
+wait_for_end_recovery(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ int status = POSTMASTER_STILL_STARTING;
+
+ pg_log_info("waiting the postmaster to reach the consistent state");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ for (;;)
+ {
+ bool in_recovery;
+
+ res = PQexec(conn, "SELECT pg_catalog.pg_is_in_recovery()");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain recovery progress");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("unexpected result from pg_is_in_recovery function");
+ exit(1);
+ }
+
+ in_recovery = (strcmp(PQgetvalue(res, 0, 0), "t") == 0);
+
+ PQclear(res);
+
+ /*
+ * Does the recovery process finish? In dry run mode, there is no
+ * recovery mode. Bail out as the recovery process has ended.
+ */
+ if (!in_recovery || dry_run)
+ {
+ status = POSTMASTER_READY;
+ break;
+ }
+
+ /* Keep waiting. */
+ pg_usleep(WAIT_INTERVAL * USEC_PER_SEC);
+ }
+
+ disconnect_database(conn);
+
+ if (status == POSTMASTER_STILL_STARTING)
+ {
+ pg_log_error("server did not end recovery");
+ exit(1);
+ }
+
+ pg_log_info("postmaster reached the consistent state");
+}
+
+/*
+ * Create a publication that includes all tables in the database.
+ */
+static void
+create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ /* Check if the publication needs to be created. */
+ appendPQExpBuffer(str,
+ "SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
+ dbinfo->pubname);
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publication information: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) == 1)
+ {
+ /*
+ * If publication name already exists and puballtables is true, let's
+ * use it. A previous run of pg_subscriber must have created this
+ * publication. Bail out.
+ */
+ if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
+ {
+ pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ return;
+ }
+ else
+ {
+ /*
+ * Unfortunately, if it reaches this code path, it will always
+ * fail (unless you decide to change the existing publication
+ * name). That's bad but it is very unlikely that the user will
+ * choose a name with pg_subscriber_ prefix followed by the exact
+ * database oid in which puballtables is false.
+ */
+ pg_log_error("publication \"%s\" does not replicate changes for all tables",
+ dbinfo->pubname);
+ pg_log_error_hint("Consider renaming this publication.");
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(str);
+
+ pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
+ dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_publication = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove publication if it couldn't finish all steps.
+ */
+static void
+drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Create a subscription with some predefined options.
+ *
+ * A replication slot was already created in a previous step. Let's use it. By
+ * default, the subscription name is used as replication slot name. It is
+ * not required to copy data. The subscription will be created but it will not
+ * be enabled now. That's because the replication progress must be set and the
+ * replication origin name (one of the function arguments) contains the
+ * subscription OID in its name. Once the subscription is created,
+ * set_replication_progress() can obtain the chosen origin name and set up its
+ * initial location.
+ */
+static void
+create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str,
+ "CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
+ "WITH (create_slot = false, copy_data = false, enabled = false)",
+ dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
+ dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_subscription = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove subscription if it couldn't finish all steps.
+ */
+static void
+drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Sets the replication progress to the consistent LSN.
+ *
+ * The subscriber caught up to the consistent LSN provided by the temporary
+ * replication slot. The goal is to set up the initial location for the logical
+ * replication that is the exact LSN that the subscriber was promoted. Once the
+ * subscription is enabled it will start streaming from that location onwards.
+ * In dry run mode, the subscription OID and LSN are set to invalid values for
+ * printing purposes.
+ */
+static void
+set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+ Oid suboid;
+ char originname[NAMEDATALEN];
+ char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+
+ Assert(conn != NULL);
+
+ appendPQExpBuffer(str,
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscription OID: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1 && !dry_run)
+ {
+ pg_log_error("could not obtain subscription OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (dry_run)
+ {
+ suboid = InvalidOid;
+ snprintf(lsnstr, sizeof(lsnstr), "%X/%X", LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ suboid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ snprintf(lsnstr, sizeof(lsnstr), "%s", lsn);
+ }
+
+ /*
+ * The origin name is defined as pg_%u. %u is the subscription OID. See
+ * ApplyWorkerMain().
+ */
+ snprintf(originname, sizeof(originname), "pg_%u", suboid);
+
+ PQclear(res);
+
+ pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
+ originname, lsnstr, dbinfo->dbname);
+
+ resetPQExpBuffer(str);
+ appendPQExpBuffer(str,
+ "SELECT pg_catalog.pg_replication_origin_advance('%s', '%s')", originname, lsnstr);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not set replication progress for the subscription \"%s\": %s",
+ dbinfo->subname, PQresultErrorMessage(res));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Enables the subscription.
+ *
+ * The subscription was created in a previous step but it was disabled. After
+ * adjusting the initial location, enabling the subscription is the last step
+ * of this setup.
+ */
+static void
+enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+int
+main(int argc, char **argv)
+{
+ static struct option long_options[] =
+ {
+ {"help", no_argument, NULL, '?'},
+ {"version", no_argument, NULL, 'V'},
+ {"pgdata", required_argument, NULL, 'D'},
+ {"publisher-conninfo", required_argument, NULL, 'P'},
+ {"subscriber-conninfo", required_argument, NULL, 'S'},
+ {"database", required_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"verbose", no_argument, NULL, 'v'},
+ {NULL, 0, NULL, 0}
+ };
+
+ int c;
+ int option_index;
+ int rc;
+
+ char *pg_ctl_cmd;
+
+ char *base_dir;
+ char *server_start_log;
+
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+
+ char *pub_base_conninfo = NULL;
+ char *sub_base_conninfo = NULL;
+ char *dbname_conninfo = NULL;
+
+ uint64 pub_sysid;
+ uint64 sub_sysid;
+ struct stat statbuf;
+
+ PGconn *conn;
+ char *consistent_lsn;
+
+ PQExpBuffer recoveryconfcontents = NULL;
+
+ char pidfile[MAXPGPATH];
+
+ int i;
+
+ pg_logging_init(argv[0]);
+ pg_logging_set_level(PG_LOG_WARNING);
+ progname = get_progname(argv[0]);
+ set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_subscriber"));
+
+ if (argc > 1)
+ {
+ if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+ {
+ usage();
+ exit(0);
+ }
+ else if (strcmp(argv[1], "-V") == 0
+ || strcmp(argv[1], "--version") == 0)
+ {
+ puts("pg_subscriber (PostgreSQL) " PG_VERSION);
+ exit(0);
+ }
+ }
+
+ atexit(cleanup_objects_atexit);
+
+ /*
+ * Don't allow it to be run as root. It uses pg_ctl which does not allow
+ * it either.
+ */
+#ifndef WIN32
+ if (geteuid() == 0)
+ {
+ pg_log_error("cannot be executed by \"root\"");
+ pg_log_error_hint("You must run %s as the PostgreSQL superuser.",
+ progname);
+ exit(1);
+ }
+#endif
+
+ while ((c = getopt_long(argc, argv, "D:P:S:d:nv",
+ long_options, &option_index)) != -1)
+ {
+ switch (c)
+ {
+ case 'D':
+ subscriber_dir = pg_strdup(optarg);
+ break;
+ case 'P':
+ pub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'S':
+ sub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'd':
+ /* Ignore duplicated database names. */
+ if (!simple_string_list_member(&database_names, optarg))
+ {
+ simple_string_list_append(&database_names, optarg);
+ num_dbs++;
+ }
+ break;
+ case 'n':
+ dry_run = true;
+ break;
+ case 'v':
+ pg_logging_increase_verbosity();
+ break;
+ default:
+ /* getopt_long already emitted a complaint */
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Any non-option arguments?
+ */
+ if (optind < argc)
+ {
+ pg_log_error("too many command-line arguments (first is \"%s\")",
+ argv[optind]);
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Required arguments
+ */
+ if (subscriber_dir == NULL)
+ {
+ pg_log_error("no subscriber data directory specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Parse connection string. Build a base connection string that might be
+ * reused by multiple databases.
+ */
+ if (pub_conninfo_str == NULL)
+ {
+ /*
+ * TODO use primary_conninfo (if available) from subscriber and
+ * extract publisher connection string. Assume that there are
+ * identical entries for physical and logical replication. If there is
+ * not, we would fail anyway.
+ */
+ pg_log_error("no publisher connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
+ "publisher");
+ if (pub_base_conninfo == NULL)
+ exit(1);
+
+ if (sub_conninfo_str == NULL)
+ {
+ pg_log_error("no subscriber connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
+ if (sub_base_conninfo == NULL)
+ exit(1);
+
+ if (database_names.head == NULL)
+ {
+ pg_log_info("no database was specified");
+
+ /*
+ * If --database option is not provided, try to obtain the dbname from
+ * the publisher conninfo. If dbname parameter is not available, error
+ * out.
+ */
+ if (dbname_conninfo)
+ {
+ simple_string_list_append(&database_names, dbname_conninfo);
+ num_dbs++;
+
+ pg_log_info("database \"%s\" was extracted from the publisher connection string",
+ dbname_conninfo);
+ }
+ else
+ {
+ pg_log_error("no database name specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
+ */
+ if (!get_exec_path(argv[0]))
+ exit(1);
+
+ /* rudimentary check for a data directory. */
+ if (!check_data_directory(subscriber_dir))
+ exit(1);
+
+ /* Store database information for publisher and subscriber. */
+ dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+
+ /*
+ * Check if the subscriber data directory has the same system identifier
+ * than the publisher data directory.
+ */
+ pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
+ sub_sysid = get_control_from_datadir(subscriber_dir);
+ if (pub_sysid != sub_sysid)
+ {
+ pg_log_error("subscriber data directory is not a copy of the source database cluster");
+ exit(1);
+ }
+
+ /*
+ * Create the output directory to store any data generated by this tool.
+ */
+ base_dir = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("directory path for subscriber is too long");
+ exit(1);
+ }
+
+ if (mkdir(base_dir, pg_dir_create_mode) < 0 && errno != EEXIST)
+ {
+ pg_log_error("could not create directory \"%s\": %m", base_dir);
+ exit(1);
+ }
+
+ /* subscriber PID file. */
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+
+ /*
+ * Stop the subscriber if it is a standby server. Before executing the
+ * transformation steps, make sure the subscriber is not running because
+ * one of the steps is to modify some recovery parameters that require a
+ * restart.
+ */
+ if (stat(pidfile, &statbuf) == 0)
+ {
+ /*
+ * Since the standby server is running, check if it is using an
+ * existing replication slot for WAL retention purposes. This
+ * replication slot has no use after the transformation, hence, it
+ * will be removed at the end of this process.
+ */
+ primary_slot_name = use_primary_slot_name();
+ if (primary_slot_name != NULL)
+ pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+
+ pg_log_info("subscriber is up and running");
+ pg_log_info("stopping the server to start the transformation steps");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+ }
+
+ /*
+ * Create a replication slot for each database on the publisher.
+ */
+ if (!create_all_logical_replication_slots(dbinfo))
+ exit(1);
+
+ /*
+ * Create a logical replication slot to get a consistent LSN.
+ *
+ * This consistent LSN will be used later to advanced the recently created
+ * replication slots. We cannot use the last created replication slot
+ * because the consistent LSN should be obtained *after* the base backup
+ * finishes (and the base backup should include the logical replication
+ * slots).
+ *
+ * XXX we should probably use the last created replication slot to get a
+ * consistent LSN but it should be changed after adding pg_basebackup
+ * support.
+ *
+ * A temporary replication slot is not used here to avoid keeping a
+ * replication connection open (depending when base backup was taken, the
+ * connection should be open for a few hours).
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+ consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
+ temp_replslot);
+
+ /*
+ * Write recovery parameters.
+ *
+ * Despite of the recovery parameters will be written to the subscriber,
+ * use a publisher connection for the follwing recovery functions. The
+ * connection is only used to check the current server version (physical
+ * replica, same server version). The subscriber is not running yet. In
+ * dry run mode, the recovery parameters *won't* be written. An invalid
+ * LSN is used for printing purposes.
+ */
+ recoveryconfcontents = GenerateRecoveryConfig(conn, NULL);
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_inclusive = true\n");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_action = promote\n");
+
+ if (dry_run)
+ {
+ appendPQExpBuffer(recoveryconfcontents, "# dry run mode");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%X/%X'\n",
+ LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
+ consistent_lsn);
+ WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ }
+ disconnect_database(conn);
+
+ pg_log_debug("recovery parameters:\n%s", recoveryconfcontents->data);
+
+ /*
+ * Start subscriber and wait until accepting connections.
+ */
+ pg_log_info("starting the subscriber");
+
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ server_start_log = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(server_start_log, MAXPGPATH, "%s/%s/server_start_%s.log", subscriber_dir, PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+
+ pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, subscriber_dir, server_start_log);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+
+ /*
+ * Waiting the subscriber to be promoted.
+ */
+ wait_for_end_recovery(dbinfo[0].subconninfo);
+
+ /*
+ * Create a publication for each database. This step should be executed
+ * after promoting the subscriber to avoid replicating unnecessary
+ * objects.
+ */
+ for (i = 0; i < num_dbs; i++)
+ {
+ char pubname[NAMEDATALEN];
+
+ /* Connect to publisher. */
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ /*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 35 characters (14 + 10 +
+ * '\0').
+ */
+ snprintf(pubname, sizeof(pubname), "pg_subscriber_%u", dbinfo[i].oid);
+ dbinfo[i].pubname = pg_strdup(pubname);
+
+ create_publication(conn, &dbinfo[i]);
+
+ disconnect_database(conn);
+ }
+
+ /*
+ * Create a subscription for each database.
+ */
+ for (i = 0; i < num_dbs; i++)
+ {
+ /* Connect to subscriber. */
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ create_subscription(conn, &dbinfo[i]);
+
+ /* Set the replication progress to the correct LSN. */
+ set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+
+ /* Enable subscription. */
+ enable_subscription(conn, &dbinfo[i]);
+
+ disconnect_database(conn);
+ }
+
+ /*
+ * The transient replication slot is no longer required. Drop it.
+ *
+ * If the physical replication slot exists, drop it.
+ *
+ * XXX we might not fail here. Instead, we provide a warning so the user
+ * eventually drops the replication slot later.
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ {
+ pg_log_warning("could not drop transient replication slot \"%s\" on publisher", temp_replslot);
+ pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
+ if (primary_slot_name != NULL)
+ pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
+ }
+ else
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ if (primary_slot_name != NULL)
+ drop_replication_slot(conn, &dbinfo[0], primary_slot_name);
+ disconnect_database(conn);
+ }
+
+ /*
+ * Stop the subscriber.
+ */
+ pg_log_info("stopping the subscriber");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+ /*
+ * Change system identifier.
+ */
+ modify_sysid(pg_resetwal_path, subscriber_dir);
+
+ /*
+ * Remove log file generated by this tool, if it runs successfully.
+ * Otherwise, file is kept that may provide useful debugging information.
+ */
+ unlink(server_start_log);
+
+ success = true;
+
+ pg_log_info("Done!");
+
+ return 0;
+}
diff --git a/src/bin/pg_basebackup/t/040_pg_subscriber.pl b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
new file mode 100644
index 0000000000..4ebff76b2d
--- /dev/null
+++ b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
@@ -0,0 +1,44 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test checking options of pg_subscriber.
+#
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+program_help_ok('pg_subscriber');
+program_version_ok('pg_subscriber');
+program_options_handling_ok('pg_subscriber');
+
+my $datadir = PostgreSQL::Test::Utils::tempdir;
+
+command_fails(['pg_subscriber'],
+ 'no subscriber data directory specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--pgdata', $datadir
+ ],
+ 'no publisher connection string specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--dry-run',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres'
+ ],
+ 'no subscriber connection string specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--verbose',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres',
+ '--subscriber-conninfo', 'dbname=postgres'
+ ],
+ 'no database name specified');
+
+done_testing();
diff --git a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
new file mode 100644
index 0000000000..fbcd0fc82b
--- /dev/null
+++ b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
@@ -0,0 +1,139 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test using a standby server as the subscriber.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node_p;
+my $node_f;
+my $node_s;
+my $result;
+
+# Set up node P as primary
+$node_p = PostgreSQL::Test::Cluster->new('node_p');
+$node_p->init(allows_streaming => 'logical');
+$node_p->start;
+
+# Set up node F as about-to-fail node
+# The extra option forces it to initialize a new cluster instead of copying a
+# previously initdb's cluster.
+$node_f = PostgreSQL::Test::Cluster->new('node_f');
+$node_f->init(allows_streaming => 'logical', extra => [ '--no-instructions' ]);
+$node_f->start;
+
+# On node P
+# - create databases
+# - create test tables
+# - insert a row
+$node_p->safe_psql(
+ 'postgres', q(
+ CREATE DATABASE pg1;
+ CREATE DATABASE pg2;
+));
+$node_p->safe_psql('pg1', 'CREATE TABLE tbl1 (a text)');
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('first row')");
+$node_p->safe_psql('pg2', 'CREATE TABLE tbl2 (a text)');
+
+# Set up node S as standby linking to node P
+$node_p->backup('backup_1');
+$node_s = PostgreSQL::Test::Cluster->new('node_s');
+$node_s->init_from_backup($node_p, 'backup_1', has_streaming => 1);
+$node_s->append_conf('postgresql.conf', 'log_min_messages = debug2');
+$node_s->set_standby_mode();
+$node_s->start;
+
+# Insert another row on node P and wait node S to catch up
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('second row')");
+$node_p->wait_for_replay_catchup($node_s);
+
+# Run pg_subscriber on about-to-fail node F
+command_fails(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $node_f->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_f->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'subscriber data directory is not a copy of the source database cluster');
+
+# dry run mode on node S
+command_ok(
+ [
+ 'pg_subscriber', '--verbose', '--dry-run',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_subscriber --dry-run on node S');
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+# Check if node S is still a standby
+is($node_s->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
+ 't', 'standby is in recovery');
+
+# Run pg_subscriber on node S
+command_ok(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_subscriber on node S');
+
+# Insert rows on P
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('third row')");
+$node_p->safe_psql('pg2', "INSERT INTO tbl2 VALUES('row 1')");
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+
+# Get subscription names
+$result = $node_s->safe_psql(
+ 'postgres', qq(
+ SELECT subname FROM pg_subscription WHERE subname ~ '^pg_subscriber_'
+));
+my @subnames = split("\n", $result);
+
+# Wait subscriber to catch up
+$node_s->wait_for_subscription_sync($node_p, $subnames[0]);
+$node_s->wait_for_subscription_sync($node_p, $subnames[1]);
+
+# Check result on database pg1
+$result = $node_s->safe_psql('pg1', 'SELECT * FROM tbl1');
+is( $result, qq(first row
+second row
+third row),
+ 'logical replication works on database pg1');
+
+# Check result on database pg2
+$result = $node_s->safe_psql('pg2', 'SELECT * FROM tbl2');
+is( $result, qq(row 1),
+ 'logical replication works on database pg2');
+
+# Different system identifier?
+my $sysid_p = $node_p->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+my $sysid_s = $node_s->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+ok($sysid_p != $sysid_s, 'system identifier was changed');
+
+# clean up
+$node_p->teardown_node;
+$node_s->teardown_node;
+
+done_testing();
--
2.43.0
[application/octet-stream] v7-0002-Address-some-comments-proposed-on-hackers.patch (56.5K, ../../TY3PR01MB98890AB22AECA2F503EB7858F5742@TY3PR01MB9889.jpnprd01.prod.outlook.com/3-v7-0002-Address-some-comments-proposed-on-hackers.patch)
download | inline diff:
From cfb77f4c599417527f7bfbcb7e8d90a4b09b5108 Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Mon, 22 Jan 2024 12:42:34 +0530
Subject: [PATCH v7 2/3] Address some comments proposed on -hackers
The patch has following changes:
* Some comments reported on the thread
* Add a timeout option for the recovery option
* Reject if the target server is not a standby
* Reject when the --subscriber-conninfo specifies non-local server
* Add -u and -p options
* Check wal_level and max_replication_slot parameters
---
doc/src/sgml/ref/pg_subscriber.sgml | 21 +-
src/bin/pg_basebackup/pg_subscriber.c | 911 +++++++++++-------
src/bin/pg_basebackup/t/040_pg_subscriber.pl | 9 +-
.../t/041_pg_subscriber_standby.pl | 8 +-
4 files changed, 601 insertions(+), 348 deletions(-)
diff --git a/doc/src/sgml/ref/pg_subscriber.sgml b/doc/src/sgml/ref/pg_subscriber.sgml
index 553185c35f..eaabfc7053 100644
--- a/doc/src/sgml/ref/pg_subscriber.sgml
+++ b/doc/src/sgml/ref/pg_subscriber.sgml
@@ -16,12 +16,18 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_subscriber</refname>
- <refpurpose>create a new logical replica from a standby server</refpurpose>
+ <refpurpose>Convert a standby replica to a logical replica</refpurpose>
</refnamediv>
<refsynopsisdiv>
<cmdsynopsis>
<command>pg_subscriber</command>
+ <arg choice="plain"><option>-D</option></arg>
+ <arg choice="plain"><replaceable>datadir</replaceable></arg>
+ <arg choice="plain"><option>-P</option>
+ <replaceable>publisher-conninfo</replaceable></arg>
+ <arg choice="plain"><option>-S</option></arg>
+ <arg choice="plain"><replaceable>subscriber-conninfo</replaceable></arg>
<arg rep="repeat"><replaceable>option</replaceable></arg>
</cmdsynopsis>
</refsynopsisdiv>
@@ -29,17 +35,18 @@ PostgreSQL documentation
<refsect1>
<title>Description</title>
<para>
- <application>pg_subscriber</application> takes the publisher and subscriber
- connection strings, a cluster directory from a standby server and a list of
- database names and it sets up a new logical replica using the physical
- recovery process.
+ pg_subscriber creates a new <link
+ linkend="logical-replication-subscription">subscriber</link> from a physical
+ standby server. This allows users to quickly set up logical replication
+ system.
</para>
<para>
- The <application>pg_subscriber</application> should be run at the target
+ The <application>pg_subscriber</application> has to be run at the target
server. The source server (known as publisher server) should accept logical
replication connections from the target server (known as subscriber server).
- The target server should accept local logical replication connection.
+ The target server should accept logical replication connection from
+ localhost.
</para>
</refsect1>
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index e998c29f9e..3880d15ef9 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -1,12 +1,12 @@
/*-------------------------------------------------------------------------
*
* pg_subscriber.c
- * Create a new logical replica from a standby server
+ * Convert a standby replica to a logical replica
*
* Copyright (C) 2024, PostgreSQL Global Development Group
*
* IDENTIFICATION
- * src/bin/pg_subscriber/pg_subscriber.c
+ * src/bin/pg_basebackup/pg_subscriber.c
*
*-------------------------------------------------------------------------
*/
@@ -32,81 +32,122 @@
#define PGS_OUTPUT_DIR "pg_subscriber_output.d"
-typedef struct LogicalRepInfo
+typedef struct LogicalRepPerdbInfo
{
- Oid oid; /* database OID */
- char *dbname; /* database name */
- char *pubconninfo; /* publication connection string for logical
- * replication */
- char *subconninfo; /* subscription connection string for logical
- * replication */
- char *pubname; /* publication name */
- char *subname; /* subscription name (also replication slot
- * name) */
-
- bool made_replslot; /* replication slot was created */
- bool made_publication; /* publication was created */
- bool made_subscription; /* subscription was created */
-} LogicalRepInfo;
+ Oid oid;
+ char *dbname;
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepPerdbInfo;
+
+typedef struct
+{
+ LogicalRepPerdbInfo *perdb; /* array of db infos */
+ int ndbs; /* number of db infos */
+} LogicalRepPerdbInfoArr;
+
+typedef struct PrimaryInfo
+{
+ char *base_conninfo;
+ uint64 sysid;
+} PrimaryInfo;
+
+typedef struct StandbyInfo
+{
+ char *base_conninfo;
+ char *bindir;
+ char *pgdata;
+ char *primary_slot_name;
+ uint64 sysid;
+} StandbyInfo;
static void cleanup_objects_atexit(void);
static void usage();
-static char *get_base_conninfo(char *conninfo, char *dbname,
- const char *noderole);
-static bool get_exec_path(const char *path);
+static char *get_base_conninfo(char *conninfo, char *dbname);
+static bool get_exec_base_path(const char *path);
static bool check_data_directory(const char *datadir);
+static void store_db_names(LogicalRepPerdbInfo **perdb, int ndbs);
+static void get_sysid_for_primary(PrimaryInfo *primary, char *dbname);
+static void get_control_for_standby(StandbyInfo *standby);
static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
-static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
-static PGconn *connect_database(const char *conninfo);
+static PGconn *connect_database(const char *base_conninfo, const char*dbname);
static void disconnect_database(PGconn *conn);
-static uint64 get_sysid_from_conn(const char *conninfo);
-static uint64 get_control_from_datadir(const char *datadir);
-static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
-static char *use_primary_slot_name(void);
-static bool create_all_logical_replication_slots(LogicalRepInfo *dbinfo);
-static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
- char *slot_name);
-static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static char *use_primary_slot_name(PrimaryInfo *primary, StandbyInfo *standby,
+ LogicalRepPerdbInfo *perdb);
+static bool create_all_logical_replication_slots(PrimaryInfo *primary,
+ LogicalRepPerdbInfoArr *dbarr);
+static char *create_logical_replication_slot(PGconn *conn, bool temporary,
+ LogicalRepPerdbInfo *perdb);
+static void modify_sysid(const char *bindir, const char *datadir);
+static void drop_replication_slot(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *slot_name);
static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
-static void wait_for_end_recovery(const char *conninfo);
-static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
-static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
-static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
-static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void wait_for_end_recovery(const char *base_conninfo,
+ const char *dbname);
+static void create_publication(PGconn *conn, PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb);
+static void drop_publication(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void create_subscription(PGconn *conn, StandbyInfo *standby,
+ char *base_conninfo,
+ LogicalRepPerdbInfo *perdb);
+static void drop_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void set_replication_progress(PGconn *conn, LogicalRepPerdbInfo *perdb, const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void start_standby_server(StandbyInfo *standby, unsigned short subport,
+ char *server_start_log);
+static char *construct_sub_conninfo(char *username, unsigned short subport);
#define USEC_PER_SEC 1000000
-#define WAIT_INTERVAL 1 /* 1 second */
+#define DEFAULT_WAIT 60
+#define WAITS_PER_SEC 10 /* should divide USEC_PER_SEC evenly */
+#define DEF_PGSPORT 50111
/* Options */
-static const char *progname;
-
-static char *subscriber_dir = NULL;
static char *pub_conninfo_str = NULL;
-static char *sub_conninfo_str = NULL;
static SimpleStringList database_names = {NULL, NULL};
-static char *primary_slot_name = NULL;
+static int wait_seconds = DEFAULT_WAIT;
+static bool retain = false;
static bool dry_run = false;
static bool success = false;
+static const char *progname;
+static LogicalRepPerdbInfoArr dbarr;
+static PrimaryInfo primary;
+static StandbyInfo standby;
-static char *pg_ctl_path = NULL;
-static char *pg_resetwal_path = NULL;
+enum PGSWaitPMResult
+{
+ PGS_POSTMASTER_READY,
+ PGS_POSTMASTER_STANDBY,
+ PGS_POSTMASTER_STILL_STARTING,
+ PGS_POSTMASTER_FAILED
+};
-static LogicalRepInfo *dbinfo;
-static int num_dbs = 0;
-static char temp_replslot[NAMEDATALEN] = {0};
-static bool made_transient_replslot = false;
+/*
+ * Build the replication slot and subscription name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 36 characters
+ * (14 + 10 + 1 + 10 + '\0'). System identifier is included to reduce the
+ * probability of collision. By default, subscription name is used as
+ * replication slot name.
+ */
+static inline void
+get_subscription_name(Oid oid, int pid, char *subname, Size szsub)
+{
+ snprintf(subname, szsub, "pg_subscriber_%u_%d", oid, pid);
+}
-enum WaitPMResult
+/*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 35 characters (14 + 10 +
+ * '\0').
+ */
+static inline void
+get_publication_name(Oid oid, char *pubname, Size szpub)
{
- POSTMASTER_READY,
- POSTMASTER_STANDBY,
- POSTMASTER_STILL_STARTING,
- POSTMASTER_FAILED
-};
+ snprintf(pubname, szpub, "pg_subscriber_%u", oid);
+}
/*
@@ -125,41 +166,39 @@ cleanup_objects_atexit(void)
if (success)
return;
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
- if (dbinfo[i].made_subscription)
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
+
+ if (perdb->made_subscription)
{
- conn = connect_database(dbinfo[i].subconninfo);
+ conn = connect_database(standby.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- drop_subscription(conn, &dbinfo[i]);
+ drop_subscription(conn, perdb);
disconnect_database(conn);
}
}
- if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ if (perdb->made_publication || perdb->made_replslot)
{
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- if (dbinfo[i].made_publication)
- drop_publication(conn, &dbinfo[i]);
- if (dbinfo[i].made_replslot)
- drop_replication_slot(conn, &dbinfo[i], NULL);
+ if (perdb->made_publication)
+ drop_publication(conn, perdb);
+ if (perdb->made_replslot)
+ {
+ char replslotname[NAMEDATALEN];
+
+ get_subscription_name(perdb->oid, (int) getpid(),
+ replslotname, NAMEDATALEN);
+ drop_replication_slot(conn, perdb, replslotname);
+ }
disconnect_database(conn);
}
}
}
-
- if (made_transient_replslot)
- {
- conn = connect_database(dbinfo[0].pubconninfo);
- if (conn != NULL)
- {
- drop_replication_slot(conn, &dbinfo[0], temp_replslot);
- disconnect_database(conn);
- }
- }
}
static void
@@ -184,17 +223,16 @@ usage(void)
/*
* Validate a connection string. Returns a base connection string that is a
- * connection string without a database name plus a fallback application name.
- * Since we might process multiple databases, each database name will be
- * appended to this base connection string to provide a final connection string.
- * If the second argument (dbname) is not null, returns dbname if the provided
- * connection string contains it. If option --database is not provided, uses
- * dbname as the only database to setup the logical replica.
- * It is the caller's responsibility to free the returned connection string and
- * dbname.
+ * connection string without a database name. Since we might process multiple
+ * databases, each database name will be appended to this base connection
+ * string to provide a final connection string. If the second argument (dbname)
+ * is not null, returns dbname if the provided connection string contains it.
+ * If option --database is not provided, uses dbname as the only database to
+ * setup the logical replica. It is the caller's responsibility to free the
+ * returned connection string and dbname.
*/
static char *
-get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+get_base_conninfo(char *conninfo, char *dbname)
{
PQExpBuffer buf = createPQExpBuffer();
PQconninfoOption *conn_opts = NULL;
@@ -203,7 +241,7 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
char *ret;
int i;
- pg_log_info("validating connection string on %s", noderole);
+ pg_log_info("validating connection string on publisher");
conn_opts = PQconninfoParse(conninfo, &errmsg);
if (conn_opts == NULL)
@@ -231,10 +269,6 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
}
}
- if (i > 0)
- appendPQExpBufferChar(buf, ' ');
- appendPQExpBuffer(buf, "fallback_application_name=%s", progname);
-
ret = pg_strdup(buf->data);
destroyPQExpBuffer(buf);
@@ -244,15 +278,16 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
}
/*
- * Get the absolute path from other PostgreSQL binaries (pg_ctl and
- * pg_resetwal) that is used by it.
+ * Get the absolute binary path from another PostgreSQL binary (pg_ctl) and set
+ * to StandbyInfo.
*/
static bool
-get_exec_path(const char *path)
+get_exec_base_path(const char *path)
{
int rc;
+ char pg_ctl_path[MAXPGPATH];
+ char *p;
- pg_ctl_path = pg_malloc(MAXPGPATH);
rc = find_other_exec(path, "pg_ctl",
"pg_ctl (PostgreSQL) " PG_VERSION "\n",
pg_ctl_path);
@@ -277,30 +312,12 @@ get_exec_path(const char *path)
pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
- pg_resetwal_path = pg_malloc(MAXPGPATH);
- rc = find_other_exec(path, "pg_resetwal",
- "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
- pg_resetwal_path);
- if (rc < 0)
- {
- char full_path[MAXPGPATH];
-
- if (find_my_exec(path, full_path) < 0)
- strlcpy(full_path, progname, sizeof(full_path));
- if (rc == -1)
- pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
- "same directory as \"%s\".\n"
- "Check your installation.",
- "pg_resetwal", progname, full_path);
- else
- pg_log_error("The program \"%s\" was found by \"%s\"\n"
- "but was not the same version as %s.\n"
- "Check your installation.",
- "pg_resetwal", full_path, progname);
- return false;
- }
+ /* Extract the directory part from the path */
+ p = strrchr(pg_ctl_path, 'p');
+ Assert(p);
- pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+ *p = '\0';
+ standby.bindir = pg_strdup(pg_ctl_path);
return true;
}
@@ -364,49 +381,36 @@ concat_conninfo_dbname(const char *conninfo, const char *dbname)
}
/*
- * Store publication and subscription information.
+ * Initialize per-db structure and store the name of databases
*/
-static LogicalRepInfo *
-store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+static void
+store_db_names(LogicalRepPerdbInfo **perdb, int ndbs)
{
- LogicalRepInfo *dbinfo;
SimpleStringListCell *cell;
int i = 0;
- dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+ *perdb = (LogicalRepPerdbInfo *) pg_malloc0(sizeof(LogicalRepPerdbInfo) *
+ ndbs);
for (cell = database_names.head; cell; cell = cell->next)
{
- char *conninfo;
-
- /* Publisher. */
- conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
- dbinfo[i].pubconninfo = conninfo;
- dbinfo[i].dbname = cell->val;
- dbinfo[i].made_replslot = false;
- dbinfo[i].made_publication = false;
- dbinfo[i].made_subscription = false;
- /* other struct fields will be filled later. */
-
- /* Subscriber. */
- conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
- dbinfo[i].subconninfo = conninfo;
-
+ (*perdb)[i].dbname = pg_strdup(cell->val);
i++;
}
-
- return dbinfo;
}
static PGconn *
-connect_database(const char *conninfo)
+connect_database(const char *base_conninfo, const char*dbname)
{
PGconn *conn;
PGresult *res;
- const char *rconninfo;
+
+ char *rconninfo;
+ char *concat_conninfo = concat_conninfo_dbname(base_conninfo,
+ dbname);
/* logical replication mode */
- rconninfo = psprintf("%s replication=database", conninfo);
+ rconninfo = psprintf("%s replication=database", concat_conninfo);
conn = PQconnectdb(rconninfo);
if (PQstatus(conn) != CONNECTION_OK)
@@ -424,6 +428,9 @@ connect_database(const char *conninfo)
}
PQclear(res);
+ pfree(rconninfo);
+ pfree(concat_conninfo);
+
return conn;
}
@@ -436,19 +443,18 @@ disconnect_database(PGconn *conn)
}
/*
- * Obtain the system identifier using the provided connection. It will be used
- * to compare if a data directory is a clone of another one.
+ * Obtain the system identifier from the primary server. It will be used to
+ * compare if a data directory is a clone of another one.
*/
-static uint64
-get_sysid_from_conn(const char *conninfo)
+static void
+get_sysid_for_primary(PrimaryInfo *primary, char *dbname)
{
PGconn *conn;
PGresult *res;
- uint64 sysid;
pg_log_info("getting system identifier from publisher");
- conn = connect_database(conninfo);
+ conn = connect_database(primary->base_conninfo, dbname);
if (conn == NULL)
exit(1);
@@ -471,43 +477,39 @@ get_sysid_from_conn(const char *conninfo)
exit(1);
}
- sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+ primary->sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
- pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+ pg_log_info("system identifier is %llu on publisher",
+ (unsigned long long) primary->sysid);
disconnect_database(conn);
-
- return sysid;
}
/*
- * Obtain the system identifier from control file. It will be used to compare
- * if a data directory is a clone of another one. This routine is used locally
- * and avoids a replication connection.
+ * Obtain the system identifier from a standby server. It will be used to
+ * compare if a data directory is a clone of another one. This routine is used
+ * locally and avoids a replication connection.
*/
-static uint64
-get_control_from_datadir(const char *datadir)
+static void
+get_control_for_standby(StandbyInfo *standby)
{
ControlFileData *cf;
bool crc_ok;
- uint64 sysid;
pg_log_info("getting system identifier from subscriber");
- cf = get_controlfile(datadir, &crc_ok);
+ cf = get_controlfile(standby->pgdata, &crc_ok);
if (!crc_ok)
{
pg_log_error("control file appears to be corrupt");
exit(1);
}
- sysid = cf->system_identifier;
+ standby->sysid = cf->system_identifier;
- pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) standby->sysid);
pfree(cf);
-
- return sysid;
}
/*
@@ -516,7 +518,7 @@ get_control_from_datadir(const char *datadir)
* files from one of the systems might be used in the other one.
*/
static void
-modify_sysid(const char *pg_resetwal_path, const char *datadir)
+modify_sysid(const char *bindir, const char *datadir)
{
ControlFileData *cf;
bool crc_ok;
@@ -551,7 +553,7 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
pg_log_info("running pg_resetwal on the subscriber");
- cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+ cmd_str = psprintf("\"%s/pg_resetwal\" -D \"%s\"", bindir, datadir);
pg_log_debug("command is: %s", cmd_str);
@@ -571,14 +573,15 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
* Return a palloc'd slot name if the replication is using one.
*/
static char *
-use_primary_slot_name(void)
+use_primary_slot_name(PrimaryInfo *primary, StandbyInfo *standby,
+ LogicalRepPerdbInfo *perdb)
{
PGconn *conn;
PGresult *res;
PQExpBuffer str = createPQExpBuffer();
char *slot_name;
- conn = connect_database(dbinfo[0].subconninfo);
+ conn = connect_database(standby->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -604,7 +607,7 @@ use_primary_slot_name(void)
disconnect_database(conn);
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -634,17 +637,19 @@ use_primary_slot_name(void)
}
static bool
-create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
+create_all_logical_replication_slots(PrimaryInfo *primary,
+ LogicalRepPerdbInfoArr *dbarr)
{
int i;
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr->ndbs; i++)
{
PGconn *conn;
PGresult *res;
char replslotname[NAMEDATALEN];
+ LogicalRepPerdbInfo *perdb = &dbarr->perdb[i];
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -664,27 +669,14 @@ create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
}
/* Remember database OID. */
- dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ perdb->oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
PQclear(res);
- /*
- * Build the replication slot name. The name must not exceed
- * NAMEDATALEN - 1. This current schema uses a maximum of 36
- * characters (14 + 10 + 1 + 10 + '\0'). System identifier is included
- * to reduce the probability of collision. By default, subscription
- * name is used as replication slot name.
- */
- snprintf(replslotname, sizeof(replslotname),
- "pg_subscriber_%u_%d",
- dbinfo[i].oid,
- (int) getpid());
- dbinfo[i].subname = pg_strdup(replslotname);
+ get_subscription_name(perdb->oid, (int) getpid(), replslotname, NAMEDATALEN);
/* Create replication slot on publisher. */
- if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
- pg_log_info("create replication slot \"%s\" on publisher", replslotname);
- else
+ if (create_logical_replication_slot(conn, false, perdb) == NULL && !dry_run)
return false;
disconnect_database(conn);
@@ -701,30 +693,36 @@ create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
* result set that contains the consistent LSN.
*/
static char *
-create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
- char *slot_name)
+create_logical_replication_slot(PGconn *conn, bool temporary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res = NULL;
char *lsn = NULL;
- bool transient_replslot = false;
+ char slot_name[NAMEDATALEN];
Assert(conn != NULL);
/*
- * If no slot name is informed, it is a transient replication slot used
- * only for catch up purposes.
+ * Construct a name of logical replication slot. The formatting is
+ * different depends on its persistency.
+ *
+ * For persistent slots: the name must be same as the subscription.
+ * For temporary slots: OID is not needed, but another string is added.
*/
- if (slot_name[0] == '\0')
- {
+ if (!temporary)
+ get_subscription_name(perdb->oid, (int) getpid(), slot_name, NAMEDATALEN);
+ else
snprintf(slot_name, NAMEDATALEN, "pg_subscriber_%d_startpoint",
(int) getpid());
- transient_replslot = true;
- }
- pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, perdb->dbname);
appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
+
+ if(temporary)
+ appendPQExpBufferStr(str, " TEMPORARY");
+
appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
pg_log_debug("command is: %s", str->data);
@@ -734,17 +732,14 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
- pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
- PQresultErrorMessage(res));
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s",
+ slot_name, perdb->dbname, PQresultErrorMessage(res));
return lsn;
}
}
/* for cleanup purposes */
- if (transient_replslot)
- made_transient_replslot = true;
- else
- dbinfo->made_replslot = true;
+ perdb->made_replslot = true;
if (!dry_run)
{
@@ -758,14 +753,15 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
}
static void
-drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+drop_replication_slot(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *slot_name)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
Assert(conn != NULL);
- pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, perdb->dbname);
appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
@@ -775,7 +771,7 @@ drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_nam
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, perdb->dbname,
PQerrorMessage(conn));
PQclear(res);
@@ -825,19 +821,22 @@ pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
* Returns after the server finishes the recovery process.
*/
static void
-wait_for_end_recovery(const char *conninfo)
+wait_for_end_recovery(const char *base_conninfo, const char *dbname)
{
PGconn *conn;
PGresult *res;
- int status = POSTMASTER_STILL_STARTING;
+ int status = PGS_POSTMASTER_STILL_STARTING;
+ int cnt;
+ int rc;
+ char *pg_ctl_cmd;
pg_log_info("waiting the postmaster to reach the consistent state");
- conn = connect_database(conninfo);
+ conn = connect_database(base_conninfo, dbname);
if (conn == NULL)
exit(1);
- for (;;)
+ for (cnt = 0; cnt < wait_seconds * WAITS_PER_SEC; cnt++)
{
bool in_recovery;
@@ -865,17 +864,32 @@ wait_for_end_recovery(const char *conninfo)
*/
if (!in_recovery || dry_run)
{
- status = POSTMASTER_READY;
+ status = PGS_POSTMASTER_READY;
break;
}
/* Keep waiting. */
- pg_usleep(WAIT_INTERVAL * USEC_PER_SEC);
+ pg_usleep(USEC_PER_SEC / WAITS_PER_SEC);
}
disconnect_database(conn);
- if (status == POSTMASTER_STILL_STARTING)
+ /*
+ * If timeout is reached exit the pg_subscriber and stop the standby node.
+ */
+ if (cnt >= wait_seconds * WAITS_PER_SEC)
+ {
+ pg_log_error("recovery timed out");
+
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
+ standby.bindir, standby.pgdata);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+ exit(1);
+ }
+
+ if (status == PGS_POSTMASTER_STILL_STARTING)
{
pg_log_error("server did not end recovery");
exit(1);
@@ -888,17 +902,21 @@ wait_for_end_recovery(const char *conninfo)
* Create a publication that includes all tables in the database.
*/
static void
-create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+create_publication(PGconn *conn, PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
/* Check if the publication needs to be created. */
appendPQExpBuffer(str,
"SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
- dbinfo->pubname);
+ pubname);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
@@ -918,7 +936,7 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
*/
if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
{
- pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ pg_log_info("publication \"%s\" already exists", pubname);
return;
}
else
@@ -931,7 +949,7 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* database oid in which puballtables is false.
*/
pg_log_error("publication \"%s\" does not replicate changes for all tables",
- dbinfo->pubname);
+ pubname);
pg_log_error_hint("Consider renaming this publication.");
PQclear(res);
PQfinish(conn);
@@ -942,9 +960,9 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
PQclear(res);
resetPQExpBuffer(str);
- pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+ pg_log_info("creating publication \"%s\" on database \"%s\"", pubname, perdb->dbname);
- appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", pubname);
pg_log_debug("command is: %s", str->data);
@@ -954,14 +972,14 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
- dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ pubname, perdb->dbname, PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
}
/* for cleanup purposes */
- dbinfo->made_publication = true;
+ perdb->made_publication = true;
if (!dry_run)
PQclear(res);
@@ -973,16 +991,19 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* Remove publication if it couldn't finish all steps.
*/
static void
-drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+drop_publication(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
- appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+ pg_log_info("dropping publication \"%s\" on database \"%s\"", pubname, perdb->dbname);
+
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", pubname);
pg_log_debug("command is: %s", str->data);
@@ -990,7 +1011,7 @@ drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", pubname, perdb->dbname, PQerrorMessage(conn));
PQclear(res);
}
@@ -1011,19 +1032,27 @@ drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* initial location.
*/
static void
-create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+create_subscription(PGconn *conn, StandbyInfo *standby, char *base_conninfo,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", subname,
+ perdb->dbname);
appendPQExpBuffer(str,
"CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
"WITH (create_slot = false, copy_data = false, enabled = false)",
- dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+ subname, concat_conninfo_dbname(base_conninfo, perdb->dbname), pubname);
pg_log_debug("command is: %s", str->data);
@@ -1033,14 +1062,14 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
- dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ subname, perdb->dbname, PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
}
/* for cleanup purposes */
- dbinfo->made_subscription = true;
+ perdb->made_subscription = true;
if (!dry_run)
PQclear(res);
@@ -1052,16 +1081,19 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* Remove subscription if it couldn't finish all steps.
*/
static void
-drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+drop_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"", subname, perdb->dbname);
- appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", subname);
pg_log_debug("command is: %s", str->data);
@@ -1069,7 +1101,7 @@ drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", subname, perdb->dbname, PQerrorMessage(conn));
PQclear(res);
}
@@ -1088,18 +1120,21 @@ drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* printing purposes.
*/
static void
-set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+set_replication_progress(PGconn *conn, LogicalRepPerdbInfo *perdb, const char *lsn)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
Oid suboid;
char originname[NAMEDATALEN];
char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
appendPQExpBuffer(str,
- "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", subname);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
@@ -1140,7 +1175,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
PQclear(res);
pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
- originname, lsnstr, dbinfo->dbname);
+ originname, lsnstr, perdb->dbname);
resetPQExpBuffer(str);
appendPQExpBuffer(str,
@@ -1154,7 +1189,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
pg_log_error("could not set replication progress for the subscription \"%s\": %s",
- dbinfo->subname, PQresultErrorMessage(res));
+ subname, PQresultErrorMessage(res));
PQfinish(conn);
exit(1);
}
@@ -1173,16 +1208,20 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
* of this setup.
*/
static void
-enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+enable_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", subname,
+ perdb->dbname);
- appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", subname);
pg_log_debug("command is: %s", str->data);
@@ -1191,7 +1230,7 @@ enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
- pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ pg_log_error("could not enable subscription \"%s\": %s", subname,
PQerrorMessage(conn));
PQfinish(conn);
exit(1);
@@ -1203,6 +1242,61 @@ enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
destroyPQExpBuffer(str);
}
+static void
+start_standby_server(StandbyInfo *standby, unsigned short subport,
+ char *server_start_log)
+{
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+ int rc;
+ char *pg_ctl_cmd;
+
+ if (server_start_log[0] == '\0')
+ {
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ len = snprintf(server_start_log, MAXPGPATH,
+ "%s/%s/server_start_%s.log", standby->pgdata,
+ PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+ }
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" start -D \"%s\" -s -o \"-p %d\" -l \"%s\"",
+ standby->bindir,
+ standby->pgdata, subport, server_start_log);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+}
+
+static char *
+construct_sub_conninfo(char *username, unsigned short subport)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *ret;
+
+ if (username)
+ appendPQExpBuffer(buf, "user=%s ", username);
+
+ appendPQExpBuffer(buf, "port=%d fallback_application_name=%s",
+ subport, progname);
+
+ ret = pg_strdup(buf->data);
+
+ destroyPQExpBuffer(buf);
+
+ return ret;
+}
+
int
main(int argc, char **argv)
{
@@ -1214,6 +1308,10 @@ main(int argc, char **argv)
{"publisher-conninfo", required_argument, NULL, 'P'},
{"subscriber-conninfo", required_argument, NULL, 'S'},
{"database", required_argument, NULL, 'd'},
+ {"timeout", required_argument, NULL, 't'},
+ {"username", required_argument, NULL, 'u'},
+ {"port", required_argument, NULL, 'p'},
+ {"retain", no_argument, NULL, 'r'},
{"dry-run", no_argument, NULL, 'n'},
{"verbose", no_argument, NULL, 'v'},
{NULL, 0, NULL, 0}
@@ -1225,20 +1323,15 @@ main(int argc, char **argv)
char *pg_ctl_cmd;
- char *base_dir;
- char *server_start_log;
-
- char timebuf[128];
- struct timeval time;
- time_t tt;
+ char base_dir[MAXPGPATH];
+ char server_start_log[MAXPGPATH] = {0};
int len;
- char *pub_base_conninfo = NULL;
- char *sub_base_conninfo = NULL;
char *dbname_conninfo = NULL;
- uint64 pub_sysid;
- uint64 sub_sysid;
+ unsigned short subport = DEF_PGSPORT;
+ char *username = NULL;
+
struct stat statbuf;
PGconn *conn;
@@ -1250,6 +1343,13 @@ main(int argc, char **argv)
int i;
+ PGresult *res;
+
+ char *wal_level;
+ int max_replication_slots;
+ int nslots_old;
+ int nslots_new;
+
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
progname = get_progname(argv[0]);
@@ -1286,28 +1386,40 @@ main(int argc, char **argv)
}
#endif
- while ((c = getopt_long(argc, argv, "D:P:S:d:nv",
+ while ((c = getopt_long(argc, argv, "D:P:S:d:t:u:p:rnv",
long_options, &option_index)) != -1)
{
switch (c)
{
case 'D':
- subscriber_dir = pg_strdup(optarg);
+ standby.pgdata = pg_strdup(optarg);
+ canonicalize_path(standby.pgdata);
break;
case 'P':
pub_conninfo_str = pg_strdup(optarg);
break;
- case 'S':
- sub_conninfo_str = pg_strdup(optarg);
- break;
case 'd':
/* Ignore duplicated database names. */
if (!simple_string_list_member(&database_names, optarg))
{
simple_string_list_append(&database_names, optarg);
- num_dbs++;
+ dbarr.ndbs++;
}
break;
+ case 't':
+ wait_seconds = atoi(optarg);
+ break;
+ case 'u':
+ pfree(username);
+ username = pg_strdup(optarg);
+ break;
+ case 'p':
+ if ((subport = atoi(optarg)) <= 0)
+ pg_fatal("invalid old port number");
+ break;
+ case 'r':
+ retain = true;
+ break;
case 'n':
dry_run = true;
break;
@@ -1335,7 +1447,7 @@ main(int argc, char **argv)
/*
* Required arguments
*/
- if (subscriber_dir == NULL)
+ if (standby.pgdata == NULL)
{
pg_log_error("no subscriber data directory specified");
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1358,21 +1470,14 @@ main(int argc, char **argv)
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
exit(1);
}
- pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
- "publisher");
- if (pub_base_conninfo == NULL)
- exit(1);
- if (sub_conninfo_str == NULL)
- {
- pg_log_error("no subscriber connection string specified");
- pg_log_error_hint("Try \"%s --help\" for more information.", progname);
- exit(1);
- }
- sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
- if (sub_base_conninfo == NULL)
+ primary.base_conninfo = get_base_conninfo(pub_conninfo_str,
+ dbname_conninfo);
+ if (primary.base_conninfo == NULL)
exit(1);
+ standby.base_conninfo = construct_sub_conninfo(username, subport);
+
if (database_names.head == NULL)
{
pg_log_info("no database was specified");
@@ -1385,7 +1490,7 @@ main(int argc, char **argv)
if (dbname_conninfo)
{
simple_string_list_append(&database_names, dbname_conninfo);
- num_dbs++;
+ dbarr.ndbs++;
pg_log_info("database \"%s\" was extracted from the publisher connection string",
dbname_conninfo);
@@ -1399,25 +1504,25 @@ main(int argc, char **argv)
}
/*
- * Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
+ * Get the absolute path of binaries on the subscriber.
*/
- if (!get_exec_path(argv[0]))
+ if (!get_exec_base_path(argv[0]))
exit(1);
/* rudimentary check for a data directory. */
- if (!check_data_directory(subscriber_dir))
+ if (!check_data_directory(standby.pgdata))
exit(1);
- /* Store database information for publisher and subscriber. */
- dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+ /* Store database information to dbarr */
+ store_db_names(&dbarr.perdb, dbarr.ndbs);
/*
* Check if the subscriber data directory has the same system identifier
* than the publisher data directory.
*/
- pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
- sub_sysid = get_control_from_datadir(subscriber_dir);
- if (pub_sysid != sub_sysid)
+ get_sysid_for_primary(&primary, dbarr.perdb[0].dbname);
+ get_control_for_standby(&standby);
+ if (primary.sysid != standby.sysid)
{
pg_log_error("subscriber data directory is not a copy of the source database cluster");
exit(1);
@@ -1426,8 +1531,8 @@ main(int argc, char **argv)
/*
* Create the output directory to store any data generated by this tool.
*/
- base_dir = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s",
+ standby.pgdata, PGS_OUTPUT_DIR);
if (len >= MAXPGPATH)
{
pg_log_error("directory path for subscriber is too long");
@@ -1441,7 +1546,153 @@ main(int argc, char **argv)
}
/* subscriber PID file. */
- snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid",
+ standby.pgdata);
+
+ /* Start the standby server anyway */
+ start_standby_server(&standby, subport, server_start_log);
+
+ /*
+ * Check wal_level in publisher and the max_replication_slots of publisher
+ */
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "SELECT count(*) from pg_replication_slots;");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain number of replication slots");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not determine parameter settings on publisher");
+ exit(1);
+ }
+
+ nslots_old = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ res = PQexec(conn, "SELECT setting FROM pg_settings "
+ "WHERE name IN ('wal_level', 'max_replication_slots') "
+ "ORDER BY name DESC;");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain guc parameters on publisher");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 2)
+ {
+ pg_log_error("could not determine parameter settings on publisher");
+ exit(1);
+ }
+
+ wal_level = PQgetvalue(res, 0, 0);
+
+ if (strcmp(wal_level, "logical") != 0)
+ {
+ pg_log_error("wal_level must be \"logical\", but is set to \"%s\"", wal_level);
+ exit(1);
+ }
+
+ max_replication_slots = atoi(PQgetvalue(res, 1, 0));
+ nslots_new = nslots_old + dbarr.ndbs + 1;
+
+ if (nslots_new > max_replication_slots)
+ {
+ pg_log_error("max_replication_slots (%d) must be greater than or equal to "
+ "the number of replication slots required (%d)", max_replication_slots, nslots_new);
+ exit(1);
+ }
+
+ PQclear(res);
+ disconnect_database(conn);
+
+ conn = connect_database(standby.base_conninfo, dbarr.perdb[0].dbname);
+ if (conn == NULL)
+ exit(1);
+
+ /*
+ * Check the max_replication_slots in subscriber
+ */
+ res = PQexec(conn, "SELECT count(*) from pg_replication_slots;");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain number of replication slots on subscriber");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not determine parameter settings on subscriber");
+ exit(1);
+ }
+
+ nslots_old = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ res = PQexec(conn, "SELECT setting FROM pg_settings "
+ "WHERE name = 'max_replication_slots';");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain guc parameters");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not determine parameter settings on publisher");
+ exit(1);
+ }
+
+ max_replication_slots = atoi(PQgetvalue(res, 0, 0));
+ nslots_new = nslots_old + dbarr.ndbs;
+
+ if (nslots_new > max_replication_slots)
+ {
+ pg_log_error("max_replication_slots (%d) must be greater than or equal to "
+ "the number of replication slots required (%d)", max_replication_slots, nslots_new);
+ exit(1);
+ }
+
+ PQclear(res);
+
+ /*
+ * Exit the pg_subscriber if the node is not a standby server.
+ */
+ res = PQexec(conn, "SELECT pg_catalog.pg_is_in_recovery()");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain recovery progress");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("unexpected result from pg_is_in_recovery function");
+ exit(1);
+ }
+
+ /* Check if the server is in recovery */
+ if (strcmp(PQgetvalue(res, 0, 0), "t") != 0)
+ {
+ pg_log_error("pg_subscriber is supported only on standby server");
+ exit(1);
+ }
+
+ PQclear(res);
+ disconnect_database(conn);
+
+ /* subscriber PID file. */
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", standby.pgdata);
/*
* Stop the subscriber if it is a standby server. Before executing the
@@ -1457,14 +1708,18 @@ main(int argc, char **argv)
* replication slot has no use after the transformation, hence, it
* will be removed at the end of this process.
*/
- primary_slot_name = use_primary_slot_name();
- if (primary_slot_name != NULL)
- pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+ standby.primary_slot_name = use_primary_slot_name(&primary,
+ &standby,
+ &dbarr.perdb[0]);
+ if (standby.primary_slot_name != NULL)
+ pg_log_info("primary has replication slot \"%s\"",
+ standby.primary_slot_name);
pg_log_info("subscriber is up and running");
pg_log_info("stopping the server to start the transformation steps");
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
+ standby.bindir, standby.pgdata);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
}
@@ -1472,7 +1727,7 @@ main(int argc, char **argv)
/*
* Create a replication slot for each database on the publisher.
*/
- if (!create_all_logical_replication_slots(dbinfo))
+ if (!create_all_logical_replication_slots(&primary, &dbarr))
exit(1);
/*
@@ -1492,11 +1747,11 @@ main(int argc, char **argv)
* replication connection open (depending when base backup was taken, the
* connection should be open for a few hours).
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
if (conn == NULL)
exit(1);
- consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
- temp_replslot);
+ consistent_lsn = create_logical_replication_slot(conn, true,
+ &dbarr.perdb[0]);
/*
* Write recovery parameters.
@@ -1522,7 +1777,7 @@ main(int argc, char **argv)
{
appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
consistent_lsn);
- WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ WriteRecoveryConfig(conn, standby.pgdata, recoveryconfcontents);
}
disconnect_database(conn);
@@ -1532,54 +1787,29 @@ main(int argc, char **argv)
* Start subscriber and wait until accepting connections.
*/
pg_log_info("starting the subscriber");
-
- /* append timestamp with ISO 8601 format. */
- gettimeofday(&time, NULL);
- tt = (time_t) time.tv_sec;
- strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
- snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
- ".%03d", (int) (time.tv_usec / 1000));
-
- server_start_log = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(server_start_log, MAXPGPATH, "%s/%s/server_start_%s.log", subscriber_dir, PGS_OUTPUT_DIR, timebuf);
- if (len >= MAXPGPATH)
- {
- pg_log_error("log file path is too long");
- exit(1);
- }
-
- pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, subscriber_dir, server_start_log);
- rc = system(pg_ctl_cmd);
- pg_ctl_status(pg_ctl_cmd, rc, 1);
+ start_standby_server(&standby, subport, server_start_log);
/*
* Waiting the subscriber to be promoted.
*/
- wait_for_end_recovery(dbinfo[0].subconninfo);
+ wait_for_end_recovery(standby.base_conninfo, dbarr.perdb[0].dbname);
/*
* Create a publication for each database. This step should be executed
* after promoting the subscriber to avoid replicating unnecessary
* objects.
*/
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
- char pubname[NAMEDATALEN];
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
/* Connect to publisher. */
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary.base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
- /*
- * Build the publication name. The name must not exceed NAMEDATALEN -
- * 1. This current schema uses a maximum of 35 characters (14 + 10 +
- * '\0').
- */
- snprintf(pubname, sizeof(pubname), "pg_subscriber_%u", dbinfo[i].oid);
- dbinfo[i].pubname = pg_strdup(pubname);
-
- create_publication(conn, &dbinfo[i]);
+ /* Also create a publication */
+ create_publication(conn, &primary, perdb);
disconnect_database(conn);
}
@@ -1587,20 +1817,25 @@ main(int argc, char **argv)
/*
* Create a subscription for each database.
*/
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
+
/* Connect to subscriber. */
- conn = connect_database(dbinfo[i].subconninfo);
+ conn = connect_database(standby.base_conninfo, perdb->dbname);
+
if (conn == NULL)
exit(1);
- create_subscription(conn, &dbinfo[i]);
+ create_subscription(conn, &standby, primary.base_conninfo, perdb);
/* Set the replication progress to the correct LSN. */
- set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+ set_replication_progress(conn, perdb, consistent_lsn);
/* Enable subscription. */
- enable_subscription(conn, &dbinfo[i]);
+ enable_subscription(conn, perdb);
+
+ drop_publication(conn, perdb);
disconnect_database(conn);
}
@@ -1613,19 +1848,21 @@ main(int argc, char **argv)
* XXX we might not fail here. Instead, we provide a warning so the user
* eventually drops the replication slot later.
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
if (conn == NULL)
{
- pg_log_warning("could not drop transient replication slot \"%s\" on publisher", temp_replslot);
- pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
+ char *primary_slot_name = standby.primary_slot_name;
+
if (primary_slot_name != NULL)
pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
}
else
{
- drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[0];
+ char *primary_slot_name = standby.primary_slot_name;
+
if (primary_slot_name != NULL)
- drop_replication_slot(conn, &dbinfo[0], primary_slot_name);
+ drop_replication_slot(conn, perdb, primary_slot_name);
disconnect_database(conn);
}
@@ -1634,20 +1871,22 @@ main(int argc, char **argv)
*/
pg_log_info("stopping the subscriber");
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
+ standby.bindir, standby.pgdata);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
/*
* Change system identifier.
*/
- modify_sysid(pg_resetwal_path, subscriber_dir);
+ modify_sysid(standby.bindir, standby.pgdata);
/*
* Remove log file generated by this tool, if it runs successfully.
* Otherwise, file is kept that may provide useful debugging information.
*/
- unlink(server_start_log);
+ if (!retain)
+ unlink(server_start_log);
success = true;
diff --git a/src/bin/pg_basebackup/t/040_pg_subscriber.pl b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
index 4ebff76b2d..9915b8cb3c 100644
--- a/src/bin/pg_basebackup/t/040_pg_subscriber.pl
+++ b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
@@ -37,8 +37,13 @@ command_fails(
'--verbose',
'--pgdata', $datadir,
'--publisher-conninfo', 'dbname=postgres',
- '--subscriber-conninfo', 'dbname=postgres'
],
'no database name specified');
-
+command_fails(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres',
+ ],
+ 'subscriber connection string specnfied non-local server');
done_testing();
diff --git a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
index fbcd0fc82b..4e26607611 100644
--- a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
+++ b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
@@ -51,25 +51,27 @@ $node_s->start;
$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('second row')");
$node_p->wait_for_replay_catchup($node_s);
+$node_f->stop;
+
# Run pg_subscriber on about-to-fail node F
command_fails(
[
'pg_subscriber', '--verbose',
'--pgdata', $node_f->data_dir,
'--publisher-conninfo', $node_p->connstr('pg1'),
- '--subscriber-conninfo', $node_f->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
],
'subscriber data directory is not a copy of the source database cluster');
+$node_s->stop;
+
# dry run mode on node S
command_ok(
[
'pg_subscriber', '--verbose', '--dry-run',
'--pgdata', $node_s->data_dir,
'--publisher-conninfo', $node_p->connstr('pg1'),
- '--subscriber-conninfo', $node_s->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
],
@@ -82,6 +84,7 @@ $node_s->start;
# Check if node S is still a standby
is($node_s->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
't', 'standby is in recovery');
+$node_s->stop;
# Run pg_subscriber on node S
command_ok(
@@ -89,7 +92,6 @@ command_ok(
'pg_subscriber', '--verbose',
'--pgdata', $node_s->data_dir,
'--publisher-conninfo', $node_p->connstr('pg1'),
- '--subscriber-conninfo', $node_s->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
],
--
2.43.0
[application/octet-stream] v7-0003-Fix-publication-does-not-exist-error.patch (1.6K, ../../TY3PR01MB98890AB22AECA2F503EB7858F5742@TY3PR01MB9889.jpnprd01.prod.outlook.com/4-v7-0003-Fix-publication-does-not-exist-error.patch)
download | inline diff:
From fe1c57b974a2228b5ab2349b31de16d04db24aac Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Mon, 22 Jan 2024 12:36:20 +0530
Subject: [PATCH v7 3/3] Fix publication does not exist error.
Fix publication does not exist error.
---
src/bin/pg_basebackup/pg_subscriber.c | 23 +++--------------------
1 file changed, 3 insertions(+), 20 deletions(-)
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index 3880d15ef9..355738c20c 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -679,6 +679,9 @@ create_all_logical_replication_slots(PrimaryInfo *primary,
if (create_logical_replication_slot(conn, false, perdb) == NULL && !dry_run)
return false;
+ /* Also create a publication */
+ create_publication(conn, primary, perdb);
+
disconnect_database(conn);
}
@@ -1794,26 +1797,6 @@ main(int argc, char **argv)
*/
wait_for_end_recovery(standby.base_conninfo, dbarr.perdb[0].dbname);
- /*
- * Create a publication for each database. This step should be executed
- * after promoting the subscriber to avoid replicating unnecessary
- * objects.
- */
- for (i = 0; i < dbarr.ndbs; i++)
- {
- LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
-
- /* Connect to publisher. */
- conn = connect_database(primary.base_conninfo, perdb->dbname);
- if (conn == NULL)
- exit(1);
-
- /* Also create a publication */
- create_publication(conn, &primary, perdb);
-
- disconnect_database(conn);
- }
-
/*
* Create a subscription for each database.
*/
--
2.43.0
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-23 02:11 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
@ 2024-01-24 06:11 ` Shubham Khanna <[email protected]>
2024-01-24 11:43 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Shubham Khanna @ 2024-01-24 06:11 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: Shlok Kyal <[email protected]>; [email protected] <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Euler Taveira <[email protected]>
On Tue, Jan 23, 2024 at 7:41 AM Hayato Kuroda (Fujitsu)
<[email protected]> wrote:
>
> Dear hackers,
>
> > We fixed some of the comments posted in the thread. We have created it
> > as top-up patch 0002 and 0003.
>
> I found that the CFbot raised an ERROR. Also, it may not work well in case
> of production build.
> PSA the fixed patch set.
Segmentation fault was found after testing the given command(There is
an extra '/' between 'new_standby2' and '-P') '$ gdb --args
./pg_subscriber -D ../new_standby2 / -P "host=localhost
port=5432 dbname=postgres" -d postgres'
While executing the above command, I got the following error:
pg_subscriber: error: too many command-line arguments (first is "/")
pg_subscriber: hint: Try "pg_subscriber --help" for more information.
Program received signal SIGSEGV, Segmentation fault.
0x0000555555557e5b in cleanup_objects_atexit () at pg_subscriber.c:173
173 if (perdb->made_subscription)
(gdb) p perdb
$1 = (LogicalRepPerdbInfo *) 0x0
Thanks and Regards,
Shubham Khanna.
^ permalink raw reply [nested|flat] 77+ messages in thread
* RE: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-23 02:11 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-24 06:11 ` Re: speed up a logical replica setup Shubham Khanna <[email protected]>
@ 2024-01-24 11:43 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-25 09:05 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-24 11:43 UTC (permalink / raw)
To: 'Shubham Khanna' <[email protected]>; [email protected] <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Euler Taveira <[email protected]>
Dear Shubham,
> Segmentation fault was found after testing the given command(There is
> an extra '/' between 'new_standby2' and '-P') '$ gdb --args
> ./pg_subscriber -D ../new_standby2 / -P "host=localhost
> port=5432 dbname=postgres" -d postgres'
> While executing the above command, I got the following error:
> pg_subscriber: error: too many command-line arguments (first is "/")
> pg_subscriber: hint: Try "pg_subscriber --help" for more information.
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x0000555555557e5b in cleanup_objects_atexit () at pg_subscriber.c:173
> 173 if (perdb->made_subscription)
> (gdb) p perdb
> $1 = (LogicalRepPerdbInfo *) 0x0
>
Good catch, I could reproduce the issue. This crash was occurred because the
cleanup function was called before initialization memory.
There are several ways to fix it, but I chose to move the callback registration
behind. The function does actual tasks only after database objects are created.
So 0004 registers the function just before doing them. The memory allocation has
been done at that time. If required, Assert() can be added in the callback.
Can you test it and confirm the issue was solved?
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
[application/octet-stream] v8-0001-Creates-a-new-logical-replica-from-a-standby-serv.patch (67.0K, ../../TY3PR01MB9889678E47B918F4D83A6FD8F57B2@TY3PR01MB9889.jpnprd01.prod.outlook.com/2-v8-0001-Creates-a-new-logical-replica-from-a-standby-serv.patch)
download | inline diff:
From 26bc1ee9371409e360588ac6aacafaf4fafb5e96 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email protected]>
Date: Mon, 5 Jun 2023 14:39:40 -0400
Subject: [PATCH v8 1/4] Creates a new logical replica from a standby server
A new tool called pg_subscriber can convert a physical replica into a
logical replica. It runs on the target server and should be able to
connect to the source server (publisher) and the target server
(subscriber).
The conversion requires a few steps. Check if the target data directory
has the same system identifier than the source data directory. Stop the
target server if it is running as a standby server. Create one
replication slot per specified database on the source server. One
additional replication slot is created at the end to get the consistent
LSN (This consistent LSN will be used as (a) a stopping point for the
recovery process and (b) a starting point for the subscriptions). Write
recovery parameters into the target data directory and start the target
server (Wait until the target server is promoted). Create one
publication (FOR ALL TABLES) per specified database on the source
server. Create one subscription per specified database on the target
server (Use replication slot and publication created in a previous step.
Don't enable the subscriptions yet). Sets the replication progress to
the consistent LSN that was got in a previous step. Enable the
subscription for each specified database on the target server. Remove
the additional replication slot that was used to get the consistent LSN.
Stop the target server. Change the system identifier from the target
server.
Depending on your workload and database size, creating a logical replica
couldn't be an option due to resource constraints (WAL backlog should be
available until all table data is synchronized). The initial data copy
and the replication progress tends to be faster on a physical replica.
The purpose of this tool is to speed up a logical replica setup.
---
doc/src/sgml/ref/allfiles.sgml | 1 +
doc/src/sgml/ref/pg_subscriber.sgml | 284 +++
doc/src/sgml/reference.sgml | 1 +
src/bin/pg_basebackup/.gitignore | 1 +
src/bin/pg_basebackup/Makefile | 8 +-
src/bin/pg_basebackup/meson.build | 19 +
src/bin/pg_basebackup/pg_subscriber.c | 1657 +++++++++++++++++
src/bin/pg_basebackup/t/040_pg_subscriber.pl | 44 +
.../t/041_pg_subscriber_standby.pl | 139 ++
9 files changed, 2153 insertions(+), 1 deletion(-)
create mode 100644 doc/src/sgml/ref/pg_subscriber.sgml
create mode 100644 src/bin/pg_basebackup/pg_subscriber.c
create mode 100644 src/bin/pg_basebackup/t/040_pg_subscriber.pl
create mode 100644 src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index 4a42999b18..3862c976d7 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -214,6 +214,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY pgResetwal SYSTEM "pg_resetwal.sgml">
<!ENTITY pgRestore SYSTEM "pg_restore.sgml">
<!ENTITY pgRewind SYSTEM "pg_rewind.sgml">
+<!ENTITY pgSubscriber SYSTEM "pg_subscriber.sgml">
<!ENTITY pgVerifyBackup SYSTEM "pg_verifybackup.sgml">
<!ENTITY pgtestfsync SYSTEM "pgtestfsync.sgml">
<!ENTITY pgtesttiming SYSTEM "pgtesttiming.sgml">
diff --git a/doc/src/sgml/ref/pg_subscriber.sgml b/doc/src/sgml/ref/pg_subscriber.sgml
new file mode 100644
index 0000000000..553185c35f
--- /dev/null
+++ b/doc/src/sgml/ref/pg_subscriber.sgml
@@ -0,0 +1,284 @@
+<!--
+doc/src/sgml/ref/pg_subscriber.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="app-pgsubscriber">
+ <indexterm zone="app-pgsubscriber">
+ <primary>pg_subscriber</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle><application>pg_subscriber</application></refentrytitle>
+ <manvolnum>1</manvolnum>
+ <refmiscinfo>Application</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>pg_subscriber</refname>
+ <refpurpose>create a new logical replica from a standby server</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_subscriber</command>
+ <arg rep="repeat"><replaceable>option</replaceable></arg>
+ </cmdsynopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+ <para>
+ <application>pg_subscriber</application> takes the publisher and subscriber
+ connection strings, a cluster directory from a standby server and a list of
+ database names and it sets up a new logical replica using the physical
+ recovery process.
+ </para>
+
+ <para>
+ The <application>pg_subscriber</application> should be run at the target
+ server. The source server (known as publisher server) should accept logical
+ replication connections from the target server (known as subscriber server).
+ The target server should accept local logical replication connection.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Options</title>
+
+ <para>
+ <application>pg_subscriber</application> accepts the following
+ command-line arguments:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-D <replaceable class="parameter">directory</replaceable></option></term>
+ <term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term>
+ <listitem>
+ <para>
+ The target directory that contains a cluster directory from a standby
+ server.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-P <replaceable class="parameter">conninfo</replaceable></option></term>
+ <term><option>--publisher-conninfo=<replaceable class="parameter">conninfo</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the publisher. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-S <replaceable class="parameter">conninfo</replaceable></option></term>
+ <term><option>--subscriber-conninfo=<replaceable class="parameter">conninfo</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the subscriber. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-d <replaceable class="parameter">dbname</replaceable></option></term>
+ <term><option>--database=<replaceable class="parameter">dbname</replaceable></option></term>
+ <listitem>
+ <para>
+ The database name to create the subscription. Multiple databases can be
+ selected by writing multiple <option>-d</option> switches.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
+ <listitem>
+ <para>
+ Do everything except actually modifying the target directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-v</option></term>
+ <term><option>--verbose</option></term>
+ <listitem>
+ <para>
+ Enables verbose mode. This will cause
+ <application>pg_subscriber</application> to output progress messages
+ and detailed information about each step.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ Other options are also available:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-V</option></term>
+ <term><option>--version</option></term>
+ <listitem>
+ <para>
+ Print the <application>pg_subscriber</application> version and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-?</option></term>
+ <term><option>--help</option></term>
+ <listitem>
+ <para>
+ Show help about <application>pg_subscriber</application> command
+ line arguments, and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Notes</title>
+
+ <para>
+ The transformation proceeds in the following steps:
+ </para>
+
+ <procedure>
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the given target data
+ directory has the same system identifier than the source data directory.
+ Since it uses the recovery process as one of the steps, it starts the
+ target server as a replica from the source server. If the system
+ identifier is not the same, <application>pg_subscriber</application> will
+ terminate with an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the target data
+ directory is used by a standby server. Stop the standby server if it is
+ running. One of the next steps is to add some recovery parameters that
+ requires a server start. This step avoids an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one replication slot for
+ each specified database on the source server. The replication slot name
+ contains a <literal>pg_subscriber</literal> prefix. These replication
+ slots will be used by the subscriptions in a future step. Another
+ replication slot is used to get a consistent start location. This
+ consistent LSN will be used as a stopping point in the <xref
+ linkend="guc-recovery-target-lsn"/> parameter and by the
+ subscriptions as a replication starting point. It guarantees that no
+ transaction will be lost.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> writes recovery parameters into
+ the target data directory and start the target server. It specifies a LSN
+ (consistent LSN that was obtained in the previous step) of write-ahead
+ log location up to which recovery will proceed. It also specifies
+ <literal>promote</literal> as the action that the server should take once
+ the recovery target is reached. This step finishes once the server ends
+ standby mode and is accepting read-write operations.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Next, <application>pg_subscriber</application> creates one publication
+ for each specified database on the source server. Each publication
+ replicates changes for all tables in the database. The publication name
+ contains a <literal>pg_subscriber</literal> prefix. These publication
+ will be used by a corresponding subscription in a next step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one subscription for
+ each specified database on the target server. Each subscription name
+ contains a <literal>pg_subscriber</literal> prefix. The replication slot
+ name is identical to the subscription name. It does not copy existing data
+ from the source server. It does not create a replication slot. Instead, it
+ uses the replication slot that was created in a previous step. The
+ subscription is created but it is not enabled yet. The reason is the
+ replication progress must be set to the consistent LSN but replication
+ origin name contains the subscription oid in its name. Hence, the
+ subscription will be enabled in a separate step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> sets the replication progress to
+ the consistent LSN that was obtained in a previous step. When the target
+ server started the recovery process, it caught up to the consistent LSN.
+ This is the exact LSN to be used as a initial location for each
+ subscription.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Finally, <application>pg_subscriber</application> enables the subscription
+ for each specified database on the target server. The subscription starts
+ streaming from the consistent LSN.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> removes the additional replication
+ slot that was used to get the consistent LSN on the source server.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> stops the target server to change
+ its system identifier.
+ </para>
+ </step>
+ </procedure>
+ </refsect1>
+
+ <refsect1>
+ <title>Examples</title>
+
+ <para>
+ To create a logical replica for databases <literal>hr</literal> and
+ <literal>finance</literal> from a standby server at <literal>foo</literal>:
+<screen>
+<prompt>$</prompt> <userinput>pg_subscriber -D /usr/local/pgsql/data -P "host=foo" -S "host=localhost" -d hr -d finance</userinput>
+</screen>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="app-pgbasebackup"/></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index aa94f6adf6..266f4e515a 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -285,6 +285,7 @@
&pgCtl;
&pgResetwal;
&pgRewind;
+ &pgSubscriber;
&pgtestfsync;
&pgtesttiming;
&pgupgrade;
diff --git a/src/bin/pg_basebackup/.gitignore b/src/bin/pg_basebackup/.gitignore
index 26048bdbd8..0e5384a1d5 100644
--- a/src/bin/pg_basebackup/.gitignore
+++ b/src/bin/pg_basebackup/.gitignore
@@ -1,5 +1,6 @@
/pg_basebackup
/pg_receivewal
/pg_recvlogical
+/pg_subscriber
/tmp_check/
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index abfb6440ec..f6281b7676 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -44,7 +44,7 @@ BBOBJS = \
bbstreamer_tar.o \
bbstreamer_zstd.o
-all: pg_basebackup pg_receivewal pg_recvlogical
+all: pg_basebackup pg_receivewal pg_recvlogical pg_subscriber
pg_basebackup: $(BBOBJS) $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) $(BBOBJS) $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
@@ -55,10 +55,14 @@ pg_receivewal: pg_receivewal.o $(OBJS) | submake-libpq submake-libpgport submake
pg_recvlogical: pg_recvlogical.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_recvlogical.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_subscriber: $(WIN32RES) pg_subscriber.o | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
install: all installdirs
$(INSTALL_PROGRAM) pg_basebackup$(X) '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
$(INSTALL_PROGRAM) pg_receivewal$(X) '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
$(INSTALL_PROGRAM) pg_recvlogical$(X) '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ $(INSTALL_PROGRAM) pg_subscriber$(X) '$(DESTDIR)$(bindir)/pg_subscriber$(X)'
installdirs:
$(MKDIR_P) '$(DESTDIR)$(bindir)'
@@ -67,10 +71,12 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ rm -f '$(DESTDIR)$(bindir)/pg_subscriber$(X)'
clean distclean:
rm -f pg_basebackup$(X) pg_receivewal$(X) pg_recvlogical$(X) \
$(BBOBJS) pg_receivewal.o pg_recvlogical.o \
+ pg_subscriber$(X) pg_subscriber.o \
$(OBJS)
rm -rf tmp_check
diff --git a/src/bin/pg_basebackup/meson.build b/src/bin/pg_basebackup/meson.build
index f7e60e6670..ccfd7bb7a5 100644
--- a/src/bin/pg_basebackup/meson.build
+++ b/src/bin/pg_basebackup/meson.build
@@ -75,6 +75,23 @@ pg_recvlogical = executable('pg_recvlogical',
)
bin_targets += pg_recvlogical
+pg_subscriber_sources = files(
+ 'pg_subscriber.c'
+)
+
+if host_system == 'windows'
+ pg_subscriber_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'pg_subscriber',
+ '--FILEDESC', 'pg_subscriber - create a new logical replica from a standby server',])
+endif
+
+pg_subscriber = executable('pg_subscriber',
+ pg_subscriber_sources,
+ dependencies: [frontend_code, libpq],
+ kwargs: default_bin_args,
+)
+bin_targets += pg_subscriber
+
tests += {
'name': 'pg_basebackup',
'sd': meson.current_source_dir(),
@@ -89,6 +106,8 @@ tests += {
't/011_in_place_tablespace.pl',
't/020_pg_receivewal.pl',
't/030_pg_recvlogical.pl',
+ 't/040_pg_subscriber.pl',
+ 't/041_pg_subscriber_standby.pl',
],
},
}
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
new file mode 100644
index 0000000000..e998c29f9e
--- /dev/null
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -0,0 +1,1657 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_subscriber.c
+ * Create a new logical replica from a standby server
+ *
+ * Copyright (C) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_subscriber/pg_subscriber.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include <signal.h>
+#include <sys/stat.h>
+#include <sys/time.h>
+#include <sys/wait.h>
+#include <time.h>
+
+#include "access/xlogdefs.h"
+#include "catalog/pg_control.h"
+#include "common/connect.h"
+#include "common/controldata_utils.h"
+#include "common/file_perm.h"
+#include "common/file_utils.h"
+#include "common/logging.h"
+#include "fe_utils/recovery_gen.h"
+#include "fe_utils/simple_list.h"
+#include "getopt_long.h"
+#include "utils/pidfile.h"
+
+#define PGS_OUTPUT_DIR "pg_subscriber_output.d"
+
+typedef struct LogicalRepInfo
+{
+ Oid oid; /* database OID */
+ char *dbname; /* database name */
+ char *pubconninfo; /* publication connection string for logical
+ * replication */
+ char *subconninfo; /* subscription connection string for logical
+ * replication */
+ char *pubname; /* publication name */
+ char *subname; /* subscription name (also replication slot
+ * name) */
+
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepInfo;
+
+static void cleanup_objects_atexit(void);
+static void usage();
+static char *get_base_conninfo(char *conninfo, char *dbname,
+ const char *noderole);
+static bool get_exec_path(const char *path);
+static bool check_data_directory(const char *datadir);
+static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
+static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
+static PGconn *connect_database(const char *conninfo);
+static void disconnect_database(PGconn *conn);
+static uint64 get_sysid_from_conn(const char *conninfo);
+static uint64 get_control_from_datadir(const char *datadir);
+static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
+static char *use_primary_slot_name(void);
+static bool create_all_logical_replication_slots(LogicalRepInfo *dbinfo);
+static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name);
+static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
+static void wait_for_end_recovery(const char *conninfo);
+static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+
+#define USEC_PER_SEC 1000000
+#define WAIT_INTERVAL 1 /* 1 second */
+
+/* Options */
+static const char *progname;
+
+static char *subscriber_dir = NULL;
+static char *pub_conninfo_str = NULL;
+static char *sub_conninfo_str = NULL;
+static SimpleStringList database_names = {NULL, NULL};
+static char *primary_slot_name = NULL;
+static bool dry_run = false;
+
+static bool success = false;
+
+static char *pg_ctl_path = NULL;
+static char *pg_resetwal_path = NULL;
+
+static LogicalRepInfo *dbinfo;
+static int num_dbs = 0;
+
+static char temp_replslot[NAMEDATALEN] = {0};
+static bool made_transient_replslot = false;
+
+enum WaitPMResult
+{
+ POSTMASTER_READY,
+ POSTMASTER_STANDBY,
+ POSTMASTER_STILL_STARTING,
+ POSTMASTER_FAILED
+};
+
+
+/*
+ * Cleanup objects that were created by pg_subscriber if there is an error.
+ *
+ * Replication slots, publications and subscriptions are created. Depending on
+ * the step it failed, it should remove the already created objects if it is
+ * possible (sometimes it won't work due to a connection issue).
+ */
+static void
+cleanup_objects_atexit(void)
+{
+ PGconn *conn;
+ int i;
+
+ if (success)
+ return;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ if (dbinfo[i].made_subscription)
+ {
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn != NULL)
+ {
+ drop_subscription(conn, &dbinfo[i]);
+ disconnect_database(conn);
+ }
+ }
+
+ if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ {
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn != NULL)
+ {
+ if (dbinfo[i].made_publication)
+ drop_publication(conn, &dbinfo[i]);
+ if (dbinfo[i].made_replslot)
+ drop_replication_slot(conn, &dbinfo[i], NULL);
+ disconnect_database(conn);
+ }
+ }
+ }
+
+ if (made_transient_replslot)
+ {
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn != NULL)
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ disconnect_database(conn);
+ }
+ }
+}
+
+static void
+usage(void)
+{
+ printf(_("%s creates a new logical replica from a standby server.\n\n"),
+ progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION]...\n"), progname);
+ printf(_("\nOptions:\n"));
+ printf(_(" -D, --pgdata=DATADIR location for the subscriber data directory\n"));
+ printf(_(" -P, --publisher-conninfo=CONNINFO publisher connection string\n"));
+ printf(_(" -S, --subscriber-conninfo=CONNINFO subscriber connection string\n"));
+ printf(_(" -d, --database=DBNAME database to create a subscription\n"));
+ printf(_(" -n, --dry-run stop before modifying anything\n"));
+ printf(_(" -v, --verbose output verbose messages\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
+ printf(_("\nReport bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+ printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
+
+/*
+ * Validate a connection string. Returns a base connection string that is a
+ * connection string without a database name plus a fallback application name.
+ * Since we might process multiple databases, each database name will be
+ * appended to this base connection string to provide a final connection string.
+ * If the second argument (dbname) is not null, returns dbname if the provided
+ * connection string contains it. If option --database is not provided, uses
+ * dbname as the only database to setup the logical replica.
+ * It is the caller's responsibility to free the returned connection string and
+ * dbname.
+ */
+static char *
+get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ PQconninfoOption *conn_opts = NULL;
+ PQconninfoOption *conn_opt;
+ char *errmsg = NULL;
+ char *ret;
+ int i;
+
+ pg_log_info("validating connection string on %s", noderole);
+
+ conn_opts = PQconninfoParse(conninfo, &errmsg);
+ if (conn_opts == NULL)
+ {
+ pg_log_error("could not parse connection string: %s", errmsg);
+ return NULL;
+ }
+
+ i = 0;
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (strcmp(conn_opt->keyword, "dbname") == 0 && conn_opt->val != NULL)
+ {
+ if (dbname)
+ dbname = pg_strdup(conn_opt->val);
+ continue;
+ }
+
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0')
+ {
+ if (i > 0)
+ appendPQExpBufferChar(buf, ' ');
+ appendPQExpBuffer(buf, "%s=%s", conn_opt->keyword, conn_opt->val);
+ i++;
+ }
+ }
+
+ if (i > 0)
+ appendPQExpBufferChar(buf, ' ');
+ appendPQExpBuffer(buf, "fallback_application_name=%s", progname);
+
+ ret = pg_strdup(buf->data);
+
+ destroyPQExpBuffer(buf);
+ PQconninfoFree(conn_opts);
+
+ return ret;
+}
+
+/*
+ * Get the absolute path from other PostgreSQL binaries (pg_ctl and
+ * pg_resetwal) that is used by it.
+ */
+static bool
+get_exec_path(const char *path)
+{
+ int rc;
+
+ pg_ctl_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_ctl",
+ "pg_ctl (PostgreSQL) " PG_VERSION "\n",
+ pg_ctl_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_ctl", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_ctl", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
+
+ pg_resetwal_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_resetwal",
+ "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
+ pg_resetwal_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_resetwal", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_resetwal", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+
+ return true;
+}
+
+/*
+ * Is it a cluster directory? These are preliminary checks. It is far from
+ * making an accurate check. If it is not a clone from the publisher, it will
+ * eventually fail in a future step.
+ */
+static bool
+check_data_directory(const char *datadir)
+{
+ struct stat statbuf;
+ char versionfile[MAXPGPATH];
+
+ pg_log_info("checking if directory \"%s\" is a cluster data directory",
+ datadir);
+
+ if (stat(datadir, &statbuf) != 0)
+ {
+ if (errno == ENOENT)
+ pg_log_error("data directory \"%s\" does not exist", datadir);
+ else
+ pg_log_error("could not access directory \"%s\": %s", datadir, strerror(errno));
+
+ return false;
+ }
+
+ snprintf(versionfile, MAXPGPATH, "%s/PG_VERSION", datadir);
+ if (stat(versionfile, &statbuf) != 0 && errno == ENOENT)
+ {
+ pg_log_error("directory \"%s\" is not a database cluster directory", datadir);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Append database name into a base connection string.
+ *
+ * dbname is the only parameter that changes so it is not included in the base
+ * connection string. This function concatenates dbname to build a "real"
+ * connection string.
+ */
+static char *
+concat_conninfo_dbname(const char *conninfo, const char *dbname)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *ret;
+
+ Assert(conninfo != NULL);
+
+ appendPQExpBufferStr(buf, conninfo);
+ appendPQExpBuffer(buf, " dbname=%s", dbname);
+
+ ret = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+
+ return ret;
+}
+
+/*
+ * Store publication and subscription information.
+ */
+static LogicalRepInfo *
+store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+{
+ LogicalRepInfo *dbinfo;
+ SimpleStringListCell *cell;
+ int i = 0;
+
+ dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+
+ for (cell = database_names.head; cell; cell = cell->next)
+ {
+ char *conninfo;
+
+ /* Publisher. */
+ conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
+ dbinfo[i].pubconninfo = conninfo;
+ dbinfo[i].dbname = cell->val;
+ dbinfo[i].made_replslot = false;
+ dbinfo[i].made_publication = false;
+ dbinfo[i].made_subscription = false;
+ /* other struct fields will be filled later. */
+
+ /* Subscriber. */
+ conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
+ dbinfo[i].subconninfo = conninfo;
+
+ i++;
+ }
+
+ return dbinfo;
+}
+
+static PGconn *
+connect_database(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ const char *rconninfo;
+
+ /* logical replication mode */
+ rconninfo = psprintf("%s replication=database", conninfo);
+
+ conn = PQconnectdb(rconninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ pg_log_error("connection to database failed: %s", PQerrorMessage(conn));
+ return NULL;
+ }
+
+ /* secure search_path */
+ res = PQexec(conn, ALWAYS_SECURE_SEARCH_PATH_SQL);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not clear search_path: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+ PQclear(res);
+
+ return conn;
+}
+
+static void
+disconnect_database(PGconn *conn)
+{
+ Assert(conn != NULL);
+
+ PQfinish(conn);
+}
+
+/*
+ * Obtain the system identifier using the provided connection. It will be used
+ * to compare if a data directory is a clone of another one.
+ */
+static uint64
+get_sysid_from_conn(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from publisher");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "IDENTIFY_SYSTEM");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not send replication command \"%s\": %s",
+ "IDENTIFY_SYSTEM", PQresultErrorMessage(res));
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+ if (PQntuples(res) != 1 || PQnfields(res) < 3)
+ {
+ pg_log_error("could not identify system: got %d rows and %d fields, expected %d rows and %d or more fields",
+ PQntuples(res), PQnfields(res), 1, 3);
+
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+
+ sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+
+ pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+
+ disconnect_database(conn);
+
+ return sysid;
+}
+
+/*
+ * Obtain the system identifier from control file. It will be used to compare
+ * if a data directory is a clone of another one. This routine is used locally
+ * and avoids a replication connection.
+ */
+static uint64
+get_control_from_datadir(const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ sysid = cf->system_identifier;
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+
+ pfree(cf);
+
+ return sysid;
+}
+
+/*
+ * Modify the system identifier. Since a standby server preserves the system
+ * identifier, it makes sense to change it to avoid situations in which WAL
+ * files from one of the systems might be used in the other one.
+ */
+static void
+modify_sysid(const char *pg_resetwal_path, const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ struct timeval tv;
+
+ char *cmd_str;
+ int rc;
+
+ pg_log_info("modifying system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ /*
+ * Select a new system identifier.
+ *
+ * XXX this code was extracted from BootStrapXLOG().
+ */
+ gettimeofday(&tv, NULL);
+ cf->system_identifier = ((uint64) tv.tv_sec) << 32;
+ cf->system_identifier |= ((uint64) tv.tv_usec) << 12;
+ cf->system_identifier |= getpid() & 0xFFF;
+
+ if (!dry_run)
+ update_controlfile(datadir, cf, true);
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) cf->system_identifier);
+
+ pg_log_info("running pg_resetwal on the subscriber");
+
+ cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+
+ pg_log_debug("command is: %s", cmd_str);
+
+ if (!dry_run)
+ {
+ rc = system(cmd_str);
+ if (rc == 0)
+ pg_log_info("subscriber successfully changed the system identifier");
+ else
+ pg_log_error("subscriber failed to change system identifier: exit code: %d", rc);
+ }
+
+ pfree(cf);
+}
+
+/*
+ * Return a palloc'd slot name if the replication is using one.
+ */
+static char *
+use_primary_slot_name(void)
+{
+ PGconn *conn;
+ PGresult *res;
+ PQExpBuffer str = createPQExpBuffer();
+ char *slot_name;
+
+ conn = connect_database(dbinfo[0].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "SELECT setting FROM pg_settings WHERE name = 'primary_slot_name'");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain parameter information: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+
+ /*
+ * If primary_slot_name is an empty string, the current replication
+ * connection is not using a replication slot, bail out.
+ */
+ if (strcmp(PQgetvalue(res, 0, 0), "") == 0)
+ {
+ PQclear(res);
+ return NULL;
+ }
+
+ slot_name = pg_strdup(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ disconnect_database(conn);
+
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ appendPQExpBuffer(str,
+ "SELECT 1 FROM pg_replication_slots r INNER JOIN pg_stat_activity a ON (r.active_pid = a.pid) WHERE slot_name = '%s'", slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain replication slot information: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
+ PQntuples(res), 1);
+ return NULL;
+ }
+
+ PQclear(res);
+ disconnect_database(conn);
+
+ return slot_name;
+}
+
+static bool
+create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
+{
+ int i;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ PGconn *conn;
+ PGresult *res;
+ char replslotname[NAMEDATALEN];
+
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT oid FROM pg_catalog.pg_database WHERE datname = current_database()");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain database OID: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain database OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ return false;
+ }
+
+ /* Remember database OID. */
+ dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+
+ PQclear(res);
+
+ /*
+ * Build the replication slot name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 36
+ * characters (14 + 10 + 1 + 10 + '\0'). System identifier is included
+ * to reduce the probability of collision. By default, subscription
+ * name is used as replication slot name.
+ */
+ snprintf(replslotname, sizeof(replslotname),
+ "pg_subscriber_%u_%d",
+ dbinfo[i].oid,
+ (int) getpid());
+ dbinfo[i].subname = pg_strdup(replslotname);
+
+ /* Create replication slot on publisher. */
+ if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
+ pg_log_info("create replication slot \"%s\" on publisher", replslotname);
+ else
+ return false;
+
+ disconnect_database(conn);
+ }
+
+ return true;
+}
+
+/*
+ * Create a logical replication slot and returns a consistent LSN. The returned
+ * LSN might be used to catch up the subscriber up to the required point.
+ *
+ * CreateReplicationSlot() is not used because it does not provide the one-row
+ * result set that contains the consistent LSN.
+ */
+static char *
+create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res = NULL;
+ char *lsn = NULL;
+ bool transient_replslot = false;
+
+ Assert(conn != NULL);
+
+ /*
+ * If no slot name is informed, it is a transient replication slot used
+ * only for catch up purposes.
+ */
+ if (slot_name[0] == '\0')
+ {
+ snprintf(slot_name, NAMEDATALEN, "pg_subscriber_%d_startpoint",
+ (int) getpid());
+ transient_replslot = true;
+ }
+
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
+ appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQresultErrorMessage(res));
+ return lsn;
+ }
+ }
+
+ /* for cleanup purposes */
+ if (transient_replslot)
+ made_transient_replslot = true;
+ else
+ dbinfo->made_replslot = true;
+
+ if (!dry_run)
+ {
+ lsn = pg_strdup(PQgetvalue(res, 0, 1));
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+
+ return lsn;
+}
+
+static void
+drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Reports a suitable message if pg_ctl fails.
+ */
+static void
+pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
+{
+ if (rc != 0)
+ {
+ if (WIFEXITED(rc))
+ {
+ pg_log_error("pg_ctl failed with exit code %d", WEXITSTATUS(rc));
+ }
+ else if (WIFSIGNALED(rc))
+ {
+#if defined(WIN32)
+ pg_log_error("pg_ctl was terminated by exception 0x%X", WTERMSIG(rc));
+ pg_log_error_detail("See C include file \"ntstatus.h\" for a description of the hexadecimal value.");
+#else
+ pg_log_error("pg_ctl was terminated by signal %d: %s",
+ WTERMSIG(rc), pg_strsignal(WTERMSIG(rc)));
+#endif
+ }
+ else
+ {
+ pg_log_error("pg_ctl exited with unrecognized status %d", rc);
+ }
+
+ pg_log_error_detail("The failed command was: %s", pg_ctl_cmd);
+ exit(1);
+ }
+
+ if (action)
+ pg_log_info("postmaster was started");
+ else
+ pg_log_info("postmaster was stopped");
+}
+
+/*
+ * Returns after the server finishes the recovery process.
+ */
+static void
+wait_for_end_recovery(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ int status = POSTMASTER_STILL_STARTING;
+
+ pg_log_info("waiting the postmaster to reach the consistent state");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ for (;;)
+ {
+ bool in_recovery;
+
+ res = PQexec(conn, "SELECT pg_catalog.pg_is_in_recovery()");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain recovery progress");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("unexpected result from pg_is_in_recovery function");
+ exit(1);
+ }
+
+ in_recovery = (strcmp(PQgetvalue(res, 0, 0), "t") == 0);
+
+ PQclear(res);
+
+ /*
+ * Does the recovery process finish? In dry run mode, there is no
+ * recovery mode. Bail out as the recovery process has ended.
+ */
+ if (!in_recovery || dry_run)
+ {
+ status = POSTMASTER_READY;
+ break;
+ }
+
+ /* Keep waiting. */
+ pg_usleep(WAIT_INTERVAL * USEC_PER_SEC);
+ }
+
+ disconnect_database(conn);
+
+ if (status == POSTMASTER_STILL_STARTING)
+ {
+ pg_log_error("server did not end recovery");
+ exit(1);
+ }
+
+ pg_log_info("postmaster reached the consistent state");
+}
+
+/*
+ * Create a publication that includes all tables in the database.
+ */
+static void
+create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ /* Check if the publication needs to be created. */
+ appendPQExpBuffer(str,
+ "SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
+ dbinfo->pubname);
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publication information: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) == 1)
+ {
+ /*
+ * If publication name already exists and puballtables is true, let's
+ * use it. A previous run of pg_subscriber must have created this
+ * publication. Bail out.
+ */
+ if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
+ {
+ pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ return;
+ }
+ else
+ {
+ /*
+ * Unfortunately, if it reaches this code path, it will always
+ * fail (unless you decide to change the existing publication
+ * name). That's bad but it is very unlikely that the user will
+ * choose a name with pg_subscriber_ prefix followed by the exact
+ * database oid in which puballtables is false.
+ */
+ pg_log_error("publication \"%s\" does not replicate changes for all tables",
+ dbinfo->pubname);
+ pg_log_error_hint("Consider renaming this publication.");
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(str);
+
+ pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
+ dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_publication = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove publication if it couldn't finish all steps.
+ */
+static void
+drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Create a subscription with some predefined options.
+ *
+ * A replication slot was already created in a previous step. Let's use it. By
+ * default, the subscription name is used as replication slot name. It is
+ * not required to copy data. The subscription will be created but it will not
+ * be enabled now. That's because the replication progress must be set and the
+ * replication origin name (one of the function arguments) contains the
+ * subscription OID in its name. Once the subscription is created,
+ * set_replication_progress() can obtain the chosen origin name and set up its
+ * initial location.
+ */
+static void
+create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str,
+ "CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
+ "WITH (create_slot = false, copy_data = false, enabled = false)",
+ dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
+ dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_subscription = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove subscription if it couldn't finish all steps.
+ */
+static void
+drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Sets the replication progress to the consistent LSN.
+ *
+ * The subscriber caught up to the consistent LSN provided by the temporary
+ * replication slot. The goal is to set up the initial location for the logical
+ * replication that is the exact LSN that the subscriber was promoted. Once the
+ * subscription is enabled it will start streaming from that location onwards.
+ * In dry run mode, the subscription OID and LSN are set to invalid values for
+ * printing purposes.
+ */
+static void
+set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+ Oid suboid;
+ char originname[NAMEDATALEN];
+ char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+
+ Assert(conn != NULL);
+
+ appendPQExpBuffer(str,
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscription OID: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1 && !dry_run)
+ {
+ pg_log_error("could not obtain subscription OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (dry_run)
+ {
+ suboid = InvalidOid;
+ snprintf(lsnstr, sizeof(lsnstr), "%X/%X", LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ suboid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ snprintf(lsnstr, sizeof(lsnstr), "%s", lsn);
+ }
+
+ /*
+ * The origin name is defined as pg_%u. %u is the subscription OID. See
+ * ApplyWorkerMain().
+ */
+ snprintf(originname, sizeof(originname), "pg_%u", suboid);
+
+ PQclear(res);
+
+ pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
+ originname, lsnstr, dbinfo->dbname);
+
+ resetPQExpBuffer(str);
+ appendPQExpBuffer(str,
+ "SELECT pg_catalog.pg_replication_origin_advance('%s', '%s')", originname, lsnstr);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not set replication progress for the subscription \"%s\": %s",
+ dbinfo->subname, PQresultErrorMessage(res));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Enables the subscription.
+ *
+ * The subscription was created in a previous step but it was disabled. After
+ * adjusting the initial location, enabling the subscription is the last step
+ * of this setup.
+ */
+static void
+enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+int
+main(int argc, char **argv)
+{
+ static struct option long_options[] =
+ {
+ {"help", no_argument, NULL, '?'},
+ {"version", no_argument, NULL, 'V'},
+ {"pgdata", required_argument, NULL, 'D'},
+ {"publisher-conninfo", required_argument, NULL, 'P'},
+ {"subscriber-conninfo", required_argument, NULL, 'S'},
+ {"database", required_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"verbose", no_argument, NULL, 'v'},
+ {NULL, 0, NULL, 0}
+ };
+
+ int c;
+ int option_index;
+ int rc;
+
+ char *pg_ctl_cmd;
+
+ char *base_dir;
+ char *server_start_log;
+
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+
+ char *pub_base_conninfo = NULL;
+ char *sub_base_conninfo = NULL;
+ char *dbname_conninfo = NULL;
+
+ uint64 pub_sysid;
+ uint64 sub_sysid;
+ struct stat statbuf;
+
+ PGconn *conn;
+ char *consistent_lsn;
+
+ PQExpBuffer recoveryconfcontents = NULL;
+
+ char pidfile[MAXPGPATH];
+
+ int i;
+
+ pg_logging_init(argv[0]);
+ pg_logging_set_level(PG_LOG_WARNING);
+ progname = get_progname(argv[0]);
+ set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_subscriber"));
+
+ if (argc > 1)
+ {
+ if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+ {
+ usage();
+ exit(0);
+ }
+ else if (strcmp(argv[1], "-V") == 0
+ || strcmp(argv[1], "--version") == 0)
+ {
+ puts("pg_subscriber (PostgreSQL) " PG_VERSION);
+ exit(0);
+ }
+ }
+
+ atexit(cleanup_objects_atexit);
+
+ /*
+ * Don't allow it to be run as root. It uses pg_ctl which does not allow
+ * it either.
+ */
+#ifndef WIN32
+ if (geteuid() == 0)
+ {
+ pg_log_error("cannot be executed by \"root\"");
+ pg_log_error_hint("You must run %s as the PostgreSQL superuser.",
+ progname);
+ exit(1);
+ }
+#endif
+
+ while ((c = getopt_long(argc, argv, "D:P:S:d:nv",
+ long_options, &option_index)) != -1)
+ {
+ switch (c)
+ {
+ case 'D':
+ subscriber_dir = pg_strdup(optarg);
+ break;
+ case 'P':
+ pub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'S':
+ sub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'd':
+ /* Ignore duplicated database names. */
+ if (!simple_string_list_member(&database_names, optarg))
+ {
+ simple_string_list_append(&database_names, optarg);
+ num_dbs++;
+ }
+ break;
+ case 'n':
+ dry_run = true;
+ break;
+ case 'v':
+ pg_logging_increase_verbosity();
+ break;
+ default:
+ /* getopt_long already emitted a complaint */
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Any non-option arguments?
+ */
+ if (optind < argc)
+ {
+ pg_log_error("too many command-line arguments (first is \"%s\")",
+ argv[optind]);
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Required arguments
+ */
+ if (subscriber_dir == NULL)
+ {
+ pg_log_error("no subscriber data directory specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Parse connection string. Build a base connection string that might be
+ * reused by multiple databases.
+ */
+ if (pub_conninfo_str == NULL)
+ {
+ /*
+ * TODO use primary_conninfo (if available) from subscriber and
+ * extract publisher connection string. Assume that there are
+ * identical entries for physical and logical replication. If there is
+ * not, we would fail anyway.
+ */
+ pg_log_error("no publisher connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
+ "publisher");
+ if (pub_base_conninfo == NULL)
+ exit(1);
+
+ if (sub_conninfo_str == NULL)
+ {
+ pg_log_error("no subscriber connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
+ if (sub_base_conninfo == NULL)
+ exit(1);
+
+ if (database_names.head == NULL)
+ {
+ pg_log_info("no database was specified");
+
+ /*
+ * If --database option is not provided, try to obtain the dbname from
+ * the publisher conninfo. If dbname parameter is not available, error
+ * out.
+ */
+ if (dbname_conninfo)
+ {
+ simple_string_list_append(&database_names, dbname_conninfo);
+ num_dbs++;
+
+ pg_log_info("database \"%s\" was extracted from the publisher connection string",
+ dbname_conninfo);
+ }
+ else
+ {
+ pg_log_error("no database name specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
+ */
+ if (!get_exec_path(argv[0]))
+ exit(1);
+
+ /* rudimentary check for a data directory. */
+ if (!check_data_directory(subscriber_dir))
+ exit(1);
+
+ /* Store database information for publisher and subscriber. */
+ dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+
+ /*
+ * Check if the subscriber data directory has the same system identifier
+ * than the publisher data directory.
+ */
+ pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
+ sub_sysid = get_control_from_datadir(subscriber_dir);
+ if (pub_sysid != sub_sysid)
+ {
+ pg_log_error("subscriber data directory is not a copy of the source database cluster");
+ exit(1);
+ }
+
+ /*
+ * Create the output directory to store any data generated by this tool.
+ */
+ base_dir = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("directory path for subscriber is too long");
+ exit(1);
+ }
+
+ if (mkdir(base_dir, pg_dir_create_mode) < 0 && errno != EEXIST)
+ {
+ pg_log_error("could not create directory \"%s\": %m", base_dir);
+ exit(1);
+ }
+
+ /* subscriber PID file. */
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+
+ /*
+ * Stop the subscriber if it is a standby server. Before executing the
+ * transformation steps, make sure the subscriber is not running because
+ * one of the steps is to modify some recovery parameters that require a
+ * restart.
+ */
+ if (stat(pidfile, &statbuf) == 0)
+ {
+ /*
+ * Since the standby server is running, check if it is using an
+ * existing replication slot for WAL retention purposes. This
+ * replication slot has no use after the transformation, hence, it
+ * will be removed at the end of this process.
+ */
+ primary_slot_name = use_primary_slot_name();
+ if (primary_slot_name != NULL)
+ pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+
+ pg_log_info("subscriber is up and running");
+ pg_log_info("stopping the server to start the transformation steps");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+ }
+
+ /*
+ * Create a replication slot for each database on the publisher.
+ */
+ if (!create_all_logical_replication_slots(dbinfo))
+ exit(1);
+
+ /*
+ * Create a logical replication slot to get a consistent LSN.
+ *
+ * This consistent LSN will be used later to advanced the recently created
+ * replication slots. We cannot use the last created replication slot
+ * because the consistent LSN should be obtained *after* the base backup
+ * finishes (and the base backup should include the logical replication
+ * slots).
+ *
+ * XXX we should probably use the last created replication slot to get a
+ * consistent LSN but it should be changed after adding pg_basebackup
+ * support.
+ *
+ * A temporary replication slot is not used here to avoid keeping a
+ * replication connection open (depending when base backup was taken, the
+ * connection should be open for a few hours).
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+ consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
+ temp_replslot);
+
+ /*
+ * Write recovery parameters.
+ *
+ * Despite of the recovery parameters will be written to the subscriber,
+ * use a publisher connection for the follwing recovery functions. The
+ * connection is only used to check the current server version (physical
+ * replica, same server version). The subscriber is not running yet. In
+ * dry run mode, the recovery parameters *won't* be written. An invalid
+ * LSN is used for printing purposes.
+ */
+ recoveryconfcontents = GenerateRecoveryConfig(conn, NULL);
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_inclusive = true\n");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_action = promote\n");
+
+ if (dry_run)
+ {
+ appendPQExpBuffer(recoveryconfcontents, "# dry run mode");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%X/%X'\n",
+ LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
+ consistent_lsn);
+ WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ }
+ disconnect_database(conn);
+
+ pg_log_debug("recovery parameters:\n%s", recoveryconfcontents->data);
+
+ /*
+ * Start subscriber and wait until accepting connections.
+ */
+ pg_log_info("starting the subscriber");
+
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ server_start_log = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(server_start_log, MAXPGPATH, "%s/%s/server_start_%s.log", subscriber_dir, PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+
+ pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, subscriber_dir, server_start_log);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+
+ /*
+ * Waiting the subscriber to be promoted.
+ */
+ wait_for_end_recovery(dbinfo[0].subconninfo);
+
+ /*
+ * Create a publication for each database. This step should be executed
+ * after promoting the subscriber to avoid replicating unnecessary
+ * objects.
+ */
+ for (i = 0; i < num_dbs; i++)
+ {
+ char pubname[NAMEDATALEN];
+
+ /* Connect to publisher. */
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ /*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 35 characters (14 + 10 +
+ * '\0').
+ */
+ snprintf(pubname, sizeof(pubname), "pg_subscriber_%u", dbinfo[i].oid);
+ dbinfo[i].pubname = pg_strdup(pubname);
+
+ create_publication(conn, &dbinfo[i]);
+
+ disconnect_database(conn);
+ }
+
+ /*
+ * Create a subscription for each database.
+ */
+ for (i = 0; i < num_dbs; i++)
+ {
+ /* Connect to subscriber. */
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ create_subscription(conn, &dbinfo[i]);
+
+ /* Set the replication progress to the correct LSN. */
+ set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+
+ /* Enable subscription. */
+ enable_subscription(conn, &dbinfo[i]);
+
+ disconnect_database(conn);
+ }
+
+ /*
+ * The transient replication slot is no longer required. Drop it.
+ *
+ * If the physical replication slot exists, drop it.
+ *
+ * XXX we might not fail here. Instead, we provide a warning so the user
+ * eventually drops the replication slot later.
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ {
+ pg_log_warning("could not drop transient replication slot \"%s\" on publisher", temp_replslot);
+ pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
+ if (primary_slot_name != NULL)
+ pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
+ }
+ else
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ if (primary_slot_name != NULL)
+ drop_replication_slot(conn, &dbinfo[0], primary_slot_name);
+ disconnect_database(conn);
+ }
+
+ /*
+ * Stop the subscriber.
+ */
+ pg_log_info("stopping the subscriber");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+ /*
+ * Change system identifier.
+ */
+ modify_sysid(pg_resetwal_path, subscriber_dir);
+
+ /*
+ * Remove log file generated by this tool, if it runs successfully.
+ * Otherwise, file is kept that may provide useful debugging information.
+ */
+ unlink(server_start_log);
+
+ success = true;
+
+ pg_log_info("Done!");
+
+ return 0;
+}
diff --git a/src/bin/pg_basebackup/t/040_pg_subscriber.pl b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
new file mode 100644
index 0000000000..4ebff76b2d
--- /dev/null
+++ b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
@@ -0,0 +1,44 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test checking options of pg_subscriber.
+#
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+program_help_ok('pg_subscriber');
+program_version_ok('pg_subscriber');
+program_options_handling_ok('pg_subscriber');
+
+my $datadir = PostgreSQL::Test::Utils::tempdir;
+
+command_fails(['pg_subscriber'],
+ 'no subscriber data directory specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--pgdata', $datadir
+ ],
+ 'no publisher connection string specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--dry-run',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres'
+ ],
+ 'no subscriber connection string specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--verbose',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres',
+ '--subscriber-conninfo', 'dbname=postgres'
+ ],
+ 'no database name specified');
+
+done_testing();
diff --git a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
new file mode 100644
index 0000000000..fbcd0fc82b
--- /dev/null
+++ b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
@@ -0,0 +1,139 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test using a standby server as the subscriber.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node_p;
+my $node_f;
+my $node_s;
+my $result;
+
+# Set up node P as primary
+$node_p = PostgreSQL::Test::Cluster->new('node_p');
+$node_p->init(allows_streaming => 'logical');
+$node_p->start;
+
+# Set up node F as about-to-fail node
+# The extra option forces it to initialize a new cluster instead of copying a
+# previously initdb's cluster.
+$node_f = PostgreSQL::Test::Cluster->new('node_f');
+$node_f->init(allows_streaming => 'logical', extra => [ '--no-instructions' ]);
+$node_f->start;
+
+# On node P
+# - create databases
+# - create test tables
+# - insert a row
+$node_p->safe_psql(
+ 'postgres', q(
+ CREATE DATABASE pg1;
+ CREATE DATABASE pg2;
+));
+$node_p->safe_psql('pg1', 'CREATE TABLE tbl1 (a text)');
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('first row')");
+$node_p->safe_psql('pg2', 'CREATE TABLE tbl2 (a text)');
+
+# Set up node S as standby linking to node P
+$node_p->backup('backup_1');
+$node_s = PostgreSQL::Test::Cluster->new('node_s');
+$node_s->init_from_backup($node_p, 'backup_1', has_streaming => 1);
+$node_s->append_conf('postgresql.conf', 'log_min_messages = debug2');
+$node_s->set_standby_mode();
+$node_s->start;
+
+# Insert another row on node P and wait node S to catch up
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('second row')");
+$node_p->wait_for_replay_catchup($node_s);
+
+# Run pg_subscriber on about-to-fail node F
+command_fails(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $node_f->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_f->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'subscriber data directory is not a copy of the source database cluster');
+
+# dry run mode on node S
+command_ok(
+ [
+ 'pg_subscriber', '--verbose', '--dry-run',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_subscriber --dry-run on node S');
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+# Check if node S is still a standby
+is($node_s->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
+ 't', 'standby is in recovery');
+
+# Run pg_subscriber on node S
+command_ok(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_subscriber on node S');
+
+# Insert rows on P
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('third row')");
+$node_p->safe_psql('pg2', "INSERT INTO tbl2 VALUES('row 1')");
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+
+# Get subscription names
+$result = $node_s->safe_psql(
+ 'postgres', qq(
+ SELECT subname FROM pg_subscription WHERE subname ~ '^pg_subscriber_'
+));
+my @subnames = split("\n", $result);
+
+# Wait subscriber to catch up
+$node_s->wait_for_subscription_sync($node_p, $subnames[0]);
+$node_s->wait_for_subscription_sync($node_p, $subnames[1]);
+
+# Check result on database pg1
+$result = $node_s->safe_psql('pg1', 'SELECT * FROM tbl1');
+is( $result, qq(first row
+second row
+third row),
+ 'logical replication works on database pg1');
+
+# Check result on database pg2
+$result = $node_s->safe_psql('pg2', 'SELECT * FROM tbl2');
+is( $result, qq(row 1),
+ 'logical replication works on database pg2');
+
+# Different system identifier?
+my $sysid_p = $node_p->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+my $sysid_s = $node_s->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+ok($sysid_p != $sysid_s, 'system identifier was changed');
+
+# clean up
+$node_p->teardown_node;
+$node_s->teardown_node;
+
+done_testing();
--
2.43.0
[application/octet-stream] v8-0002-Address-some-comments-proposed-on-hackers.patch (56.5K, ../../TY3PR01MB9889678E47B918F4D83A6FD8F57B2@TY3PR01MB9889.jpnprd01.prod.outlook.com/3-v8-0002-Address-some-comments-proposed-on-hackers.patch)
download | inline diff:
From 327fa75f88f913b4731e73b066e1d30b9225ed44 Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Mon, 22 Jan 2024 12:42:34 +0530
Subject: [PATCH v8 2/4] Address some comments proposed on -hackers
The patch has following changes:
* Some comments reported on the thread
* Add a timeout option for the recovery option
* Reject if the target server is not a standby
* Reject when the --subscriber-conninfo specifies non-local server
* Add -u and -p options
* Check wal_level and max_replication_slot parameters
---
doc/src/sgml/ref/pg_subscriber.sgml | 21 +-
src/bin/pg_basebackup/pg_subscriber.c | 911 +++++++++++-------
src/bin/pg_basebackup/t/040_pg_subscriber.pl | 9 +-
.../t/041_pg_subscriber_standby.pl | 8 +-
4 files changed, 601 insertions(+), 348 deletions(-)
diff --git a/doc/src/sgml/ref/pg_subscriber.sgml b/doc/src/sgml/ref/pg_subscriber.sgml
index 553185c35f..eaabfc7053 100644
--- a/doc/src/sgml/ref/pg_subscriber.sgml
+++ b/doc/src/sgml/ref/pg_subscriber.sgml
@@ -16,12 +16,18 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_subscriber</refname>
- <refpurpose>create a new logical replica from a standby server</refpurpose>
+ <refpurpose>Convert a standby replica to a logical replica</refpurpose>
</refnamediv>
<refsynopsisdiv>
<cmdsynopsis>
<command>pg_subscriber</command>
+ <arg choice="plain"><option>-D</option></arg>
+ <arg choice="plain"><replaceable>datadir</replaceable></arg>
+ <arg choice="plain"><option>-P</option>
+ <replaceable>publisher-conninfo</replaceable></arg>
+ <arg choice="plain"><option>-S</option></arg>
+ <arg choice="plain"><replaceable>subscriber-conninfo</replaceable></arg>
<arg rep="repeat"><replaceable>option</replaceable></arg>
</cmdsynopsis>
</refsynopsisdiv>
@@ -29,17 +35,18 @@ PostgreSQL documentation
<refsect1>
<title>Description</title>
<para>
- <application>pg_subscriber</application> takes the publisher and subscriber
- connection strings, a cluster directory from a standby server and a list of
- database names and it sets up a new logical replica using the physical
- recovery process.
+ pg_subscriber creates a new <link
+ linkend="logical-replication-subscription">subscriber</link> from a physical
+ standby server. This allows users to quickly set up logical replication
+ system.
</para>
<para>
- The <application>pg_subscriber</application> should be run at the target
+ The <application>pg_subscriber</application> has to be run at the target
server. The source server (known as publisher server) should accept logical
replication connections from the target server (known as subscriber server).
- The target server should accept local logical replication connection.
+ The target server should accept logical replication connection from
+ localhost.
</para>
</refsect1>
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index e998c29f9e..3880d15ef9 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -1,12 +1,12 @@
/*-------------------------------------------------------------------------
*
* pg_subscriber.c
- * Create a new logical replica from a standby server
+ * Convert a standby replica to a logical replica
*
* Copyright (C) 2024, PostgreSQL Global Development Group
*
* IDENTIFICATION
- * src/bin/pg_subscriber/pg_subscriber.c
+ * src/bin/pg_basebackup/pg_subscriber.c
*
*-------------------------------------------------------------------------
*/
@@ -32,81 +32,122 @@
#define PGS_OUTPUT_DIR "pg_subscriber_output.d"
-typedef struct LogicalRepInfo
+typedef struct LogicalRepPerdbInfo
{
- Oid oid; /* database OID */
- char *dbname; /* database name */
- char *pubconninfo; /* publication connection string for logical
- * replication */
- char *subconninfo; /* subscription connection string for logical
- * replication */
- char *pubname; /* publication name */
- char *subname; /* subscription name (also replication slot
- * name) */
-
- bool made_replslot; /* replication slot was created */
- bool made_publication; /* publication was created */
- bool made_subscription; /* subscription was created */
-} LogicalRepInfo;
+ Oid oid;
+ char *dbname;
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepPerdbInfo;
+
+typedef struct
+{
+ LogicalRepPerdbInfo *perdb; /* array of db infos */
+ int ndbs; /* number of db infos */
+} LogicalRepPerdbInfoArr;
+
+typedef struct PrimaryInfo
+{
+ char *base_conninfo;
+ uint64 sysid;
+} PrimaryInfo;
+
+typedef struct StandbyInfo
+{
+ char *base_conninfo;
+ char *bindir;
+ char *pgdata;
+ char *primary_slot_name;
+ uint64 sysid;
+} StandbyInfo;
static void cleanup_objects_atexit(void);
static void usage();
-static char *get_base_conninfo(char *conninfo, char *dbname,
- const char *noderole);
-static bool get_exec_path(const char *path);
+static char *get_base_conninfo(char *conninfo, char *dbname);
+static bool get_exec_base_path(const char *path);
static bool check_data_directory(const char *datadir);
+static void store_db_names(LogicalRepPerdbInfo **perdb, int ndbs);
+static void get_sysid_for_primary(PrimaryInfo *primary, char *dbname);
+static void get_control_for_standby(StandbyInfo *standby);
static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
-static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
-static PGconn *connect_database(const char *conninfo);
+static PGconn *connect_database(const char *base_conninfo, const char*dbname);
static void disconnect_database(PGconn *conn);
-static uint64 get_sysid_from_conn(const char *conninfo);
-static uint64 get_control_from_datadir(const char *datadir);
-static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
-static char *use_primary_slot_name(void);
-static bool create_all_logical_replication_slots(LogicalRepInfo *dbinfo);
-static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
- char *slot_name);
-static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static char *use_primary_slot_name(PrimaryInfo *primary, StandbyInfo *standby,
+ LogicalRepPerdbInfo *perdb);
+static bool create_all_logical_replication_slots(PrimaryInfo *primary,
+ LogicalRepPerdbInfoArr *dbarr);
+static char *create_logical_replication_slot(PGconn *conn, bool temporary,
+ LogicalRepPerdbInfo *perdb);
+static void modify_sysid(const char *bindir, const char *datadir);
+static void drop_replication_slot(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *slot_name);
static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
-static void wait_for_end_recovery(const char *conninfo);
-static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
-static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
-static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
-static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void wait_for_end_recovery(const char *base_conninfo,
+ const char *dbname);
+static void create_publication(PGconn *conn, PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb);
+static void drop_publication(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void create_subscription(PGconn *conn, StandbyInfo *standby,
+ char *base_conninfo,
+ LogicalRepPerdbInfo *perdb);
+static void drop_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void set_replication_progress(PGconn *conn, LogicalRepPerdbInfo *perdb, const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void start_standby_server(StandbyInfo *standby, unsigned short subport,
+ char *server_start_log);
+static char *construct_sub_conninfo(char *username, unsigned short subport);
#define USEC_PER_SEC 1000000
-#define WAIT_INTERVAL 1 /* 1 second */
+#define DEFAULT_WAIT 60
+#define WAITS_PER_SEC 10 /* should divide USEC_PER_SEC evenly */
+#define DEF_PGSPORT 50111
/* Options */
-static const char *progname;
-
-static char *subscriber_dir = NULL;
static char *pub_conninfo_str = NULL;
-static char *sub_conninfo_str = NULL;
static SimpleStringList database_names = {NULL, NULL};
-static char *primary_slot_name = NULL;
+static int wait_seconds = DEFAULT_WAIT;
+static bool retain = false;
static bool dry_run = false;
static bool success = false;
+static const char *progname;
+static LogicalRepPerdbInfoArr dbarr;
+static PrimaryInfo primary;
+static StandbyInfo standby;
-static char *pg_ctl_path = NULL;
-static char *pg_resetwal_path = NULL;
+enum PGSWaitPMResult
+{
+ PGS_POSTMASTER_READY,
+ PGS_POSTMASTER_STANDBY,
+ PGS_POSTMASTER_STILL_STARTING,
+ PGS_POSTMASTER_FAILED
+};
-static LogicalRepInfo *dbinfo;
-static int num_dbs = 0;
-static char temp_replslot[NAMEDATALEN] = {0};
-static bool made_transient_replslot = false;
+/*
+ * Build the replication slot and subscription name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 36 characters
+ * (14 + 10 + 1 + 10 + '\0'). System identifier is included to reduce the
+ * probability of collision. By default, subscription name is used as
+ * replication slot name.
+ */
+static inline void
+get_subscription_name(Oid oid, int pid, char *subname, Size szsub)
+{
+ snprintf(subname, szsub, "pg_subscriber_%u_%d", oid, pid);
+}
-enum WaitPMResult
+/*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 35 characters (14 + 10 +
+ * '\0').
+ */
+static inline void
+get_publication_name(Oid oid, char *pubname, Size szpub)
{
- POSTMASTER_READY,
- POSTMASTER_STANDBY,
- POSTMASTER_STILL_STARTING,
- POSTMASTER_FAILED
-};
+ snprintf(pubname, szpub, "pg_subscriber_%u", oid);
+}
/*
@@ -125,41 +166,39 @@ cleanup_objects_atexit(void)
if (success)
return;
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
- if (dbinfo[i].made_subscription)
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
+
+ if (perdb->made_subscription)
{
- conn = connect_database(dbinfo[i].subconninfo);
+ conn = connect_database(standby.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- drop_subscription(conn, &dbinfo[i]);
+ drop_subscription(conn, perdb);
disconnect_database(conn);
}
}
- if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ if (perdb->made_publication || perdb->made_replslot)
{
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- if (dbinfo[i].made_publication)
- drop_publication(conn, &dbinfo[i]);
- if (dbinfo[i].made_replslot)
- drop_replication_slot(conn, &dbinfo[i], NULL);
+ if (perdb->made_publication)
+ drop_publication(conn, perdb);
+ if (perdb->made_replslot)
+ {
+ char replslotname[NAMEDATALEN];
+
+ get_subscription_name(perdb->oid, (int) getpid(),
+ replslotname, NAMEDATALEN);
+ drop_replication_slot(conn, perdb, replslotname);
+ }
disconnect_database(conn);
}
}
}
-
- if (made_transient_replslot)
- {
- conn = connect_database(dbinfo[0].pubconninfo);
- if (conn != NULL)
- {
- drop_replication_slot(conn, &dbinfo[0], temp_replslot);
- disconnect_database(conn);
- }
- }
}
static void
@@ -184,17 +223,16 @@ usage(void)
/*
* Validate a connection string. Returns a base connection string that is a
- * connection string without a database name plus a fallback application name.
- * Since we might process multiple databases, each database name will be
- * appended to this base connection string to provide a final connection string.
- * If the second argument (dbname) is not null, returns dbname if the provided
- * connection string contains it. If option --database is not provided, uses
- * dbname as the only database to setup the logical replica.
- * It is the caller's responsibility to free the returned connection string and
- * dbname.
+ * connection string without a database name. Since we might process multiple
+ * databases, each database name will be appended to this base connection
+ * string to provide a final connection string. If the second argument (dbname)
+ * is not null, returns dbname if the provided connection string contains it.
+ * If option --database is not provided, uses dbname as the only database to
+ * setup the logical replica. It is the caller's responsibility to free the
+ * returned connection string and dbname.
*/
static char *
-get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+get_base_conninfo(char *conninfo, char *dbname)
{
PQExpBuffer buf = createPQExpBuffer();
PQconninfoOption *conn_opts = NULL;
@@ -203,7 +241,7 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
char *ret;
int i;
- pg_log_info("validating connection string on %s", noderole);
+ pg_log_info("validating connection string on publisher");
conn_opts = PQconninfoParse(conninfo, &errmsg);
if (conn_opts == NULL)
@@ -231,10 +269,6 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
}
}
- if (i > 0)
- appendPQExpBufferChar(buf, ' ');
- appendPQExpBuffer(buf, "fallback_application_name=%s", progname);
-
ret = pg_strdup(buf->data);
destroyPQExpBuffer(buf);
@@ -244,15 +278,16 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
}
/*
- * Get the absolute path from other PostgreSQL binaries (pg_ctl and
- * pg_resetwal) that is used by it.
+ * Get the absolute binary path from another PostgreSQL binary (pg_ctl) and set
+ * to StandbyInfo.
*/
static bool
-get_exec_path(const char *path)
+get_exec_base_path(const char *path)
{
int rc;
+ char pg_ctl_path[MAXPGPATH];
+ char *p;
- pg_ctl_path = pg_malloc(MAXPGPATH);
rc = find_other_exec(path, "pg_ctl",
"pg_ctl (PostgreSQL) " PG_VERSION "\n",
pg_ctl_path);
@@ -277,30 +312,12 @@ get_exec_path(const char *path)
pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
- pg_resetwal_path = pg_malloc(MAXPGPATH);
- rc = find_other_exec(path, "pg_resetwal",
- "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
- pg_resetwal_path);
- if (rc < 0)
- {
- char full_path[MAXPGPATH];
-
- if (find_my_exec(path, full_path) < 0)
- strlcpy(full_path, progname, sizeof(full_path));
- if (rc == -1)
- pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
- "same directory as \"%s\".\n"
- "Check your installation.",
- "pg_resetwal", progname, full_path);
- else
- pg_log_error("The program \"%s\" was found by \"%s\"\n"
- "but was not the same version as %s.\n"
- "Check your installation.",
- "pg_resetwal", full_path, progname);
- return false;
- }
+ /* Extract the directory part from the path */
+ p = strrchr(pg_ctl_path, 'p');
+ Assert(p);
- pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+ *p = '\0';
+ standby.bindir = pg_strdup(pg_ctl_path);
return true;
}
@@ -364,49 +381,36 @@ concat_conninfo_dbname(const char *conninfo, const char *dbname)
}
/*
- * Store publication and subscription information.
+ * Initialize per-db structure and store the name of databases
*/
-static LogicalRepInfo *
-store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+static void
+store_db_names(LogicalRepPerdbInfo **perdb, int ndbs)
{
- LogicalRepInfo *dbinfo;
SimpleStringListCell *cell;
int i = 0;
- dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+ *perdb = (LogicalRepPerdbInfo *) pg_malloc0(sizeof(LogicalRepPerdbInfo) *
+ ndbs);
for (cell = database_names.head; cell; cell = cell->next)
{
- char *conninfo;
-
- /* Publisher. */
- conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
- dbinfo[i].pubconninfo = conninfo;
- dbinfo[i].dbname = cell->val;
- dbinfo[i].made_replslot = false;
- dbinfo[i].made_publication = false;
- dbinfo[i].made_subscription = false;
- /* other struct fields will be filled later. */
-
- /* Subscriber. */
- conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
- dbinfo[i].subconninfo = conninfo;
-
+ (*perdb)[i].dbname = pg_strdup(cell->val);
i++;
}
-
- return dbinfo;
}
static PGconn *
-connect_database(const char *conninfo)
+connect_database(const char *base_conninfo, const char*dbname)
{
PGconn *conn;
PGresult *res;
- const char *rconninfo;
+
+ char *rconninfo;
+ char *concat_conninfo = concat_conninfo_dbname(base_conninfo,
+ dbname);
/* logical replication mode */
- rconninfo = psprintf("%s replication=database", conninfo);
+ rconninfo = psprintf("%s replication=database", concat_conninfo);
conn = PQconnectdb(rconninfo);
if (PQstatus(conn) != CONNECTION_OK)
@@ -424,6 +428,9 @@ connect_database(const char *conninfo)
}
PQclear(res);
+ pfree(rconninfo);
+ pfree(concat_conninfo);
+
return conn;
}
@@ -436,19 +443,18 @@ disconnect_database(PGconn *conn)
}
/*
- * Obtain the system identifier using the provided connection. It will be used
- * to compare if a data directory is a clone of another one.
+ * Obtain the system identifier from the primary server. It will be used to
+ * compare if a data directory is a clone of another one.
*/
-static uint64
-get_sysid_from_conn(const char *conninfo)
+static void
+get_sysid_for_primary(PrimaryInfo *primary, char *dbname)
{
PGconn *conn;
PGresult *res;
- uint64 sysid;
pg_log_info("getting system identifier from publisher");
- conn = connect_database(conninfo);
+ conn = connect_database(primary->base_conninfo, dbname);
if (conn == NULL)
exit(1);
@@ -471,43 +477,39 @@ get_sysid_from_conn(const char *conninfo)
exit(1);
}
- sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+ primary->sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
- pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+ pg_log_info("system identifier is %llu on publisher",
+ (unsigned long long) primary->sysid);
disconnect_database(conn);
-
- return sysid;
}
/*
- * Obtain the system identifier from control file. It will be used to compare
- * if a data directory is a clone of another one. This routine is used locally
- * and avoids a replication connection.
+ * Obtain the system identifier from a standby server. It will be used to
+ * compare if a data directory is a clone of another one. This routine is used
+ * locally and avoids a replication connection.
*/
-static uint64
-get_control_from_datadir(const char *datadir)
+static void
+get_control_for_standby(StandbyInfo *standby)
{
ControlFileData *cf;
bool crc_ok;
- uint64 sysid;
pg_log_info("getting system identifier from subscriber");
- cf = get_controlfile(datadir, &crc_ok);
+ cf = get_controlfile(standby->pgdata, &crc_ok);
if (!crc_ok)
{
pg_log_error("control file appears to be corrupt");
exit(1);
}
- sysid = cf->system_identifier;
+ standby->sysid = cf->system_identifier;
- pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) standby->sysid);
pfree(cf);
-
- return sysid;
}
/*
@@ -516,7 +518,7 @@ get_control_from_datadir(const char *datadir)
* files from one of the systems might be used in the other one.
*/
static void
-modify_sysid(const char *pg_resetwal_path, const char *datadir)
+modify_sysid(const char *bindir, const char *datadir)
{
ControlFileData *cf;
bool crc_ok;
@@ -551,7 +553,7 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
pg_log_info("running pg_resetwal on the subscriber");
- cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+ cmd_str = psprintf("\"%s/pg_resetwal\" -D \"%s\"", bindir, datadir);
pg_log_debug("command is: %s", cmd_str);
@@ -571,14 +573,15 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
* Return a palloc'd slot name if the replication is using one.
*/
static char *
-use_primary_slot_name(void)
+use_primary_slot_name(PrimaryInfo *primary, StandbyInfo *standby,
+ LogicalRepPerdbInfo *perdb)
{
PGconn *conn;
PGresult *res;
PQExpBuffer str = createPQExpBuffer();
char *slot_name;
- conn = connect_database(dbinfo[0].subconninfo);
+ conn = connect_database(standby->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -604,7 +607,7 @@ use_primary_slot_name(void)
disconnect_database(conn);
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -634,17 +637,19 @@ use_primary_slot_name(void)
}
static bool
-create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
+create_all_logical_replication_slots(PrimaryInfo *primary,
+ LogicalRepPerdbInfoArr *dbarr)
{
int i;
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr->ndbs; i++)
{
PGconn *conn;
PGresult *res;
char replslotname[NAMEDATALEN];
+ LogicalRepPerdbInfo *perdb = &dbarr->perdb[i];
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -664,27 +669,14 @@ create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
}
/* Remember database OID. */
- dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ perdb->oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
PQclear(res);
- /*
- * Build the replication slot name. The name must not exceed
- * NAMEDATALEN - 1. This current schema uses a maximum of 36
- * characters (14 + 10 + 1 + 10 + '\0'). System identifier is included
- * to reduce the probability of collision. By default, subscription
- * name is used as replication slot name.
- */
- snprintf(replslotname, sizeof(replslotname),
- "pg_subscriber_%u_%d",
- dbinfo[i].oid,
- (int) getpid());
- dbinfo[i].subname = pg_strdup(replslotname);
+ get_subscription_name(perdb->oid, (int) getpid(), replslotname, NAMEDATALEN);
/* Create replication slot on publisher. */
- if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
- pg_log_info("create replication slot \"%s\" on publisher", replslotname);
- else
+ if (create_logical_replication_slot(conn, false, perdb) == NULL && !dry_run)
return false;
disconnect_database(conn);
@@ -701,30 +693,36 @@ create_all_logical_replication_slots(LogicalRepInfo *dbinfo)
* result set that contains the consistent LSN.
*/
static char *
-create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
- char *slot_name)
+create_logical_replication_slot(PGconn *conn, bool temporary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res = NULL;
char *lsn = NULL;
- bool transient_replslot = false;
+ char slot_name[NAMEDATALEN];
Assert(conn != NULL);
/*
- * If no slot name is informed, it is a transient replication slot used
- * only for catch up purposes.
+ * Construct a name of logical replication slot. The formatting is
+ * different depends on its persistency.
+ *
+ * For persistent slots: the name must be same as the subscription.
+ * For temporary slots: OID is not needed, but another string is added.
*/
- if (slot_name[0] == '\0')
- {
+ if (!temporary)
+ get_subscription_name(perdb->oid, (int) getpid(), slot_name, NAMEDATALEN);
+ else
snprintf(slot_name, NAMEDATALEN, "pg_subscriber_%d_startpoint",
(int) getpid());
- transient_replslot = true;
- }
- pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, perdb->dbname);
appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
+
+ if(temporary)
+ appendPQExpBufferStr(str, " TEMPORARY");
+
appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
pg_log_debug("command is: %s", str->data);
@@ -734,17 +732,14 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
- pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
- PQresultErrorMessage(res));
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s",
+ slot_name, perdb->dbname, PQresultErrorMessage(res));
return lsn;
}
}
/* for cleanup purposes */
- if (transient_replslot)
- made_transient_replslot = true;
- else
- dbinfo->made_replslot = true;
+ perdb->made_replslot = true;
if (!dry_run)
{
@@ -758,14 +753,15 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
}
static void
-drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+drop_replication_slot(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *slot_name)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
Assert(conn != NULL);
- pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, perdb->dbname);
appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
@@ -775,7 +771,7 @@ drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_nam
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, perdb->dbname,
PQerrorMessage(conn));
PQclear(res);
@@ -825,19 +821,22 @@ pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
* Returns after the server finishes the recovery process.
*/
static void
-wait_for_end_recovery(const char *conninfo)
+wait_for_end_recovery(const char *base_conninfo, const char *dbname)
{
PGconn *conn;
PGresult *res;
- int status = POSTMASTER_STILL_STARTING;
+ int status = PGS_POSTMASTER_STILL_STARTING;
+ int cnt;
+ int rc;
+ char *pg_ctl_cmd;
pg_log_info("waiting the postmaster to reach the consistent state");
- conn = connect_database(conninfo);
+ conn = connect_database(base_conninfo, dbname);
if (conn == NULL)
exit(1);
- for (;;)
+ for (cnt = 0; cnt < wait_seconds * WAITS_PER_SEC; cnt++)
{
bool in_recovery;
@@ -865,17 +864,32 @@ wait_for_end_recovery(const char *conninfo)
*/
if (!in_recovery || dry_run)
{
- status = POSTMASTER_READY;
+ status = PGS_POSTMASTER_READY;
break;
}
/* Keep waiting. */
- pg_usleep(WAIT_INTERVAL * USEC_PER_SEC);
+ pg_usleep(USEC_PER_SEC / WAITS_PER_SEC);
}
disconnect_database(conn);
- if (status == POSTMASTER_STILL_STARTING)
+ /*
+ * If timeout is reached exit the pg_subscriber and stop the standby node.
+ */
+ if (cnt >= wait_seconds * WAITS_PER_SEC)
+ {
+ pg_log_error("recovery timed out");
+
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
+ standby.bindir, standby.pgdata);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+ exit(1);
+ }
+
+ if (status == PGS_POSTMASTER_STILL_STARTING)
{
pg_log_error("server did not end recovery");
exit(1);
@@ -888,17 +902,21 @@ wait_for_end_recovery(const char *conninfo)
* Create a publication that includes all tables in the database.
*/
static void
-create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+create_publication(PGconn *conn, PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
/* Check if the publication needs to be created. */
appendPQExpBuffer(str,
"SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
- dbinfo->pubname);
+ pubname);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
@@ -918,7 +936,7 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
*/
if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
{
- pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ pg_log_info("publication \"%s\" already exists", pubname);
return;
}
else
@@ -931,7 +949,7 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* database oid in which puballtables is false.
*/
pg_log_error("publication \"%s\" does not replicate changes for all tables",
- dbinfo->pubname);
+ pubname);
pg_log_error_hint("Consider renaming this publication.");
PQclear(res);
PQfinish(conn);
@@ -942,9 +960,9 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
PQclear(res);
resetPQExpBuffer(str);
- pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+ pg_log_info("creating publication \"%s\" on database \"%s\"", pubname, perdb->dbname);
- appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", pubname);
pg_log_debug("command is: %s", str->data);
@@ -954,14 +972,14 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
- dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ pubname, perdb->dbname, PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
}
/* for cleanup purposes */
- dbinfo->made_publication = true;
+ perdb->made_publication = true;
if (!dry_run)
PQclear(res);
@@ -973,16 +991,19 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* Remove publication if it couldn't finish all steps.
*/
static void
-drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+drop_publication(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
- appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+ pg_log_info("dropping publication \"%s\" on database \"%s\"", pubname, perdb->dbname);
+
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", pubname);
pg_log_debug("command is: %s", str->data);
@@ -990,7 +1011,7 @@ drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", pubname, perdb->dbname, PQerrorMessage(conn));
PQclear(res);
}
@@ -1011,19 +1032,27 @@ drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* initial location.
*/
static void
-create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+create_subscription(PGconn *conn, StandbyInfo *standby, char *base_conninfo,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", subname,
+ perdb->dbname);
appendPQExpBuffer(str,
"CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
"WITH (create_slot = false, copy_data = false, enabled = false)",
- dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+ subname, concat_conninfo_dbname(base_conninfo, perdb->dbname), pubname);
pg_log_debug("command is: %s", str->data);
@@ -1033,14 +1062,14 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
- dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ subname, perdb->dbname, PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
}
/* for cleanup purposes */
- dbinfo->made_subscription = true;
+ perdb->made_subscription = true;
if (!dry_run)
PQclear(res);
@@ -1052,16 +1081,19 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* Remove subscription if it couldn't finish all steps.
*/
static void
-drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+drop_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"", subname, perdb->dbname);
- appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", subname);
pg_log_debug("command is: %s", str->data);
@@ -1069,7 +1101,7 @@ drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", subname, perdb->dbname, PQerrorMessage(conn));
PQclear(res);
}
@@ -1088,18 +1120,21 @@ drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* printing purposes.
*/
static void
-set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+set_replication_progress(PGconn *conn, LogicalRepPerdbInfo *perdb, const char *lsn)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
Oid suboid;
char originname[NAMEDATALEN];
char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
appendPQExpBuffer(str,
- "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", subname);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
@@ -1140,7 +1175,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
PQclear(res);
pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
- originname, lsnstr, dbinfo->dbname);
+ originname, lsnstr, perdb->dbname);
resetPQExpBuffer(str);
appendPQExpBuffer(str,
@@ -1154,7 +1189,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
pg_log_error("could not set replication progress for the subscription \"%s\": %s",
- dbinfo->subname, PQresultErrorMessage(res));
+ subname, PQresultErrorMessage(res));
PQfinish(conn);
exit(1);
}
@@ -1173,16 +1208,20 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
* of this setup.
*/
static void
-enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+enable_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", subname,
+ perdb->dbname);
- appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", subname);
pg_log_debug("command is: %s", str->data);
@@ -1191,7 +1230,7 @@ enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
- pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ pg_log_error("could not enable subscription \"%s\": %s", subname,
PQerrorMessage(conn));
PQfinish(conn);
exit(1);
@@ -1203,6 +1242,61 @@ enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
destroyPQExpBuffer(str);
}
+static void
+start_standby_server(StandbyInfo *standby, unsigned short subport,
+ char *server_start_log)
+{
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+ int rc;
+ char *pg_ctl_cmd;
+
+ if (server_start_log[0] == '\0')
+ {
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ len = snprintf(server_start_log, MAXPGPATH,
+ "%s/%s/server_start_%s.log", standby->pgdata,
+ PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+ }
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" start -D \"%s\" -s -o \"-p %d\" -l \"%s\"",
+ standby->bindir,
+ standby->pgdata, subport, server_start_log);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+}
+
+static char *
+construct_sub_conninfo(char *username, unsigned short subport)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *ret;
+
+ if (username)
+ appendPQExpBuffer(buf, "user=%s ", username);
+
+ appendPQExpBuffer(buf, "port=%d fallback_application_name=%s",
+ subport, progname);
+
+ ret = pg_strdup(buf->data);
+
+ destroyPQExpBuffer(buf);
+
+ return ret;
+}
+
int
main(int argc, char **argv)
{
@@ -1214,6 +1308,10 @@ main(int argc, char **argv)
{"publisher-conninfo", required_argument, NULL, 'P'},
{"subscriber-conninfo", required_argument, NULL, 'S'},
{"database", required_argument, NULL, 'd'},
+ {"timeout", required_argument, NULL, 't'},
+ {"username", required_argument, NULL, 'u'},
+ {"port", required_argument, NULL, 'p'},
+ {"retain", no_argument, NULL, 'r'},
{"dry-run", no_argument, NULL, 'n'},
{"verbose", no_argument, NULL, 'v'},
{NULL, 0, NULL, 0}
@@ -1225,20 +1323,15 @@ main(int argc, char **argv)
char *pg_ctl_cmd;
- char *base_dir;
- char *server_start_log;
-
- char timebuf[128];
- struct timeval time;
- time_t tt;
+ char base_dir[MAXPGPATH];
+ char server_start_log[MAXPGPATH] = {0};
int len;
- char *pub_base_conninfo = NULL;
- char *sub_base_conninfo = NULL;
char *dbname_conninfo = NULL;
- uint64 pub_sysid;
- uint64 sub_sysid;
+ unsigned short subport = DEF_PGSPORT;
+ char *username = NULL;
+
struct stat statbuf;
PGconn *conn;
@@ -1250,6 +1343,13 @@ main(int argc, char **argv)
int i;
+ PGresult *res;
+
+ char *wal_level;
+ int max_replication_slots;
+ int nslots_old;
+ int nslots_new;
+
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
progname = get_progname(argv[0]);
@@ -1286,28 +1386,40 @@ main(int argc, char **argv)
}
#endif
- while ((c = getopt_long(argc, argv, "D:P:S:d:nv",
+ while ((c = getopt_long(argc, argv, "D:P:S:d:t:u:p:rnv",
long_options, &option_index)) != -1)
{
switch (c)
{
case 'D':
- subscriber_dir = pg_strdup(optarg);
+ standby.pgdata = pg_strdup(optarg);
+ canonicalize_path(standby.pgdata);
break;
case 'P':
pub_conninfo_str = pg_strdup(optarg);
break;
- case 'S':
- sub_conninfo_str = pg_strdup(optarg);
- break;
case 'd':
/* Ignore duplicated database names. */
if (!simple_string_list_member(&database_names, optarg))
{
simple_string_list_append(&database_names, optarg);
- num_dbs++;
+ dbarr.ndbs++;
}
break;
+ case 't':
+ wait_seconds = atoi(optarg);
+ break;
+ case 'u':
+ pfree(username);
+ username = pg_strdup(optarg);
+ break;
+ case 'p':
+ if ((subport = atoi(optarg)) <= 0)
+ pg_fatal("invalid old port number");
+ break;
+ case 'r':
+ retain = true;
+ break;
case 'n':
dry_run = true;
break;
@@ -1335,7 +1447,7 @@ main(int argc, char **argv)
/*
* Required arguments
*/
- if (subscriber_dir == NULL)
+ if (standby.pgdata == NULL)
{
pg_log_error("no subscriber data directory specified");
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1358,21 +1470,14 @@ main(int argc, char **argv)
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
exit(1);
}
- pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
- "publisher");
- if (pub_base_conninfo == NULL)
- exit(1);
- if (sub_conninfo_str == NULL)
- {
- pg_log_error("no subscriber connection string specified");
- pg_log_error_hint("Try \"%s --help\" for more information.", progname);
- exit(1);
- }
- sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
- if (sub_base_conninfo == NULL)
+ primary.base_conninfo = get_base_conninfo(pub_conninfo_str,
+ dbname_conninfo);
+ if (primary.base_conninfo == NULL)
exit(1);
+ standby.base_conninfo = construct_sub_conninfo(username, subport);
+
if (database_names.head == NULL)
{
pg_log_info("no database was specified");
@@ -1385,7 +1490,7 @@ main(int argc, char **argv)
if (dbname_conninfo)
{
simple_string_list_append(&database_names, dbname_conninfo);
- num_dbs++;
+ dbarr.ndbs++;
pg_log_info("database \"%s\" was extracted from the publisher connection string",
dbname_conninfo);
@@ -1399,25 +1504,25 @@ main(int argc, char **argv)
}
/*
- * Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
+ * Get the absolute path of binaries on the subscriber.
*/
- if (!get_exec_path(argv[0]))
+ if (!get_exec_base_path(argv[0]))
exit(1);
/* rudimentary check for a data directory. */
- if (!check_data_directory(subscriber_dir))
+ if (!check_data_directory(standby.pgdata))
exit(1);
- /* Store database information for publisher and subscriber. */
- dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+ /* Store database information to dbarr */
+ store_db_names(&dbarr.perdb, dbarr.ndbs);
/*
* Check if the subscriber data directory has the same system identifier
* than the publisher data directory.
*/
- pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
- sub_sysid = get_control_from_datadir(subscriber_dir);
- if (pub_sysid != sub_sysid)
+ get_sysid_for_primary(&primary, dbarr.perdb[0].dbname);
+ get_control_for_standby(&standby);
+ if (primary.sysid != standby.sysid)
{
pg_log_error("subscriber data directory is not a copy of the source database cluster");
exit(1);
@@ -1426,8 +1531,8 @@ main(int argc, char **argv)
/*
* Create the output directory to store any data generated by this tool.
*/
- base_dir = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s",
+ standby.pgdata, PGS_OUTPUT_DIR);
if (len >= MAXPGPATH)
{
pg_log_error("directory path for subscriber is too long");
@@ -1441,7 +1546,153 @@ main(int argc, char **argv)
}
/* subscriber PID file. */
- snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid",
+ standby.pgdata);
+
+ /* Start the standby server anyway */
+ start_standby_server(&standby, subport, server_start_log);
+
+ /*
+ * Check wal_level in publisher and the max_replication_slots of publisher
+ */
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "SELECT count(*) from pg_replication_slots;");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain number of replication slots");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not determine parameter settings on publisher");
+ exit(1);
+ }
+
+ nslots_old = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ res = PQexec(conn, "SELECT setting FROM pg_settings "
+ "WHERE name IN ('wal_level', 'max_replication_slots') "
+ "ORDER BY name DESC;");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain guc parameters on publisher");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 2)
+ {
+ pg_log_error("could not determine parameter settings on publisher");
+ exit(1);
+ }
+
+ wal_level = PQgetvalue(res, 0, 0);
+
+ if (strcmp(wal_level, "logical") != 0)
+ {
+ pg_log_error("wal_level must be \"logical\", but is set to \"%s\"", wal_level);
+ exit(1);
+ }
+
+ max_replication_slots = atoi(PQgetvalue(res, 1, 0));
+ nslots_new = nslots_old + dbarr.ndbs + 1;
+
+ if (nslots_new > max_replication_slots)
+ {
+ pg_log_error("max_replication_slots (%d) must be greater than or equal to "
+ "the number of replication slots required (%d)", max_replication_slots, nslots_new);
+ exit(1);
+ }
+
+ PQclear(res);
+ disconnect_database(conn);
+
+ conn = connect_database(standby.base_conninfo, dbarr.perdb[0].dbname);
+ if (conn == NULL)
+ exit(1);
+
+ /*
+ * Check the max_replication_slots in subscriber
+ */
+ res = PQexec(conn, "SELECT count(*) from pg_replication_slots;");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain number of replication slots on subscriber");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not determine parameter settings on subscriber");
+ exit(1);
+ }
+
+ nslots_old = atoi(PQgetvalue(res, 0, 0));
+ PQclear(res);
+
+ res = PQexec(conn, "SELECT setting FROM pg_settings "
+ "WHERE name = 'max_replication_slots';");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain guc parameters");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not determine parameter settings on publisher");
+ exit(1);
+ }
+
+ max_replication_slots = atoi(PQgetvalue(res, 0, 0));
+ nslots_new = nslots_old + dbarr.ndbs;
+
+ if (nslots_new > max_replication_slots)
+ {
+ pg_log_error("max_replication_slots (%d) must be greater than or equal to "
+ "the number of replication slots required (%d)", max_replication_slots, nslots_new);
+ exit(1);
+ }
+
+ PQclear(res);
+
+ /*
+ * Exit the pg_subscriber if the node is not a standby server.
+ */
+ res = PQexec(conn, "SELECT pg_catalog.pg_is_in_recovery()");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain recovery progress");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("unexpected result from pg_is_in_recovery function");
+ exit(1);
+ }
+
+ /* Check if the server is in recovery */
+ if (strcmp(PQgetvalue(res, 0, 0), "t") != 0)
+ {
+ pg_log_error("pg_subscriber is supported only on standby server");
+ exit(1);
+ }
+
+ PQclear(res);
+ disconnect_database(conn);
+
+ /* subscriber PID file. */
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", standby.pgdata);
/*
* Stop the subscriber if it is a standby server. Before executing the
@@ -1457,14 +1708,18 @@ main(int argc, char **argv)
* replication slot has no use after the transformation, hence, it
* will be removed at the end of this process.
*/
- primary_slot_name = use_primary_slot_name();
- if (primary_slot_name != NULL)
- pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+ standby.primary_slot_name = use_primary_slot_name(&primary,
+ &standby,
+ &dbarr.perdb[0]);
+ if (standby.primary_slot_name != NULL)
+ pg_log_info("primary has replication slot \"%s\"",
+ standby.primary_slot_name);
pg_log_info("subscriber is up and running");
pg_log_info("stopping the server to start the transformation steps");
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
+ standby.bindir, standby.pgdata);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
}
@@ -1472,7 +1727,7 @@ main(int argc, char **argv)
/*
* Create a replication slot for each database on the publisher.
*/
- if (!create_all_logical_replication_slots(dbinfo))
+ if (!create_all_logical_replication_slots(&primary, &dbarr))
exit(1);
/*
@@ -1492,11 +1747,11 @@ main(int argc, char **argv)
* replication connection open (depending when base backup was taken, the
* connection should be open for a few hours).
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
if (conn == NULL)
exit(1);
- consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
- temp_replslot);
+ consistent_lsn = create_logical_replication_slot(conn, true,
+ &dbarr.perdb[0]);
/*
* Write recovery parameters.
@@ -1522,7 +1777,7 @@ main(int argc, char **argv)
{
appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
consistent_lsn);
- WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ WriteRecoveryConfig(conn, standby.pgdata, recoveryconfcontents);
}
disconnect_database(conn);
@@ -1532,54 +1787,29 @@ main(int argc, char **argv)
* Start subscriber and wait until accepting connections.
*/
pg_log_info("starting the subscriber");
-
- /* append timestamp with ISO 8601 format. */
- gettimeofday(&time, NULL);
- tt = (time_t) time.tv_sec;
- strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
- snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
- ".%03d", (int) (time.tv_usec / 1000));
-
- server_start_log = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(server_start_log, MAXPGPATH, "%s/%s/server_start_%s.log", subscriber_dir, PGS_OUTPUT_DIR, timebuf);
- if (len >= MAXPGPATH)
- {
- pg_log_error("log file path is too long");
- exit(1);
- }
-
- pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, subscriber_dir, server_start_log);
- rc = system(pg_ctl_cmd);
- pg_ctl_status(pg_ctl_cmd, rc, 1);
+ start_standby_server(&standby, subport, server_start_log);
/*
* Waiting the subscriber to be promoted.
*/
- wait_for_end_recovery(dbinfo[0].subconninfo);
+ wait_for_end_recovery(standby.base_conninfo, dbarr.perdb[0].dbname);
/*
* Create a publication for each database. This step should be executed
* after promoting the subscriber to avoid replicating unnecessary
* objects.
*/
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
- char pubname[NAMEDATALEN];
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
/* Connect to publisher. */
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary.base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
- /*
- * Build the publication name. The name must not exceed NAMEDATALEN -
- * 1. This current schema uses a maximum of 35 characters (14 + 10 +
- * '\0').
- */
- snprintf(pubname, sizeof(pubname), "pg_subscriber_%u", dbinfo[i].oid);
- dbinfo[i].pubname = pg_strdup(pubname);
-
- create_publication(conn, &dbinfo[i]);
+ /* Also create a publication */
+ create_publication(conn, &primary, perdb);
disconnect_database(conn);
}
@@ -1587,20 +1817,25 @@ main(int argc, char **argv)
/*
* Create a subscription for each database.
*/
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
+
/* Connect to subscriber. */
- conn = connect_database(dbinfo[i].subconninfo);
+ conn = connect_database(standby.base_conninfo, perdb->dbname);
+
if (conn == NULL)
exit(1);
- create_subscription(conn, &dbinfo[i]);
+ create_subscription(conn, &standby, primary.base_conninfo, perdb);
/* Set the replication progress to the correct LSN. */
- set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+ set_replication_progress(conn, perdb, consistent_lsn);
/* Enable subscription. */
- enable_subscription(conn, &dbinfo[i]);
+ enable_subscription(conn, perdb);
+
+ drop_publication(conn, perdb);
disconnect_database(conn);
}
@@ -1613,19 +1848,21 @@ main(int argc, char **argv)
* XXX we might not fail here. Instead, we provide a warning so the user
* eventually drops the replication slot later.
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
if (conn == NULL)
{
- pg_log_warning("could not drop transient replication slot \"%s\" on publisher", temp_replslot);
- pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
+ char *primary_slot_name = standby.primary_slot_name;
+
if (primary_slot_name != NULL)
pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
}
else
{
- drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[0];
+ char *primary_slot_name = standby.primary_slot_name;
+
if (primary_slot_name != NULL)
- drop_replication_slot(conn, &dbinfo[0], primary_slot_name);
+ drop_replication_slot(conn, perdb, primary_slot_name);
disconnect_database(conn);
}
@@ -1634,20 +1871,22 @@ main(int argc, char **argv)
*/
pg_log_info("stopping the subscriber");
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
+ standby.bindir, standby.pgdata);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
/*
* Change system identifier.
*/
- modify_sysid(pg_resetwal_path, subscriber_dir);
+ modify_sysid(standby.bindir, standby.pgdata);
/*
* Remove log file generated by this tool, if it runs successfully.
* Otherwise, file is kept that may provide useful debugging information.
*/
- unlink(server_start_log);
+ if (!retain)
+ unlink(server_start_log);
success = true;
diff --git a/src/bin/pg_basebackup/t/040_pg_subscriber.pl b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
index 4ebff76b2d..9915b8cb3c 100644
--- a/src/bin/pg_basebackup/t/040_pg_subscriber.pl
+++ b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
@@ -37,8 +37,13 @@ command_fails(
'--verbose',
'--pgdata', $datadir,
'--publisher-conninfo', 'dbname=postgres',
- '--subscriber-conninfo', 'dbname=postgres'
],
'no database name specified');
-
+command_fails(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres',
+ ],
+ 'subscriber connection string specnfied non-local server');
done_testing();
diff --git a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
index fbcd0fc82b..4e26607611 100644
--- a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
+++ b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
@@ -51,25 +51,27 @@ $node_s->start;
$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('second row')");
$node_p->wait_for_replay_catchup($node_s);
+$node_f->stop;
+
# Run pg_subscriber on about-to-fail node F
command_fails(
[
'pg_subscriber', '--verbose',
'--pgdata', $node_f->data_dir,
'--publisher-conninfo', $node_p->connstr('pg1'),
- '--subscriber-conninfo', $node_f->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
],
'subscriber data directory is not a copy of the source database cluster');
+$node_s->stop;
+
# dry run mode on node S
command_ok(
[
'pg_subscriber', '--verbose', '--dry-run',
'--pgdata', $node_s->data_dir,
'--publisher-conninfo', $node_p->connstr('pg1'),
- '--subscriber-conninfo', $node_s->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
],
@@ -82,6 +84,7 @@ $node_s->start;
# Check if node S is still a standby
is($node_s->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
't', 'standby is in recovery');
+$node_s->stop;
# Run pg_subscriber on node S
command_ok(
@@ -89,7 +92,6 @@ command_ok(
'pg_subscriber', '--verbose',
'--pgdata', $node_s->data_dir,
'--publisher-conninfo', $node_p->connstr('pg1'),
- '--subscriber-conninfo', $node_s->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
],
--
2.43.0
[application/octet-stream] v8-0003-Fix-publication-does-not-exist-error.patch (1.6K, ../../TY3PR01MB9889678E47B918F4D83A6FD8F57B2@TY3PR01MB9889.jpnprd01.prod.outlook.com/4-v8-0003-Fix-publication-does-not-exist-error.patch)
download | inline diff:
From 9f538ed90a02fb13404ff0df4884d1e809a13f1c Mon Sep 17 00:00:00 2001
From: Shlok Kyal <[email protected]>
Date: Mon, 22 Jan 2024 12:36:20 +0530
Subject: [PATCH v8 3/4] Fix publication does not exist error.
Fix publication does not exist error.
---
src/bin/pg_basebackup/pg_subscriber.c | 23 +++--------------------
1 file changed, 3 insertions(+), 20 deletions(-)
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index 3880d15ef9..355738c20c 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -679,6 +679,9 @@ create_all_logical_replication_slots(PrimaryInfo *primary,
if (create_logical_replication_slot(conn, false, perdb) == NULL && !dry_run)
return false;
+ /* Also create a publication */
+ create_publication(conn, primary, perdb);
+
disconnect_database(conn);
}
@@ -1794,26 +1797,6 @@ main(int argc, char **argv)
*/
wait_for_end_recovery(standby.base_conninfo, dbarr.perdb[0].dbname);
- /*
- * Create a publication for each database. This step should be executed
- * after promoting the subscriber to avoid replicating unnecessary
- * objects.
- */
- for (i = 0; i < dbarr.ndbs; i++)
- {
- LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
-
- /* Connect to publisher. */
- conn = connect_database(primary.base_conninfo, perdb->dbname);
- if (conn == NULL)
- exit(1);
-
- /* Also create a publication */
- create_publication(conn, &primary, perdb);
-
- disconnect_database(conn);
- }
-
/*
* Create a subscription for each database.
*/
--
2.43.0
[application/octet-stream] v8-0004-Move-a-registration-of-atexit-callback-to-behind.patch (1.1K, ../../TY3PR01MB9889678E47B918F4D83A6FD8F57B2@TY3PR01MB9889.jpnprd01.prod.outlook.com/5-v8-0004-Move-a-registration-of-atexit-callback-to-behind.patch)
download | inline diff:
From eb912015aa8b5d8481b852d91e3d146fc1a31703 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <[email protected]>
Date: Wed, 24 Jan 2024 07:18:47 +0000
Subject: [PATCH v8 4/4] Move a registration of atexit() callback to behind
---
src/bin/pg_basebackup/pg_subscriber.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index 355738c20c..17a6b552af 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -1373,8 +1373,6 @@ main(int argc, char **argv)
}
}
- atexit(cleanup_objects_atexit);
-
/*
* Don't allow it to be run as root. It uses pg_ctl which does not allow
* it either.
@@ -1727,6 +1725,12 @@ main(int argc, char **argv)
pg_ctl_status(pg_ctl_cmd, rc, 0);
}
+ /*
+ * Subsequent operations define some database objects on both primary and
+ * standby. The callback is useful to clean up them in case of failure.
+ */
+ atexit(cleanup_objects_atexit);
+
/*
* Create a replication slot for each database on the publisher.
*/
--
2.43.0
^ permalink raw reply [nested|flat] 77+ messages in thread
* RE: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-23 02:11 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-24 06:11 ` Re: speed up a logical replica setup Shubham Khanna <[email protected]>
2024-01-24 11:43 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
@ 2024-01-25 09:05 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-26 00:28 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-25 09:05 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Euler Taveira <[email protected]>; 'Shubham Khanna' <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>
Dear hackers,
Here are comments for v8 patch set. I may revise them by myself,
but I want to post here to share all of them.
01.
```
/* Options */
static char *pub_conninfo_str = NULL;
static SimpleStringList database_names = {NULL, NULL};
static int wait_seconds = DEFAULT_WAIT;
static bool retain = false;
static bool dry_run = false;
```
Just to confirm - is there a policy why we store the specified options? If you
want to store as global ones, username and port should follow (my fault...).
Or, should we have a structure to store them?
02.
```
{"subscriber-conninfo", required_argument, NULL, 'S'},
```
This is my fault, but "--subscriber-conninfo" is still remained. It should be
removed if it is not really needed.
03.
```
{"username", required_argument, NULL, 'u'},
```
Should we accept 'U' instead of 'u'?
04.
```
{"dry-run", no_argument, NULL, 'n'},
```
I'm not sure why the dry_run mode exists. In terms pg_resetwal, it shows the
which value would be changed based on the input. As for the pg_upgrade, it checks
whether the node can be upgraded for now. I think, we should have the checking
feature, so it should be renamed to --check. Also, the process should exit earlier
at that time.
05.
I felt we should accept some settings from enviroment variables, like pg_upgrade.
Currently, below items should be acceted.
- data directory
- username
- port
- timeout
06.
```
pg_logging_set_level(PG_LOG_WARNING);
```
If the default log level is warning, there are no ways to output debug logs.
(-v option only raises one, so INFO would be output)
I think it should be PG_LOG_INFO.
07.
Can we combine verifications into two functions, e.g., check_primary() and check_standby/check_subscriber()?
08.
Not sure, but if we want to record outputs by pg_subscriber, the sub-directory
should be created. The name should contain the timestamp.
09.
Not sure, but should we check max_slot_wal_keep_size of primary server? It can
avoid to fail starting of logical replicaiton.
10.
```
nslots_new = nslots_old + dbarr.ndbs;
if (nslots_new > max_replication_slots)
{
pg_log_error("max_replication_slots (%d) must be greater than or equal to "
"the number of replication slots required (%d)", max_replication_slots, nslots_new);
exit(1);
}
```
I think standby server must not have replication slots. Because subsequent
pg_resetwal command discards all the WAL file, so WAL records pointed by them
are removed. Currently pg_resetwal does not raise ERROR at that time.
11.
```
/*
* Stop the subscriber if it is a standby server. Before executing the
* transformation steps, make sure the subscriber is not running because
* one of the steps is to modify some recovery parameters that require a
* restart.
*/
if (stat(pidfile, &statbuf) == 0)
```
I kept just in case, but I'm not sure it is still needed. How do you think?
Removing it can reduce an inclusion of pidfile.h.
12.
```
pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
standby.bindir, standby.pgdata);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
```
There are two places to stop the instance. Can you divide it into a function?
13.
```
* A temporary replication slot is not used here to avoid keeping a
* replication connection open (depending when base backup was taken, the
* connection should be open for a few hours).
*/
conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
if (conn == NULL)
exit(1);
consistent_lsn = create_logical_replication_slot(conn, true,
&dbarr.perdb[0]);
```
I didn't notice the comment, but still not sure the reason. Why we must reserve
the slot until pg_subscriber finishes? IIUC, the slot would be never used, it
is created only for getting a consistent_lsn. So we do not have to keep.
Also, just before, logical replication slots for each databases are created, so
WAL records are surely reserved.
14.
```
pg_log_info("starting the subscriber");
start_standby_server(&standby, subport, server_start_log);
```
This info should be in the function.
15.
```
/*
* Create a subscription for each database.
*/
for (i = 0; i < dbarr.ndbs; i++)
```
This can be divided into a function, like create_all_subscriptions().
16.
My fault: usage() must be updated.
17. use_primary_slot_name
```
if (PQntuples(res) != 1)
{
pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
PQntuples(res), 1);
return NULL;
}
```
Error message should be changed. I think this error means the standby has wrong primary_slot_name, right?
18. misc
Sometimes the pid of pg_subscriber is referred. It can be stored as global variable.
19.
C99-style has been allowed, so loop variables like "i" can be declared in the for-statement, like
```
for (int i = 0; i < MAX; i++)
```
20.
Some comments, docs, and outputs must be fixed when the name is changed.
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-23 02:11 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-24 06:11 ` Re: speed up a logical replica setup Shubham Khanna <[email protected]>
2024-01-24 11:43 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-25 09:05 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
@ 2024-01-26 00:28 ` Euler Taveira <[email protected]>
2024-01-26 05:51 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Euler Taveira @ 2024-01-26 00:28 UTC (permalink / raw)
To: [email protected] <[email protected]>; [email protected] <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; 'Shubham Khanna' <[email protected]>
On Thu, Jan 25, 2024, at 6:05 AM, Hayato Kuroda (Fujitsu) wrote:
> 01.
> ```
> /* Options */
> static char *pub_conninfo_str = NULL;
> static SimpleStringList database_names = {NULL, NULL};
> static int wait_seconds = DEFAULT_WAIT;
> static bool retain = false;
> static bool dry_run = false;
> ```
>
> Just to confirm - is there a policy why we store the specified options? If you
> want to store as global ones, username and port should follow (my fault...).
> Or, should we have a structure to store them?
It is a matter of style I would say. Check other client applications. Some of
them also use global variable. There are others that group options into a
struct. I would say that since it has a short lifetime, I don't think the
current style is harmful.
> 04.
> ```
> {"dry-run", no_argument, NULL, 'n'},
> ```
>
> I'm not sure why the dry_run mode exists. In terms pg_resetwal, it shows the
> which value would be changed based on the input. As for the pg_upgrade, it checks
> whether the node can be upgraded for now. I think, we should have the checking
> feature, so it should be renamed to --check. Also, the process should exit earlier
> at that time.
It is extremely useful because (a) you have a physical replication setup and
don't know if it is prepared for logical replication, (b) check GUCs (is
max_wal_senders sufficient for this pg_subscriber command? Or is
max_replication_slots sufficient to setup the logical replication even though I
already have some used replication slots?), (c) connectivity and (d)
credentials.
> 05.
> I felt we should accept some settings from enviroment variables, like pg_upgrade.
> Currently, below items should be acceted.
>
> - data directory
> - username
> - port
> - timeout
Maybe PGDATA.
> 06.
> ```
> pg_logging_set_level(PG_LOG_WARNING);
> ```
>
> If the default log level is warning, there are no ways to output debug logs.
> (-v option only raises one, so INFO would be output)
> I think it should be PG_LOG_INFO.
You need to specify multiple -v options.
> 07.
> Can we combine verifications into two functions, e.g., check_primary() and check_standby/check_subscriber()?
I think v9 does it.
> 08.
> Not sure, but if we want to record outputs by pg_subscriber, the sub-directory
> should be created. The name should contain the timestamp.
The log file already contains the timestamp. Why?
> 09.
> Not sure, but should we check max_slot_wal_keep_size of primary server? It can
> avoid to fail starting of logical replicaiton.
A broken physical replication *before* running this tool is its responsibility?
Hmm. We might add another check that can be noticed during dry run mode.
> 10.
> ```
> nslots_new = nslots_old + dbarr.ndbs;
>
> if (nslots_new > max_replication_slots)
> {
> pg_log_error("max_replication_slots (%d) must be greater than or equal to "
> "the number of replication slots required (%d)", max_replication_slots, nslots_new);
> exit(1);
> }
> ```
>
> I think standby server must not have replication slots. Because subsequent
> pg_resetwal command discards all the WAL file, so WAL records pointed by them
> are removed. Currently pg_resetwal does not raise ERROR at that time.
Again, dry run mode might provide a message for it.
> 11.
> ```
> /*
> * Stop the subscriber if it is a standby server. Before executing the
> * transformation steps, make sure the subscriber is not running because
> * one of the steps is to modify some recovery parameters that require a
> * restart.
> */
> if (stat(pidfile, &statbuf) == 0)
> ```
>
> I kept just in case, but I'm not sure it is still needed. How do you think?
> Removing it can reduce an inclusion of pidfile.h.
Are you suggesting another way to check if the standby is up and running?
> 12.
> ```
> pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
> standby.bindir, standby.pgdata);
> rc = system(pg_ctl_cmd);
> pg_ctl_status(pg_ctl_cmd, rc, 0);
> ```
>
>
> There are two places to stop the instance. Can you divide it into a function?
Yes.
> 13.
> ```
> * A temporary replication slot is not used here to avoid keeping a
> * replication connection open (depending when base backup was taken, the
> * connection should be open for a few hours).
> */
> conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
> if (conn == NULL)
> exit(1);
> consistent_lsn = create_logical_replication_slot(conn, true,
> &dbarr.perdb[0]);
> ```
>
> I didn't notice the comment, but still not sure the reason. Why we must reserve
> the slot until pg_subscriber finishes? IIUC, the slot would be never used, it
> is created only for getting a consistent_lsn. So we do not have to keep.
> Also, just before, logical replication slots for each databases are created, so
> WAL records are surely reserved.
This comment needs to be updated. It was written at the time I was pursuing
base backup support too. It doesn't matter if you remove this transient
replication slot earlier because all of the replication slots created to the
subscriptions were created *before* the one for the consistent LSN. Hence, no
additional WAL retention due to this transient replication slot.
> 14.
>
> ```
> pg_log_info("starting the subscriber");
> start_standby_server(&standby, subport, server_start_log);
> ```
>
> This info should be in the function.
Ok.
> 15.
> ```
> /*
> * Create a subscription for each database.
> */
> for (i = 0; i < dbarr.ndbs; i++)
> ```
>
> This can be divided into a function, like create_all_subscriptions().
Ok.
> 16.
> My fault: usage() must be updated.
>
> 17. use_primary_slot_name
> ```
> if (PQntuples(res) != 1)
> {
> pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
> PQntuples(res), 1);
> return NULL;
> }
> ```
>
> Error message should be changed. I think this error means the standby has wrong primary_slot_name, right?
I refactored this code a bit but the message is the same. It detects 2 cases:
(a) you set primary_slot_name but you don't have a replication slot with the
same name and (b) a cannot-happen bug that provides > 1 rows. It is a broken
setup so maybe a hint saying so is enough.
> 18. misc
> Sometimes the pid of pg_subscriber is referred. It can be stored as global variable.
I prefer to keep getpid() call.
> 19.
> C99-style has been allowed, so loop variables like "i" can be declared in the for-statement, like
>
> ```
> for (int i = 0; i < MAX; i++)
> ```
v9 does it.
> 20.
> Some comments, docs, and outputs must be fixed when the name is changed.
Next patch.
--
Euler Taveira
EDB https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 77+ messages in thread
* RE: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-23 02:11 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-24 06:11 ` Re: speed up a logical replica setup Shubham Khanna <[email protected]>
2024-01-24 11:43 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-25 09:05 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-26 00:28 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
@ 2024-01-26 05:51 ` Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-26 05:51 UTC (permalink / raw)
To: 'Euler Taveira' <[email protected]>; [email protected] <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; 'Shubham Khanna' <[email protected]>
Dear Euler,
Thanks for updating the patch! Before reading yours, I wanted to reply some of comments.
>
I'm still thinking about replacing --subscriber-conninfo with separate items
(username, port, password?, host = socket dir). Maybe it is an overengineering.
The user can always prepare the environment to avoid unwanted and/or external
connections.
>
For me, required amount of fixes are not so different from current one. How about
others?
>
It is extremely useful because (a) you have a physical replication setup and
don't know if it is prepared for logical replication, (b) check GUCs (is
max_wal_senders sufficient for this pg_subscriber command? Or is
max_replication_slots sufficient to setup the logical replication even though I
already have some used replication slots?), (c) connectivity and (d)
credentials.
>
Yeah, it is useful for verification purpose, so let's keep this option.
But I still think the naming should be "--check". Also, there are many
`if (!dry_run)` but most of them can be removed if the process exits earlier.
Thought?
>
> 05.
> I felt we should accept some settings from enviroment variables, like pg_upgrade.
> Currently, below items should be acceted.
>
> - data directory
> - username
> - port
> - timeout
Maybe PGDATA.
>
Sorry, I cannot follow this. Did you mean that the target data directory should
be able to be specified by PGDATA? OF so, +1.
>
> 06.
> ```
> pg_logging_set_level(PG_LOG_WARNING);
> ```
>
> If the default log level is warning, there are no ways to output debug logs.
> (-v option only raises one, so INFO would be output)
> I think it should be PG_LOG_INFO.
You need to specify multiple -v options.
>
Hmm. I felt the specification was bit strange...but at least it must be
described on the documentation. pg_dump.sgml has similar lines.
>
> 08.
> Not sure, but if we want to record outputs by pg_subscriber, the sub-directory
> should be created. The name should contain the timestamp.
The log file already contains the timestamp. Why?
>
This comment assumed outputs by pg_subscriber were also recorded to a file.
In this case and if the file also has the same timestamp, I think they can be
gathered in the same place. No need if outputs are not recorded.
>
> 09.
> Not sure, but should we check max_slot_wal_keep_size of primary server? It can
> avoid to fail starting of logical replicaiton.
A broken physical replication *before* running this tool is its responsibility?
Hmm. We might add another check that can be noticed during dry run mode.
>
I thought that we should not generate any broken objects, but indeed, not sure
it is our scope. How do other think?
>
> 11.
> ```
> /*
> * Stop the subscriber if it is a standby server. Before executing the
> * transformation steps, make sure the subscriber is not running because
> * one of the steps is to modify some recovery parameters that require a
> * restart.
> */
> if (stat(pidfile, &statbuf) == 0)
> ```
>
> I kept just in case, but I'm not sure it is still needed. How do you think?
> Removing it can reduce an inclusion of pidfile.h.
Are you suggesting another way to check if the standby is up and running?
>
Running `pg_ctl stop` itself can detect whether the process has been still alive.
It would exit with 1 when the process is not there.
>
> I didn't notice the comment, but still not sure the reason. Why we must reserve
> the slot until pg_subscriber finishes? IIUC, the slot would be never used, it
> is created only for getting a consistent_lsn. So we do not have to keep.
> Also, just before, logical replication slots for each databases are created, so
> WAL records are surely reserved.
>
I want to confirm the conclusion - will you remove the creation of a transient slot?
Also, not tested, I'm now considering that we can reuse the primary_conninfo value.
We are assuming that the target server is standby and the current upstream one will
convert to publisher. In this case, the connection string is already specified as
primary_conninfo so --publisher-conninfo may not be needed. The parameter does
not contain database name, so --databases is still needed. I imagine like:
1. Parse options
2. Turn on standby
3. Verify the standby
4. Turn off standby
5. Get primary_conninfo from standby
6. Connect to primary (concat of primary_conninfo and an option is used)
7. Verify the primary
...
How do you think?
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/global/
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
@ 2024-01-24 01:29 ` Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
1 sibling, 1 reply; 77+ messages in thread
From: Euler Taveira @ 2024-01-24 01:29 UTC (permalink / raw)
To: Shlok Kyal <[email protected]>; [email protected] <[email protected]>; +Cc: [email protected] <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>
On Mon, Jan 22, 2024, at 6:30 AM, Shlok Kyal wrote:
> We fixed some of the comments posted in the thread. We have created it
> as top-up patch 0002 and 0003.
Cool.
> 0002 patch contains the following changes:
> * Add a timeout option for the recovery option, per [1]. The code was
> basically ported from pg_ctl.c.
> * Reject if the target server is not a standby, per [2]
> * Raise FATAL error if --subscriber-conninfo specifies non-local server, per [3]
> (not sure it is really needed, so feel free reject the part.)
> * Add check for max_replication_slots and wal_level; as per [4]
> * Add -u and -p options; as per [5]
> * Addressed comment except 5 and 8 in [6] and comment in [7]
My suggestion is that you create separate patches for each change. It helps
with review and alternative proposals. Some of these items conflict with what I
have in my local branch and removing one of them is time consuming. For this
one, I did the job but let's avoid rework.
> 0003 patch contains fix for bug reported in [8].
LGTM. As I said in the other email, I included it.
I'll post a new one soon.
--
Euler Taveira
EDB https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
@ 2024-01-25 23:34 ` Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Euler Taveira @ 2024-01-25 23:34 UTC (permalink / raw)
To: Shlok Kyal <[email protected]>; [email protected] <[email protected]>; +Cc: [email protected] <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>
On Tue, Jan 23, 2024, at 10:29 PM, Euler Taveira wrote:
> I'll post a new one soon.
I'm attaching another patch that fixes some of the issues pointed out by
Hayato, Shlok, and Junwang.
* publication doesn't exist. The analysis [1] was done by Hayato but I didn't
use the proposed patch. Instead I refactored the code a bit [2] and call it
from a new function (setup_publisher) that is called before the promotion.
* fix wrong path name in the initial comment [3]
* change terminology: logical replica -> physical replica [3]
* primary / standby is ready for logical replication? setup_publisher() and
setup_subscriber() check if required GUCs are set accordingly. For primary,
it checks wal_level = logical, max_replication_slots has remain replication
slots for the proposed setup and also max_wal_senders available. For standby,
it checks max_replication_slots for replication origin and also remain number
of background workers to start the subscriber.
* retain option: I extracted this one from Hayato's patch [4]
* target server must be a standby. It seems we agree that this restriction
simplifies the code a bit but can be relaxed in the future (if/when base
backup support is added.)
* recovery timeout option: I decided to include it but I think the use case is
too narrow. It helps in broken setups. However, it can be an issue in some
scenarios like time-delayed replica, large replication lag, slow hardware,
slow network. I didn't use the proposed patch [5]. Instead, I came up with a
simple one that defaults to forever. The proposed one defaults to 60 seconds
but I'm afraid that due to one of the scenarios I said in a previous
sentence, we cancel a legitimate case. Maybe we should add a message during
dry run saying that due to a replication lag, it will take longer to run.
* refactor primary_slot_name code. With the new setup_publisher and
setup_subscriber functions, I splitted the function that detects the
primary_slot_name use into 2 pieces just to avoid extra connections to have
the job done.
* remove fallback_application_name as suggested by Hayato [5] because logical
replication already includes one.
I'm still thinking about replacing --subscriber-conninfo with separate items
(username, port, password?, host = socket dir). Maybe it is an overengineering.
The user can always prepare the environment to avoid unwanted and/or external
connections.
I didn't change the name from pg_subscriber to pg_createsubscriber yet but if I
didn't hear objections about it, I'll do it in the next patch.
[1] https://www.postgresql.org/message-id/TY3PR01MB9889C5D55206DDD978627D07F5752%40TY3PR01MB9889.jpnprd0...
[2] https://www.postgresql.org/message-id/73ab86ca-3fd5-49b3-9c80-73d1525202f1%40app.fastmail.com
[3] https://www.postgresql.org/message-id/TY3PR01MB9889678E47B918F4D83A6FD8F57B2%40TY3PR01MB9889.jpnprd0...
[4] https://www.postgresql.org/message-id/TY3PR01MB9889678E47B918F4D83A6FD8F57B2%40TY3PR01MB9889.jpnprd0...
[5] https://www.postgresql.org/message-id/TY3PR01MB9889593399165B9A04106741F5662%40TY3PR01MB9889.jpnprd0...
--
Euler Taveira
EDB https://www.enterprisedb.com/
Attachments:
[text/x-patch] v9-0001-Creates-a-new-logical-replica-from-a-standby-serv.patch (73.2K, ../../[email protected]/3-v9-0001-Creates-a-new-logical-replica-from-a-standby-serv.patch)
download | inline diff:
From d9ef01a806c3d8697faa444283f19c2deaa58850 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email protected]>
Date: Mon, 5 Jun 2023 14:39:40 -0400
Subject: [PATCH v9] Creates a new logical replica from a standby server
A new tool called pg_subscriber can convert a physical replica into a
logical replica. It runs on the target server and should be able to
connect to the source server (publisher) and the target server
(subscriber).
The conversion requires a few steps. Check if the target data directory
has the same system identifier than the source data directory. Stop the
target server if it is running as a standby server. Create one
replication slot per specified database on the source server. One
additional replication slot is created at the end to get the consistent
LSN (This consistent LSN will be used as (a) a stopping point for the
recovery process and (b) a starting point for the subscriptions). Write
recovery parameters into the target data directory and start the target
server (Wait until the target server is promoted). Create one
publication (FOR ALL TABLES) per specified database on the source
server. Create one subscription per specified database on the target
server (Use replication slot and publication created in a previous step.
Don't enable the subscriptions yet). Sets the replication progress to
the consistent LSN that was got in a previous step. Enable the
subscription for each specified database on the target server. Remove
the additional replication slot that was used to get the consistent LSN.
Stop the target server. Change the system identifier from the target
server.
Depending on your workload and database size, creating a logical replica
couldn't be an option due to resource constraints (WAL backlog should be
available until all table data is synchronized). The initial data copy
and the replication progress tends to be faster on a physical replica.
The purpose of this tool is to speed up a logical replica setup.
---
doc/src/sgml/ref/allfiles.sgml | 1 +
doc/src/sgml/ref/pg_subscriber.sgml | 305 +++
doc/src/sgml/reference.sgml | 1 +
src/bin/pg_basebackup/.gitignore | 1 +
src/bin/pg_basebackup/Makefile | 8 +-
src/bin/pg_basebackup/meson.build | 19 +
src/bin/pg_basebackup/pg_subscriber.c | 1805 +++++++++++++++++
src/bin/pg_basebackup/t/040_pg_subscriber.pl | 44 +
.../t/041_pg_subscriber_standby.pl | 139 ++
9 files changed, 2322 insertions(+), 1 deletion(-)
create mode 100644 doc/src/sgml/ref/pg_subscriber.sgml
create mode 100644 src/bin/pg_basebackup/pg_subscriber.c
create mode 100644 src/bin/pg_basebackup/t/040_pg_subscriber.pl
create mode 100644 src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index 4a42999b18..3862c976d7 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -214,6 +214,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY pgResetwal SYSTEM "pg_resetwal.sgml">
<!ENTITY pgRestore SYSTEM "pg_restore.sgml">
<!ENTITY pgRewind SYSTEM "pg_rewind.sgml">
+<!ENTITY pgSubscriber SYSTEM "pg_subscriber.sgml">
<!ENTITY pgVerifyBackup SYSTEM "pg_verifybackup.sgml">
<!ENTITY pgtestfsync SYSTEM "pgtestfsync.sgml">
<!ENTITY pgtesttiming SYSTEM "pgtesttiming.sgml">
diff --git a/doc/src/sgml/ref/pg_subscriber.sgml b/doc/src/sgml/ref/pg_subscriber.sgml
new file mode 100644
index 0000000000..99d4fcee49
--- /dev/null
+++ b/doc/src/sgml/ref/pg_subscriber.sgml
@@ -0,0 +1,305 @@
+<!--
+doc/src/sgml/ref/pg_subscriber.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="app-pgsubscriber">
+ <indexterm zone="app-pgsubscriber">
+ <primary>pg_subscriber</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle><application>pg_subscriber</application></refentrytitle>
+ <manvolnum>1</manvolnum>
+ <refmiscinfo>Application</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>pg_subscriber</refname>
+ <refpurpose>convert a physical replica into a new logical replica</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_subscriber</command>
+ <arg rep="repeat"><replaceable>option</replaceable></arg>
+ </cmdsynopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+ <para>
+ <application>pg_subscriber</application> takes the publisher and subscriber
+ connection strings, a cluster directory from a physical replica and a list of
+ database names and it sets up a new logical replica using the physical
+ recovery process.
+ </para>
+
+ <para>
+ The <application>pg_subscriber</application> should be run at the target
+ server. The source server (known as publisher server) should accept logical
+ replication connections from the target server (known as subscriber server).
+ The target server should accept local logical replication connection.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Options</title>
+
+ <para>
+ <application>pg_subscriber</application> accepts the following
+ command-line arguments:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-D <replaceable class="parameter">directory</replaceable></option></term>
+ <term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term>
+ <listitem>
+ <para>
+ The target directory that contains a cluster directory from a physical
+ replica.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-P <replaceable class="parameter">conninfo</replaceable></option></term>
+ <term><option>--publisher-conninfo=<replaceable class="parameter">conninfo</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the publisher. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-S <replaceable class="parameter">conninfo</replaceable></option></term>
+ <term><option>--subscriber-conninfo=<replaceable class="parameter">conninfo</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the subscriber. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-d <replaceable class="parameter">dbname</replaceable></option></term>
+ <term><option>--database=<replaceable class="parameter">dbname</replaceable></option></term>
+ <listitem>
+ <para>
+ The database name to create the subscription. Multiple databases can be
+ selected by writing multiple <option>-d</option> switches.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
+ <listitem>
+ <para>
+ Do everything except actually modifying the target directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-r</option></term>
+ <term><option>--retain</option></term>
+ <listitem>
+ <para>
+ Retain log file even after successful completion.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-t <replaceable class="parameter">seconds</replaceable></option></term>
+ <term><option>--timeout=<replaceable class="parameter">seconds</replaceable></option></term>
+ <listitem>
+ <para>
+ The maximum number of seconds to wait for recovery to end. Setting to 0
+ disables. The default is 0.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-v</option></term>
+ <term><option>--verbose</option></term>
+ <listitem>
+ <para>
+ Enables verbose mode. This will cause
+ <application>pg_subscriber</application> to output progress messages
+ and detailed information about each step.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ Other options are also available:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-V</option></term>
+ <term><option>--version</option></term>
+ <listitem>
+ <para>
+ Print the <application>pg_subscriber</application> version and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-?</option></term>
+ <term><option>--help</option></term>
+ <listitem>
+ <para>
+ Show help about <application>pg_subscriber</application> command
+ line arguments, and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Notes</title>
+
+ <para>
+ The transformation proceeds in the following steps:
+ </para>
+
+ <procedure>
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the given target data
+ directory has the same system identifier than the source data directory.
+ Since it uses the recovery process as one of the steps, it starts the
+ target server as a replica from the source server. If the system
+ identifier is not the same, <application>pg_subscriber</application> will
+ terminate with an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the target data
+ directory is used by a physical replica. Stop the physical replica if it is
+ running. One of the next steps is to add some recovery parameters that
+ requires a server start. This step avoids an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one replication slot for
+ each specified database on the source server. The replication slot name
+ contains a <literal>pg_subscriber</literal> prefix. These replication
+ slots will be used by the subscriptions in a future step. Another
+ replication slot is used to get a consistent start location. This
+ consistent LSN will be used as a stopping point in the <xref
+ linkend="guc-recovery-target-lsn"/> parameter and by the
+ subscriptions as a replication starting point. It guarantees that no
+ transaction will be lost.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> writes recovery parameters into
+ the target data directory and start the target server. It specifies a LSN
+ (consistent LSN that was obtained in the previous step) of write-ahead
+ log location up to which recovery will proceed. It also specifies
+ <literal>promote</literal> as the action that the server should take once
+ the recovery target is reached. This step finishes once the server ends
+ standby mode and is accepting read-write operations.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Next, <application>pg_subscriber</application> creates one publication
+ for each specified database on the source server. Each publication
+ replicates changes for all tables in the database. The publication name
+ contains a <literal>pg_subscriber</literal> prefix. These publication
+ will be used by a corresponding subscription in a next step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one subscription for
+ each specified database on the target server. Each subscription name
+ contains a <literal>pg_subscriber</literal> prefix. The replication slot
+ name is identical to the subscription name. It does not copy existing data
+ from the source server. It does not create a replication slot. Instead, it
+ uses the replication slot that was created in a previous step. The
+ subscription is created but it is not enabled yet. The reason is the
+ replication progress must be set to the consistent LSN but replication
+ origin name contains the subscription oid in its name. Hence, the
+ subscription will be enabled in a separate step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> sets the replication progress to
+ the consistent LSN that was obtained in a previous step. When the target
+ server started the recovery process, it caught up to the consistent LSN.
+ This is the exact LSN to be used as a initial location for each
+ subscription.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Finally, <application>pg_subscriber</application> enables the subscription
+ for each specified database on the target server. The subscription starts
+ streaming from the consistent LSN.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> removes the additional replication
+ slot that was used to get the consistent LSN on the source server.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> stops the target server to change
+ its system identifier.
+ </para>
+ </step>
+ </procedure>
+ </refsect1>
+
+ <refsect1>
+ <title>Examples</title>
+
+ <para>
+ To create a logical replica for databases <literal>hr</literal> and
+ <literal>finance</literal> from a physical replica at <literal>foo</literal>:
+<screen>
+<prompt>$</prompt> <userinput>pg_subscriber -D /usr/local/pgsql/data -P "host=foo" -S "host=localhost" -d hr -d finance</userinput>
+</screen>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="app-pgbasebackup"/></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index aa94f6adf6..266f4e515a 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -285,6 +285,7 @@
&pgCtl;
&pgResetwal;
&pgRewind;
+ &pgSubscriber;
&pgtestfsync;
&pgtesttiming;
&pgupgrade;
diff --git a/src/bin/pg_basebackup/.gitignore b/src/bin/pg_basebackup/.gitignore
index 26048bdbd8..0e5384a1d5 100644
--- a/src/bin/pg_basebackup/.gitignore
+++ b/src/bin/pg_basebackup/.gitignore
@@ -1,5 +1,6 @@
/pg_basebackup
/pg_receivewal
/pg_recvlogical
+/pg_subscriber
/tmp_check/
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index abfb6440ec..f6281b7676 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -44,7 +44,7 @@ BBOBJS = \
bbstreamer_tar.o \
bbstreamer_zstd.o
-all: pg_basebackup pg_receivewal pg_recvlogical
+all: pg_basebackup pg_receivewal pg_recvlogical pg_subscriber
pg_basebackup: $(BBOBJS) $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) $(BBOBJS) $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
@@ -55,10 +55,14 @@ pg_receivewal: pg_receivewal.o $(OBJS) | submake-libpq submake-libpgport submake
pg_recvlogical: pg_recvlogical.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_recvlogical.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_subscriber: $(WIN32RES) pg_subscriber.o | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
install: all installdirs
$(INSTALL_PROGRAM) pg_basebackup$(X) '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
$(INSTALL_PROGRAM) pg_receivewal$(X) '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
$(INSTALL_PROGRAM) pg_recvlogical$(X) '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ $(INSTALL_PROGRAM) pg_subscriber$(X) '$(DESTDIR)$(bindir)/pg_subscriber$(X)'
installdirs:
$(MKDIR_P) '$(DESTDIR)$(bindir)'
@@ -67,10 +71,12 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ rm -f '$(DESTDIR)$(bindir)/pg_subscriber$(X)'
clean distclean:
rm -f pg_basebackup$(X) pg_receivewal$(X) pg_recvlogical$(X) \
$(BBOBJS) pg_receivewal.o pg_recvlogical.o \
+ pg_subscriber$(X) pg_subscriber.o \
$(OBJS)
rm -rf tmp_check
diff --git a/src/bin/pg_basebackup/meson.build b/src/bin/pg_basebackup/meson.build
index f7e60e6670..ccfd7bb7a5 100644
--- a/src/bin/pg_basebackup/meson.build
+++ b/src/bin/pg_basebackup/meson.build
@@ -75,6 +75,23 @@ pg_recvlogical = executable('pg_recvlogical',
)
bin_targets += pg_recvlogical
+pg_subscriber_sources = files(
+ 'pg_subscriber.c'
+)
+
+if host_system == 'windows'
+ pg_subscriber_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'pg_subscriber',
+ '--FILEDESC', 'pg_subscriber - create a new logical replica from a standby server',])
+endif
+
+pg_subscriber = executable('pg_subscriber',
+ pg_subscriber_sources,
+ dependencies: [frontend_code, libpq],
+ kwargs: default_bin_args,
+)
+bin_targets += pg_subscriber
+
tests += {
'name': 'pg_basebackup',
'sd': meson.current_source_dir(),
@@ -89,6 +106,8 @@ tests += {
't/011_in_place_tablespace.pl',
't/020_pg_receivewal.pl',
't/030_pg_recvlogical.pl',
+ 't/040_pg_subscriber.pl',
+ 't/041_pg_subscriber_standby.pl',
],
},
}
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
new file mode 100644
index 0000000000..cb97dbda5e
--- /dev/null
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -0,0 +1,1805 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_subscriber.c
+ * Create a new logical replica from a standby server
+ *
+ * Copyright (C) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_basebackup/pg_subscriber.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include <signal.h>
+#include <sys/stat.h>
+#include <sys/time.h>
+#include <sys/wait.h>
+#include <time.h>
+
+#include "access/xlogdefs.h"
+#include "catalog/pg_control.h"
+#include "common/connect.h"
+#include "common/controldata_utils.h"
+#include "common/file_perm.h"
+#include "common/file_utils.h"
+#include "common/logging.h"
+#include "fe_utils/recovery_gen.h"
+#include "fe_utils/simple_list.h"
+#include "getopt_long.h"
+#include "utils/pidfile.h"
+
+#define PGS_OUTPUT_DIR "pg_subscriber_output.d"
+
+typedef struct LogicalRepInfo
+{
+ Oid oid; /* database OID */
+ char *dbname; /* database name */
+ char *pubconninfo; /* publication connection string for logical
+ * replication */
+ char *subconninfo; /* subscription connection string for logical
+ * replication */
+ char *pubname; /* publication name */
+ char *subname; /* subscription name (also replication slot
+ * name) */
+
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepInfo;
+
+static void cleanup_objects_atexit(void);
+static void usage();
+static char *get_base_conninfo(char *conninfo, char *dbname,
+ const char *noderole);
+static bool get_exec_path(const char *path);
+static bool check_data_directory(const char *datadir);
+static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
+static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
+static PGconn *connect_database(const char *conninfo);
+static void disconnect_database(PGconn *conn);
+static uint64 get_sysid_from_conn(const char *conninfo);
+static uint64 get_control_from_datadir(const char *datadir);
+static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
+static bool setup_publisher(LogicalRepInfo *dbinfo);
+static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name);
+static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
+static void wait_for_end_recovery(const char *conninfo);
+static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+
+#define USEC_PER_SEC 1000000
+#define WAIT_INTERVAL 1 /* 1 second */
+
+/* Options */
+static const char *progname;
+
+static char *subscriber_dir = NULL;
+static char *pub_conninfo_str = NULL;
+static char *sub_conninfo_str = NULL;
+static SimpleStringList database_names = {NULL, NULL};
+static char *primary_slot_name = NULL;
+static bool dry_run = false;
+static bool retain = false;
+static int recovery_timeout = 0;
+
+static bool success = false;
+
+static char *pg_ctl_path = NULL;
+static char *pg_resetwal_path = NULL;
+
+static LogicalRepInfo *dbinfo;
+static int num_dbs = 0;
+
+static char temp_replslot[NAMEDATALEN] = {0};
+static bool made_transient_replslot = false;
+
+enum WaitPMResult
+{
+ POSTMASTER_READY,
+ POSTMASTER_STANDBY,
+ POSTMASTER_STILL_STARTING,
+ POSTMASTER_FAILED
+};
+
+
+/*
+ * Cleanup objects that were created by pg_subscriber if there is an error.
+ *
+ * Replication slots, publications and subscriptions are created. Depending on
+ * the step it failed, it should remove the already created objects if it is
+ * possible (sometimes it won't work due to a connection issue).
+ */
+static void
+cleanup_objects_atexit(void)
+{
+ PGconn *conn;
+ int i;
+
+ if (success)
+ return;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ if (dbinfo[i].made_subscription)
+ {
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn != NULL)
+ {
+ drop_subscription(conn, &dbinfo[i]);
+ disconnect_database(conn);
+ }
+ }
+
+ if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ {
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn != NULL)
+ {
+ if (dbinfo[i].made_publication)
+ drop_publication(conn, &dbinfo[i]);
+ if (dbinfo[i].made_replslot)
+ drop_replication_slot(conn, &dbinfo[i], NULL);
+ disconnect_database(conn);
+ }
+ }
+ }
+
+ if (made_transient_replslot)
+ {
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn != NULL)
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ disconnect_database(conn);
+ }
+ }
+}
+
+static void
+usage(void)
+{
+ printf(_("%s creates a new logical replica from a standby server.\n\n"),
+ progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION]...\n"), progname);
+ printf(_("\nOptions:\n"));
+ printf(_(" -D, --pgdata=DATADIR location for the subscriber data directory\n"));
+ printf(_(" -P, --publisher-conninfo=CONNINFO publisher connection string\n"));
+ printf(_(" -S, --subscriber-conninfo=CONNINFO subscriber connection string\n"));
+ printf(_(" -d, --database=DBNAME database to create a subscription\n"));
+ printf(_(" -n, --dry-run stop before modifying anything\n"));
+ printf(_(" -t, --recovery-timeout=SECS seconds to wait for recovery to end\n"));
+ printf(_(" -r, --retain retain log file after success\n"));
+ printf(_(" -v, --verbose output verbose messages\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
+ printf(_("\nReport bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+ printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
+
+/*
+ * Validate a connection string. Returns a base connection string that is a
+ * connection string without a database name.
+ * Since we might process multiple databases, each database name will be
+ * appended to this base connection string to provide a final connection string.
+ * If the second argument (dbname) is not null, returns dbname if the provided
+ * connection string contains it. If option --database is not provided, uses
+ * dbname as the only database to setup the logical replica.
+ * It is the caller's responsibility to free the returned connection string and
+ * dbname.
+ */
+static char *
+get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ PQconninfoOption *conn_opts = NULL;
+ PQconninfoOption *conn_opt;
+ char *errmsg = NULL;
+ char *ret;
+ int i;
+
+ pg_log_info("validating connection string on %s", noderole);
+
+ conn_opts = PQconninfoParse(conninfo, &errmsg);
+ if (conn_opts == NULL)
+ {
+ pg_log_error("could not parse connection string: %s", errmsg);
+ return NULL;
+ }
+
+ i = 0;
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (strcmp(conn_opt->keyword, "dbname") == 0 && conn_opt->val != NULL)
+ {
+ if (dbname)
+ dbname = pg_strdup(conn_opt->val);
+ continue;
+ }
+
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0')
+ {
+ if (i > 0)
+ appendPQExpBufferChar(buf, ' ');
+ appendPQExpBuffer(buf, "%s=%s", conn_opt->keyword, conn_opt->val);
+ i++;
+ }
+ }
+
+ ret = pg_strdup(buf->data);
+
+ destroyPQExpBuffer(buf);
+ PQconninfoFree(conn_opts);
+
+ return ret;
+}
+
+/*
+ * Get the absolute path from other PostgreSQL binaries (pg_ctl and
+ * pg_resetwal) that is used by it.
+ */
+static bool
+get_exec_path(const char *path)
+{
+ int rc;
+
+ pg_ctl_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_ctl",
+ "pg_ctl (PostgreSQL) " PG_VERSION "\n",
+ pg_ctl_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_ctl", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_ctl", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
+
+ pg_resetwal_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_resetwal",
+ "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
+ pg_resetwal_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_resetwal", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_resetwal", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+
+ return true;
+}
+
+/*
+ * Is it a cluster directory? These are preliminary checks. It is far from
+ * making an accurate check. If it is not a clone from the publisher, it will
+ * eventually fail in a future step.
+ */
+static bool
+check_data_directory(const char *datadir)
+{
+ struct stat statbuf;
+ char versionfile[MAXPGPATH];
+
+ pg_log_info("checking if directory \"%s\" is a cluster data directory",
+ datadir);
+
+ if (stat(datadir, &statbuf) != 0)
+ {
+ if (errno == ENOENT)
+ pg_log_error("data directory \"%s\" does not exist", datadir);
+ else
+ pg_log_error("could not access directory \"%s\": %s", datadir, strerror(errno));
+
+ return false;
+ }
+
+ snprintf(versionfile, MAXPGPATH, "%s/PG_VERSION", datadir);
+ if (stat(versionfile, &statbuf) != 0 && errno == ENOENT)
+ {
+ pg_log_error("directory \"%s\" is not a database cluster directory", datadir);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Append database name into a base connection string.
+ *
+ * dbname is the only parameter that changes so it is not included in the base
+ * connection string. This function concatenates dbname to build a "real"
+ * connection string.
+ */
+static char *
+concat_conninfo_dbname(const char *conninfo, const char *dbname)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *ret;
+
+ Assert(conninfo != NULL);
+
+ appendPQExpBufferStr(buf, conninfo);
+ appendPQExpBuffer(buf, " dbname=%s", dbname);
+
+ ret = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+
+ return ret;
+}
+
+/*
+ * Store publication and subscription information.
+ */
+static LogicalRepInfo *
+store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+{
+ LogicalRepInfo *dbinfo;
+ SimpleStringListCell *cell;
+ int i = 0;
+
+ dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+
+ for (cell = database_names.head; cell; cell = cell->next)
+ {
+ char *conninfo;
+
+ /* Publisher. */
+ conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
+ dbinfo[i].pubconninfo = conninfo;
+ dbinfo[i].dbname = cell->val;
+ dbinfo[i].made_replslot = false;
+ dbinfo[i].made_publication = false;
+ dbinfo[i].made_subscription = false;
+ /* other struct fields will be filled later. */
+
+ /* Subscriber. */
+ conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
+ dbinfo[i].subconninfo = conninfo;
+
+ i++;
+ }
+
+ return dbinfo;
+}
+
+static PGconn *
+connect_database(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ const char *rconninfo;
+
+ /* logical replication mode */
+ rconninfo = psprintf("%s replication=database", conninfo);
+
+ conn = PQconnectdb(rconninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ pg_log_error("connection to database failed: %s", PQerrorMessage(conn));
+ return NULL;
+ }
+
+ /* secure search_path */
+ res = PQexec(conn, ALWAYS_SECURE_SEARCH_PATH_SQL);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not clear search_path: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+ PQclear(res);
+
+ return conn;
+}
+
+static void
+disconnect_database(PGconn *conn)
+{
+ Assert(conn != NULL);
+
+ PQfinish(conn);
+}
+
+/*
+ * Obtain the system identifier using the provided connection. It will be used
+ * to compare if a data directory is a clone of another one.
+ */
+static uint64
+get_sysid_from_conn(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from publisher");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "IDENTIFY_SYSTEM");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not send replication command \"%s\": %s",
+ "IDENTIFY_SYSTEM", PQresultErrorMessage(res));
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+ if (PQntuples(res) != 1 || PQnfields(res) < 3)
+ {
+ pg_log_error("could not identify system: got %d rows and %d fields, expected %d rows and %d or more fields",
+ PQntuples(res), PQnfields(res), 1, 3);
+
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+
+ sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+
+ pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+
+ disconnect_database(conn);
+
+ return sysid;
+}
+
+/*
+ * Obtain the system identifier from control file. It will be used to compare
+ * if a data directory is a clone of another one. This routine is used locally
+ * and avoids a replication connection.
+ */
+static uint64
+get_control_from_datadir(const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ sysid = cf->system_identifier;
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+
+ pfree(cf);
+
+ return sysid;
+}
+
+/*
+ * Modify the system identifier. Since a standby server preserves the system
+ * identifier, it makes sense to change it to avoid situations in which WAL
+ * files from one of the systems might be used in the other one.
+ */
+static void
+modify_sysid(const char *pg_resetwal_path, const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ struct timeval tv;
+
+ char *cmd_str;
+ int rc;
+
+ pg_log_info("modifying system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ /*
+ * Select a new system identifier.
+ *
+ * XXX this code was extracted from BootStrapXLOG().
+ */
+ gettimeofday(&tv, NULL);
+ cf->system_identifier = ((uint64) tv.tv_sec) << 32;
+ cf->system_identifier |= ((uint64) tv.tv_usec) << 12;
+ cf->system_identifier |= getpid() & 0xFFF;
+
+ if (!dry_run)
+ update_controlfile(datadir, cf, true);
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) cf->system_identifier);
+
+ pg_log_info("running pg_resetwal on the subscriber");
+
+ cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+
+ pg_log_debug("command is: %s", cmd_str);
+
+ if (!dry_run)
+ {
+ rc = system(cmd_str);
+ if (rc == 0)
+ pg_log_info("subscriber successfully changed the system identifier");
+ else
+ pg_log_error("subscriber failed to change system identifier: exit code: %d", rc);
+ }
+
+ pfree(cf);
+}
+
+/*
+ * Is the source server ready for logical replication? If so, create the
+ * publications and replication slots in preparation for logical replication.
+ */
+static bool
+setup_publisher(LogicalRepInfo *dbinfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ PQExpBuffer str = createPQExpBuffer();
+
+ char *wal_level;
+ int max_repslots;
+ int cur_repslots;
+ int max_walsenders;
+ int cur_walsenders;
+
+ pg_log_info("checking settings on publisher");
+
+ /*
+ * Logical replication requires a few parameters to be set on publisher.
+ * Since these parameters are not a requirement for physical replication,
+ * we should check it to make sure it won't fail.
+ *
+ * wal_level = logical
+ * max_replication_slots >= current + number of dbs to be converted
+ * max_wal_senders >= current + number of dbs to be converted
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "WITH wl AS (SELECT setting AS wallevel FROM pg_settings WHERE name = 'wal_level'),"
+ " total_mrs AS (SELECT setting AS tmrs FROM pg_settings WHERE name = 'max_replication_slots'),"
+ " cur_mrs AS (SELECT count(*) AS cmrs FROM pg_replication_slots),"
+ " total_mws AS (SELECT setting AS tmws FROM pg_settings WHERE name = 'max_wal_senders'),"
+ " cur_mws AS (SELECT count(*) AS cmws FROM pg_stat_activity WHERE backend_type = 'walsender')"
+ "SELECT wallevel, tmrs, cmrs, tmws, cmws FROM wl, total_mrs, cur_mrs, total_mws, cur_mws");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publisher settings: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ wal_level = strdup(PQgetvalue(res, 0, 0));
+ max_repslots = atoi(PQgetvalue(res, 0, 1));
+ cur_repslots = atoi(PQgetvalue(res, 0, 2));
+ max_walsenders = atoi(PQgetvalue(res, 0, 3));
+ cur_walsenders = atoi(PQgetvalue(res, 0, 4));
+
+ PQclear(res);
+
+ pg_log_debug("subscriber: wal_level: %s", wal_level);
+ pg_log_debug("subscriber: max_replication_slots: %d", max_repslots);
+ pg_log_debug("subscriber: current replication slots: %d", cur_repslots);
+ pg_log_debug("subscriber: max_wal_senders: %d", max_walsenders);
+ pg_log_debug("subscriber: current wal senders: %d", cur_walsenders);
+
+ /*
+ * If standby sets primary_slot_name, check if this replication slot is in
+ * use on primary for WAL retention purposes. This replication slot has no
+ * use after the transformation, hence, it will be removed at the end of
+ * this process.
+ */
+ if (primary_slot_name)
+ {
+ appendPQExpBuffer(str,
+ "SELECT 1 FROM pg_replication_slots WHERE active AND slot_name = '%s'", primary_slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain replication slot information: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
+ PQntuples(res), 1);
+ pg_free(primary_slot_name); /* it is not being used. */
+ primary_slot_name = NULL;
+ return false;
+ }
+ else
+ {
+ pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+ }
+
+ PQclear(res);
+ }
+
+ disconnect_database(conn);
+
+ if (strcmp(wal_level, "logical") != 0)
+ {
+ pg_log_error("publisher requires wal_level >= logical");
+ return false;
+ }
+
+ if (max_repslots - cur_repslots < num_dbs)
+ {
+ pg_log_error("publisher requires %d replication slots, but only %d remain", num_dbs, max_repslots - cur_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", cur_repslots + num_dbs);
+ return false;
+ }
+
+ if (max_walsenders - cur_walsenders < num_dbs)
+ {
+ pg_log_error("publisher requires %d wal sender processes, but only %d remain", num_dbs, max_walsenders - cur_walsenders);
+ pg_log_error_hint("Consider increasing max_wal_senders to at least %d.", cur_walsenders + num_dbs);
+ return false;
+ }
+
+ for (int i = 0; i < num_dbs; i++)
+ {
+ char pubname[NAMEDATALEN];
+ char replslotname[NAMEDATALEN];
+
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT oid FROM pg_catalog.pg_database WHERE datname = current_database()");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain database OID: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain database OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ return false;
+ }
+
+ /* Remember database OID. */
+ dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+
+ PQclear(res);
+
+ /*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 35 characters (14 + 10 +
+ * '\0').
+ */
+ snprintf(pubname, sizeof(pubname), "pg_subscriber_%u", dbinfo[i].oid);
+ dbinfo[i].pubname = pg_strdup(pubname);
+
+ /*
+ * Create publication on publisher. This step should be executed
+ * *before* promoting the subscriber to avoid any transactions between
+ * consistent LSN and the new publication rows (such transactions
+ * wouldn't see the new publication rows resulting in an error).
+ */
+ create_publication(conn, &dbinfo[i]);
+
+ /*
+ * Build the replication slot name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 36
+ * characters (14 + 10 + 1 + 10 + '\0'). System identifier is included
+ * to reduce the probability of collision. By default, subscription
+ * name is used as replication slot name.
+ */
+ snprintf(replslotname, sizeof(replslotname),
+ "pg_subscriber_%u_%d",
+ dbinfo[i].oid,
+ (int) getpid());
+ dbinfo[i].subname = pg_strdup(replslotname);
+
+ /* Create replication slot on publisher. */
+ if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
+ pg_log_info("create replication slot \"%s\" on publisher", replslotname);
+ else
+ return false;
+
+ disconnect_database(conn);
+ }
+
+ return true;
+}
+
+/*
+ * Is the target server ready for logical replication?
+ */
+static bool
+setup_subscriber(LogicalRepInfo *dbinfo)
+{
+ PGconn *conn;
+ PGresult *res;
+
+ int max_lrworkers;
+ int max_repslots;
+ int max_wprocs;
+
+ pg_log_info("checking settings on subscriber");
+
+ /*
+ * Logical replication requires a few parameters to be set on subscriber.
+ * Since these parameters are not a requirement for physical replication,
+ * we should check it to make sure it won't fail.
+ *
+ * max_replication_slots >= number of dbs to be converted
+ * max_logical_replication_workers >= number of dbs to be converted
+ * max_worker_processes >= 1 + number of dbs to be converted
+ */
+ conn = connect_database(dbinfo[0].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT setting FROM pg_settings WHERE name IN ('max_logical_replication_workers', 'max_replication_slots', 'max_worker_processes', 'primary_slot_name') ORDER BY name");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscriber settings: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ max_lrworkers = atoi(PQgetvalue(res, 0, 0));
+ max_repslots = atoi(PQgetvalue(res, 1, 0));
+ max_wprocs = atoi(PQgetvalue(res, 2, 0));
+ if (strcmp(PQgetvalue(res, 3, 0), "") != 0)
+ primary_slot_name = pg_strdup(PQgetvalue(res, 3, 0));
+
+ pg_log_debug("subscriber: max_logical_replication_workers: %d", max_lrworkers);
+ pg_log_debug("subscriber: max_replication_slots: %d", max_repslots);
+ pg_log_debug("subscriber: max_worker_processes: %d", max_wprocs);
+ pg_log_debug("subscriber: primary_slot_name: %s", primary_slot_name);
+
+ PQclear(res);
+
+ disconnect_database(conn);
+
+ if (max_repslots < num_dbs)
+ {
+ pg_log_error("subscriber requires %d replication slots, but only %d remain", num_dbs, max_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", num_dbs);
+ return false;
+ }
+
+ if (max_lrworkers < num_dbs)
+ {
+ pg_log_error("subscriber requires %d logical replication workers, but only %d remain", num_dbs, max_lrworkers);
+ pg_log_error_hint("Consider increasing max_logical_replication_workers to at least %d.", num_dbs);
+ return false;
+ }
+
+ if (max_wprocs < num_dbs + 1)
+ {
+ pg_log_error("subscriber requires %d worker processes, but only %d remain", num_dbs + 1, max_wprocs);
+ pg_log_error_hint("Consider increasing max_worker_processes to at least %d.", num_dbs + 1);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Create a logical replication slot and returns a consistent LSN. The returned
+ * LSN might be used to catch up the subscriber up to the required point.
+ *
+ * CreateReplicationSlot() is not used because it does not provide the one-row
+ * result set that contains the consistent LSN.
+ */
+static char *
+create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res = NULL;
+ char *lsn = NULL;
+ bool transient_replslot = false;
+
+ Assert(conn != NULL);
+
+ /*
+ * If no slot name is informed, it is a transient replication slot used
+ * only for catch up purposes.
+ */
+ if (slot_name[0] == '\0')
+ {
+ snprintf(slot_name, NAMEDATALEN, "pg_subscriber_%d_startpoint",
+ (int) getpid());
+ transient_replslot = true;
+ }
+
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
+ appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQresultErrorMessage(res));
+ return lsn;
+ }
+ }
+
+ /* for cleanup purposes */
+ if (transient_replslot)
+ made_transient_replslot = true;
+ else
+ dbinfo->made_replslot = true;
+
+ if (!dry_run)
+ {
+ lsn = pg_strdup(PQgetvalue(res, 0, 1));
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+
+ return lsn;
+}
+
+static void
+drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Reports a suitable message if pg_ctl fails.
+ */
+static void
+pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
+{
+ if (rc != 0)
+ {
+ if (WIFEXITED(rc))
+ {
+ pg_log_error("pg_ctl failed with exit code %d", WEXITSTATUS(rc));
+ }
+ else if (WIFSIGNALED(rc))
+ {
+#if defined(WIN32)
+ pg_log_error("pg_ctl was terminated by exception 0x%X", WTERMSIG(rc));
+ pg_log_error_detail("See C include file \"ntstatus.h\" for a description of the hexadecimal value.");
+#else
+ pg_log_error("pg_ctl was terminated by signal %d: %s",
+ WTERMSIG(rc), pg_strsignal(WTERMSIG(rc)));
+#endif
+ }
+ else
+ {
+ pg_log_error("pg_ctl exited with unrecognized status %d", rc);
+ }
+
+ pg_log_error_detail("The failed command was: %s", pg_ctl_cmd);
+ exit(1);
+ }
+
+ if (action)
+ pg_log_info("postmaster was started");
+ else
+ pg_log_info("postmaster was stopped");
+}
+
+/*
+ * Returns after the server finishes the recovery process.
+ *
+ * If recovery_timeout option is set, terminate abnormally without finishing
+ * the recovery process. By default, it waits forever.
+ */
+static void
+wait_for_end_recovery(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ int status = POSTMASTER_STILL_STARTING;
+ int timer = 0;
+
+ char *pg_ctl_cmd;
+ int rc;
+
+ pg_log_info("waiting the postmaster to reach the consistent state");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ for (;;)
+ {
+ bool in_recovery;
+
+ res = PQexec(conn, "SELECT pg_catalog.pg_is_in_recovery()");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain recovery progress");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("unexpected result from pg_is_in_recovery function");
+ exit(1);
+ }
+
+ in_recovery = (strcmp(PQgetvalue(res, 0, 0), "t") == 0);
+
+ PQclear(res);
+
+ /*
+ * Does the recovery process finish? In dry run mode, there is no
+ * recovery mode. Bail out as the recovery process has ended.
+ */
+ if (!in_recovery || dry_run)
+ {
+ status = POSTMASTER_READY;
+ break;
+ }
+
+ /*
+ * Bail out after recovery_timeout seconds if this option is set.
+ */
+ if (recovery_timeout > 0 && timer >= recovery_timeout)
+ {
+ pg_log_error("recovery timed out");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+ exit(1);
+ }
+
+ /* Keep waiting. */
+ pg_usleep(WAIT_INTERVAL * USEC_PER_SEC);
+
+ timer += WAIT_INTERVAL;
+ }
+
+ disconnect_database(conn);
+
+ if (status == POSTMASTER_STILL_STARTING)
+ {
+ pg_log_error("server did not end recovery");
+ exit(1);
+ }
+
+ pg_log_info("postmaster reached the consistent state");
+}
+
+/*
+ * Create a publication that includes all tables in the database.
+ */
+static void
+create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ /* Check if the publication needs to be created. */
+ appendPQExpBuffer(str,
+ "SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
+ dbinfo->pubname);
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publication information: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) == 1)
+ {
+ /*
+ * If publication name already exists and puballtables is true, let's
+ * use it. A previous run of pg_subscriber must have created this
+ * publication. Bail out.
+ */
+ if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
+ {
+ pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ return;
+ }
+ else
+ {
+ /*
+ * Unfortunately, if it reaches this code path, it will always
+ * fail (unless you decide to change the existing publication
+ * name). That's bad but it is very unlikely that the user will
+ * choose a name with pg_subscriber_ prefix followed by the exact
+ * database oid in which puballtables is false.
+ */
+ pg_log_error("publication \"%s\" does not replicate changes for all tables",
+ dbinfo->pubname);
+ pg_log_error_hint("Consider renaming this publication.");
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(str);
+
+ pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
+ dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_publication = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove publication if it couldn't finish all steps.
+ */
+static void
+drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Create a subscription with some predefined options.
+ *
+ * A replication slot was already created in a previous step. Let's use it. By
+ * default, the subscription name is used as replication slot name. It is
+ * not required to copy data. The subscription will be created but it will not
+ * be enabled now. That's because the replication progress must be set and the
+ * replication origin name (one of the function arguments) contains the
+ * subscription OID in its name. Once the subscription is created,
+ * set_replication_progress() can obtain the chosen origin name and set up its
+ * initial location.
+ */
+static void
+create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str,
+ "CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
+ "WITH (create_slot = false, copy_data = false, enabled = false)",
+ dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
+ dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_subscription = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove subscription if it couldn't finish all steps.
+ */
+static void
+drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Sets the replication progress to the consistent LSN.
+ *
+ * The subscriber caught up to the consistent LSN provided by the temporary
+ * replication slot. The goal is to set up the initial location for the logical
+ * replication that is the exact LSN that the subscriber was promoted. Once the
+ * subscription is enabled it will start streaming from that location onwards.
+ * In dry run mode, the subscription OID and LSN are set to invalid values for
+ * printing purposes.
+ */
+static void
+set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+ Oid suboid;
+ char originname[NAMEDATALEN];
+ char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+
+ Assert(conn != NULL);
+
+ appendPQExpBuffer(str,
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscription OID: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1 && !dry_run)
+ {
+ pg_log_error("could not obtain subscription OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (dry_run)
+ {
+ suboid = InvalidOid;
+ snprintf(lsnstr, sizeof(lsnstr), "%X/%X", LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ suboid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ snprintf(lsnstr, sizeof(lsnstr), "%s", lsn);
+ }
+
+ /*
+ * The origin name is defined as pg_%u. %u is the subscription OID. See
+ * ApplyWorkerMain().
+ */
+ snprintf(originname, sizeof(originname), "pg_%u", suboid);
+
+ PQclear(res);
+
+ pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
+ originname, lsnstr, dbinfo->dbname);
+
+ resetPQExpBuffer(str);
+ appendPQExpBuffer(str,
+ "SELECT pg_catalog.pg_replication_origin_advance('%s', '%s')", originname, lsnstr);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not set replication progress for the subscription \"%s\": %s",
+ dbinfo->subname, PQresultErrorMessage(res));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Enables the subscription.
+ *
+ * The subscription was created in a previous step but it was disabled. After
+ * adjusting the initial location, enabling the subscription is the last step
+ * of this setup.
+ */
+static void
+enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+int
+main(int argc, char **argv)
+{
+ static struct option long_options[] =
+ {
+ {"help", no_argument, NULL, '?'},
+ {"version", no_argument, NULL, 'V'},
+ {"pgdata", required_argument, NULL, 'D'},
+ {"publisher-conninfo", required_argument, NULL, 'P'},
+ {"subscriber-conninfo", required_argument, NULL, 'S'},
+ {"database", required_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"recovery-timeout", required_argument, NULL, 't'},
+ {"retain", no_argument, NULL, 'r'},
+ {"verbose", no_argument, NULL, 'v'},
+ {NULL, 0, NULL, 0}
+ };
+
+ int c;
+ int option_index;
+ int rc;
+
+ char *pg_ctl_cmd;
+
+ char *base_dir;
+ char *server_start_log;
+
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+
+ char *pub_base_conninfo = NULL;
+ char *sub_base_conninfo = NULL;
+ char *dbname_conninfo = NULL;
+
+ uint64 pub_sysid;
+ uint64 sub_sysid;
+ struct stat statbuf;
+
+ PGconn *conn;
+ char *consistent_lsn;
+
+ PQExpBuffer recoveryconfcontents = NULL;
+
+ char pidfile[MAXPGPATH];
+
+ int i;
+
+ pg_logging_init(argv[0]);
+ pg_logging_set_level(PG_LOG_WARNING);
+ progname = get_progname(argv[0]);
+ set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_subscriber"));
+
+ if (argc > 1)
+ {
+ if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+ {
+ usage();
+ exit(0);
+ }
+ else if (strcmp(argv[1], "-V") == 0
+ || strcmp(argv[1], "--version") == 0)
+ {
+ puts("pg_subscriber (PostgreSQL) " PG_VERSION);
+ exit(0);
+ }
+ }
+
+ atexit(cleanup_objects_atexit);
+
+ /*
+ * Don't allow it to be run as root. It uses pg_ctl which does not allow
+ * it either.
+ */
+#ifndef WIN32
+ if (geteuid() == 0)
+ {
+ pg_log_error("cannot be executed by \"root\"");
+ pg_log_error_hint("You must run %s as the PostgreSQL superuser.",
+ progname);
+ exit(1);
+ }
+#endif
+
+ while ((c = getopt_long(argc, argv, "D:P:S:d:nv",
+ long_options, &option_index)) != -1)
+ {
+ switch (c)
+ {
+ case 'D':
+ subscriber_dir = pg_strdup(optarg);
+ break;
+ case 'P':
+ pub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'S':
+ sub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'd':
+ /* Ignore duplicated database names. */
+ if (!simple_string_list_member(&database_names, optarg))
+ {
+ simple_string_list_append(&database_names, optarg);
+ num_dbs++;
+ }
+ break;
+ case 'n':
+ dry_run = true;
+ break;
+ case 'r':
+ retain = true;
+ break;
+ case 't':
+ recovery_timeout = atoi(optarg);
+ break;
+ case 'v':
+ pg_logging_increase_verbosity();
+ break;
+ default:
+ /* getopt_long already emitted a complaint */
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Any non-option arguments?
+ */
+ if (optind < argc)
+ {
+ pg_log_error("too many command-line arguments (first is \"%s\")",
+ argv[optind]);
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Required arguments
+ */
+ if (subscriber_dir == NULL)
+ {
+ pg_log_error("no subscriber data directory specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Parse connection string. Build a base connection string that might be
+ * reused by multiple databases.
+ */
+ if (pub_conninfo_str == NULL)
+ {
+ /*
+ * TODO use primary_conninfo (if available) from subscriber and
+ * extract publisher connection string. Assume that there are
+ * identical entries for physical and logical replication. If there is
+ * not, we would fail anyway.
+ */
+ pg_log_error("no publisher connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
+ "publisher");
+ if (pub_base_conninfo == NULL)
+ exit(1);
+
+ if (sub_conninfo_str == NULL)
+ {
+ pg_log_error("no subscriber connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
+ if (sub_base_conninfo == NULL)
+ exit(1);
+
+ if (database_names.head == NULL)
+ {
+ pg_log_info("no database was specified");
+
+ /*
+ * If --database option is not provided, try to obtain the dbname from
+ * the publisher conninfo. If dbname parameter is not available, error
+ * out.
+ */
+ if (dbname_conninfo)
+ {
+ simple_string_list_append(&database_names, dbname_conninfo);
+ num_dbs++;
+
+ pg_log_info("database \"%s\" was extracted from the publisher connection string",
+ dbname_conninfo);
+ }
+ else
+ {
+ pg_log_error("no database name specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
+ */
+ if (!get_exec_path(argv[0]))
+ exit(1);
+
+ /* rudimentary check for a data directory. */
+ if (!check_data_directory(subscriber_dir))
+ exit(1);
+
+ /* Store database information for publisher and subscriber. */
+ dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+
+ /*
+ * Check if the subscriber data directory has the same system identifier
+ * than the publisher data directory.
+ */
+ pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
+ sub_sysid = get_control_from_datadir(subscriber_dir);
+ if (pub_sysid != sub_sysid)
+ {
+ pg_log_error("subscriber data directory is not a copy of the source database cluster");
+ exit(1);
+ }
+
+ /*
+ * Create the output directory to store any data generated by this tool.
+ */
+ base_dir = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("directory path for subscriber is too long");
+ exit(1);
+ }
+
+ if (mkdir(base_dir, pg_dir_create_mode) < 0 && errno != EEXIST)
+ {
+ pg_log_error("could not create directory \"%s\": %m", base_dir);
+ exit(1);
+ }
+
+ /* subscriber PID file. */
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+
+ /*
+ * The standby server must be running. That's because some checks will be
+ * done (is it ready for a logical replication setup?). After that, stop
+ * the subscriber in preparation to modify some recovery parameters that
+ * require a restart.
+ */
+ if (stat(pidfile, &statbuf) == 0)
+ {
+ /*
+ * Check if the standby server is ready for logical replication.
+ */
+ if (!setup_subscriber(dbinfo))
+ exit(1);
+
+ /*
+ * Check if the primary server is ready for logical replication and
+ * create the required objects for each database on publisher. This
+ * step is here mainly because if we stop the standby we cannot verify
+ * if the primary slot is in use. We could use an extra connection for
+ * it but it doesn't seem worth.
+ */
+ if (!setup_publisher(dbinfo))
+ exit(1);
+
+ pg_log_info("standby is up and running");
+ pg_log_info("stopping the server to start the transformation steps");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+ }
+ else
+ {
+ pg_log_error("standby is not running");
+ pg_log_error_hint("Start the standby and try again.");
+ exit(1);
+ }
+
+ /*
+ * Create a logical replication slot to get a consistent LSN.
+ *
+ * This consistent LSN will be used later to advanced the recently created
+ * replication slots. We cannot use the last created replication slot
+ * because the consistent LSN should be obtained *after* the base backup
+ * finishes (and the base backup should include the logical replication
+ * slots).
+ *
+ * XXX we should probably use the last created replication slot to get a
+ * consistent LSN but it should be changed after adding pg_basebackup
+ * support.
+ *
+ * A temporary replication slot is not used here to avoid keeping a
+ * replication connection open (depending when base backup was taken, the
+ * connection should be open for a few hours).
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+ consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
+ temp_replslot);
+
+ /*
+ * Write recovery parameters.
+ *
+ * Despite of the recovery parameters will be written to the subscriber,
+ * use a publisher connection for the follwing recovery functions. The
+ * connection is only used to check the current server version (physical
+ * replica, same server version). The subscriber is not running yet. In
+ * dry run mode, the recovery parameters *won't* be written. An invalid
+ * LSN is used for printing purposes.
+ */
+ recoveryconfcontents = GenerateRecoveryConfig(conn, NULL);
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_inclusive = true\n");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_action = promote\n");
+
+ if (dry_run)
+ {
+ appendPQExpBuffer(recoveryconfcontents, "# dry run mode");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%X/%X'\n",
+ LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
+ consistent_lsn);
+ WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ }
+ disconnect_database(conn);
+
+ pg_log_debug("recovery parameters:\n%s", recoveryconfcontents->data);
+
+ /*
+ * Start subscriber and wait until accepting connections.
+ */
+ pg_log_info("starting the subscriber");
+
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ server_start_log = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(server_start_log, MAXPGPATH, "%s/%s/server_start_%s.log", subscriber_dir, PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+
+ pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, subscriber_dir, server_start_log);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+
+ /*
+ * Waiting the subscriber to be promoted.
+ */
+ wait_for_end_recovery(dbinfo[0].subconninfo);
+
+ /*
+ * Create a subscription for each database.
+ */
+ for (i = 0; i < num_dbs; i++)
+ {
+ /* Connect to subscriber. */
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ create_subscription(conn, &dbinfo[i]);
+
+ /* Set the replication progress to the correct LSN. */
+ set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+
+ /* Enable subscription. */
+ enable_subscription(conn, &dbinfo[i]);
+
+ disconnect_database(conn);
+ }
+
+ /*
+ * The transient replication slot is no longer required. Drop it.
+ *
+ * If the physical replication slot exists, drop it.
+ *
+ * XXX we might not fail here. Instead, we provide a warning so the user
+ * eventually drops the replication slot later.
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ {
+ if (primary_slot_name != NULL)
+ pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
+ pg_log_warning("could not drop transient replication slot \"%s\" on publisher", temp_replslot);
+ pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
+ }
+ else
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ if (primary_slot_name != NULL)
+ drop_replication_slot(conn, &dbinfo[0], primary_slot_name);
+ disconnect_database(conn);
+ }
+
+ /*
+ * Stop the subscriber.
+ */
+ pg_log_info("stopping the subscriber");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+ /*
+ * Change system identifier.
+ */
+ modify_sysid(pg_resetwal_path, subscriber_dir);
+
+ /*
+ * The log file is kept if retain option is specified or this tool does
+ * not run successfully. Otherwise, log file is removed.
+ */
+ if (!retain)
+ unlink(server_start_log);
+
+ success = true;
+
+ pg_log_info("Done!");
+
+ return 0;
+}
diff --git a/src/bin/pg_basebackup/t/040_pg_subscriber.pl b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
new file mode 100644
index 0000000000..4ebff76b2d
--- /dev/null
+++ b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
@@ -0,0 +1,44 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test checking options of pg_subscriber.
+#
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+program_help_ok('pg_subscriber');
+program_version_ok('pg_subscriber');
+program_options_handling_ok('pg_subscriber');
+
+my $datadir = PostgreSQL::Test::Utils::tempdir;
+
+command_fails(['pg_subscriber'],
+ 'no subscriber data directory specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--pgdata', $datadir
+ ],
+ 'no publisher connection string specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--dry-run',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres'
+ ],
+ 'no subscriber connection string specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--verbose',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres',
+ '--subscriber-conninfo', 'dbname=postgres'
+ ],
+ 'no database name specified');
+
+done_testing();
diff --git a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
new file mode 100644
index 0000000000..fbcd0fc82b
--- /dev/null
+++ b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
@@ -0,0 +1,139 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test using a standby server as the subscriber.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node_p;
+my $node_f;
+my $node_s;
+my $result;
+
+# Set up node P as primary
+$node_p = PostgreSQL::Test::Cluster->new('node_p');
+$node_p->init(allows_streaming => 'logical');
+$node_p->start;
+
+# Set up node F as about-to-fail node
+# The extra option forces it to initialize a new cluster instead of copying a
+# previously initdb's cluster.
+$node_f = PostgreSQL::Test::Cluster->new('node_f');
+$node_f->init(allows_streaming => 'logical', extra => [ '--no-instructions' ]);
+$node_f->start;
+
+# On node P
+# - create databases
+# - create test tables
+# - insert a row
+$node_p->safe_psql(
+ 'postgres', q(
+ CREATE DATABASE pg1;
+ CREATE DATABASE pg2;
+));
+$node_p->safe_psql('pg1', 'CREATE TABLE tbl1 (a text)');
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('first row')");
+$node_p->safe_psql('pg2', 'CREATE TABLE tbl2 (a text)');
+
+# Set up node S as standby linking to node P
+$node_p->backup('backup_1');
+$node_s = PostgreSQL::Test::Cluster->new('node_s');
+$node_s->init_from_backup($node_p, 'backup_1', has_streaming => 1);
+$node_s->append_conf('postgresql.conf', 'log_min_messages = debug2');
+$node_s->set_standby_mode();
+$node_s->start;
+
+# Insert another row on node P and wait node S to catch up
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('second row')");
+$node_p->wait_for_replay_catchup($node_s);
+
+# Run pg_subscriber on about-to-fail node F
+command_fails(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $node_f->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_f->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'subscriber data directory is not a copy of the source database cluster');
+
+# dry run mode on node S
+command_ok(
+ [
+ 'pg_subscriber', '--verbose', '--dry-run',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_subscriber --dry-run on node S');
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+# Check if node S is still a standby
+is($node_s->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
+ 't', 'standby is in recovery');
+
+# Run pg_subscriber on node S
+command_ok(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_subscriber on node S');
+
+# Insert rows on P
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('third row')");
+$node_p->safe_psql('pg2', "INSERT INTO tbl2 VALUES('row 1')");
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+
+# Get subscription names
+$result = $node_s->safe_psql(
+ 'postgres', qq(
+ SELECT subname FROM pg_subscription WHERE subname ~ '^pg_subscriber_'
+));
+my @subnames = split("\n", $result);
+
+# Wait subscriber to catch up
+$node_s->wait_for_subscription_sync($node_p, $subnames[0]);
+$node_s->wait_for_subscription_sync($node_p, $subnames[1]);
+
+# Check result on database pg1
+$result = $node_s->safe_psql('pg1', 'SELECT * FROM tbl1');
+is( $result, qq(first row
+second row
+third row),
+ 'logical replication works on database pg1');
+
+# Check result on database pg2
+$result = $node_s->safe_psql('pg2', 'SELECT * FROM tbl2');
+is( $result, qq(row 1),
+ 'logical replication works on database pg2');
+
+# Different system identifier?
+my $sysid_p = $node_p->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+my $sysid_s = $node_s->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+ok($sysid_p != $sysid_s, 'system identifier was changed');
+
+# clean up
+$node_p->teardown_node;
+$node_s->teardown_node;
+
+done_testing();
--
2.30.2
^ permalink raw reply [nested|flat] 77+ messages in thread
* RE: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
@ 2024-01-26 07:55 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-26 07:55 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; 'Euler Taveira' <[email protected]>; Shlok Kyal <[email protected]>
Dear Euler,
Again, thanks for updating the patch! There are my random comments for v9.
01.
I cannot find your replies for my comments#7 [1] but you reverted related changes.
I'm not sure you are still considering it or you decided not to include changes.
Can you clarify your opinion?
(It is needed because changes are huge so it quite affects other developments...)
02.
```
+ <term><option>-t <replaceable class="parameter">seconds</replaceable></option></term>
+ <term><option>--timeout=<replaceable class="parameter">seconds</replaceable></option></term>
```
But source codes required `--recovery-timeout`. Please update either of them,
03.
```
+ * Create a new logical replica from a standby server
```
Junwang pointed out to change here but the change was reverted [2]
Can you clarify your opinion as well?
04.
```
+/*
+ * Is the source server ready for logical replication? If so, create the
+ * publications and replication slots in preparation for logical replication.
+ */
+static bool
+setup_publisher(LogicalRepInfo *dbinfo)
```
But this function verifies the source server. I felt they should be in the
different function.
05.
```
+/*
+ * Is the target server ready for logical replication?
+ */
+static bool
+setup_subscriber(LogicalRepInfo *dbinfo)
````
Actually, this function does not set up subscriber. It just verifies whether the
target can become a subscriber, right? If should be renamed.
06.
```
+ atexit(cleanup_objects_atexit);
```
The registration of the cleanup function is too early. This sometimes triggers
a core-dump. E.g.,
```
$ pg_subscriber --publisher-conninfo --subscriber-conninfo 'user=postgres port=5432' --verbose --database 'postgres' --pgdata data_N2/
pg_subscriber: error: too many command-line arguments (first is "user=postgres port=5432")
pg_subscriber: hint: Try "pg_subscriber --help" for more information.
Segmentation fault (core dumped)
$ gdb ...
(gdb) bt
#0 cleanup_objects_atexit () at pg_subscriber.c:131
#1 0x00007fb982cffce9 in __run_exit_handlers () from /lib64/libc.so.6
#2 0x00007fb982cffd37 in exit () from /lib64/libc.so.6
#3 0x00000000004054e6 in main (argc=9, argv=0x7ffc59074158) at pg_subscriber.c:1500
(gdb) f 3
#3 0x00000000004054e6 in main (argc=9, argv=0x7ffc59074158) at pg_subscriber.c:1500
1500 exit(1);
(gdb) list
1495 if (optind < argc)
1496 {
1497 pg_log_error("too many command-line arguments (first is \"%s\")",
1498 argv[optind]);
1499 pg_log_error_hint("Try \"%s --help\" for more information.", progname);
1500 exit(1);
1501 }
1502
1503 /*
1504 * Required arguments
```
I still think it should be done just before the creation of objects [3].
07.
Missing a removal of publications on the standby.
08.
Missing registration of LogicalRepInfo in the typedefs.list.
09
```
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_subscriber</command>
+ <arg rep="repeat"><replaceable>option</replaceable></arg>
+ </cmdsynopsis>
+ </refsynopsisdiv>
```
Can you reply my comment#2 [4]? I think mandatory options should be written.
10.
Just to confirm - will you implement start_standby/stop_standby functions in next version?
[1]: https://www.postgresql.org/message-id/TY3PR01MB9889C362FF76102C88FA1C29F56F2%40TY3PR01MB9889.jpnprd0...
[2]: https://www.postgresql.org/message-id/CAEG8a3%2BwL_2R8n12BmRz7yBP3EBNdHDhmdgxQFA9vS%2BzPR%2B3Kw%40ma...
[3]: https://www.postgresql.org/message-id/TY3PR01MB9889678E47B918F4D83A6FD8F57B2%40TY3PR01MB9889.jpnprd0...
[4]: https://www.postgresql.org/message-id/TY3PR01MB9889C362FF76102C88FA1C29F56F2%40TY3PR01MB9889.jpnprd0...
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/global/
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
@ 2024-01-29 01:10 ` Euler Taveira <[email protected]>
2024-01-29 09:57 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-30 02:01 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
0 siblings, 2 replies; 77+ messages in thread
From: Euler Taveira @ 2024-01-29 01:10 UTC (permalink / raw)
To: [email protected] <[email protected]>; [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Shlok Kyal <[email protected]>
On Fri, Jan 26, 2024, at 4:55 AM, Hayato Kuroda (Fujitsu) wrote:
> Again, thanks for updating the patch! There are my random comments for v9.
Thanks for checking v9. I already incorporated some of the points below into
the next patch. Give me a couple of hours to include some important points.
> 01.
> I cannot find your replies for my comments#7 [1] but you reverted related changes.
> I'm not sure you are still considering it or you decided not to include changes.
> Can you clarify your opinion?
> (It is needed because changes are huge so it quite affects other developments...)
It is still on my list. As I said in a previous email I'm having a hard time
reviewing pieces from your 0002 patch because you include a bunch of things
into one patch.
> 02.
> ```
> + <term><option>-t <replaceable class="parameter">seconds</replaceable></option></term>
> + <term><option>--timeout=<replaceable class="parameter">seconds</replaceable></option></term>
> ```
>
> But source codes required `--recovery-timeout`. Please update either of them,
Oops. Fixed. My preference is --recovery-timeout because someone can decide to
include a --timeout option for this tool.
> 03.
> ```
> + * Create a new logical replica from a standby server
> ```
>
> Junwang pointed out to change here but the change was reverted [2]
> Can you clarify your opinion as well?
I'll review the documentation once I fix the code. Since the tool includes the
*create* verb into its name, it seems strange use another verb (convert) in the
description. Maybe we should just remove the word *new* and a long description
explains the action is to turn the standby (physical replica) into a logical
replica.
> 04.
> ```
> +/*
> + * Is the source server ready for logical replication? If so, create the
> + * publications and replication slots in preparation for logical replication.
> + */
> +static bool
> +setup_publisher(LogicalRepInfo *dbinfo)
> ```
>
> But this function verifies the source server. I felt they should be in the
> different function.
I split setup_publisher() into check_publisher() and setup_publisher().
> 05.
> ```
> +/*
> + * Is the target server ready for logical replication?
> + */
> +static bool
> +setup_subscriber(LogicalRepInfo *dbinfo)
> ````
>
> Actually, this function does not set up subscriber. It just verifies whether the
> target can become a subscriber, right? If should be renamed.
I renamed setup_subscriber() -> check_subscriber().
> 06.
> ```
> + atexit(cleanup_objects_atexit);
> ```
>
> The registration of the cleanup function is too early. This sometimes triggers
> a core-dump. E.g.,
I forgot to apply the atexit() patch.
> 07.
> Missing a removal of publications on the standby.
It was on my list to do. It will be in the next patch.
> 08.
> Missing registration of LogicalRepInfo in the typedefs.list.
Done.
> 09
> ```
> + <refsynopsisdiv>
> + <cmdsynopsis>
> + <command>pg_subscriber</command>
> + <arg rep="repeat"><replaceable>option</replaceable></arg>
> + </cmdsynopsis>
> + </refsynopsisdiv>
> ```
>
> Can you reply my comment#2 [4]? I think mandatory options should be written.
I included the mandatory options into the synopsis.
> 10.
> Just to confirm - will you implement start_standby/stop_standby functions in next version?
It is still on my list.
--
Euler Taveira
EDB https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 77+ messages in thread
* RE: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
@ 2024-01-29 09:57 ` Hayato Kuroda (Fujitsu) <[email protected]>
1 sibling, 0 replies; 77+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-29 09:57 UTC (permalink / raw)
To: 'Euler Taveira' <[email protected]>; [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Shlok Kyal <[email protected]>
Dear Euler,
>
It is still on my list. As I said in a previous email I'm having a hard time
reviewing pieces from your 0002 patch because you include a bunch of things
...
It is still on my list.
>
I understood that patches we posted were bad. Sorry for inconvenience.
So I extracted changes only related with them. Can you review them and include
If it seems OK?
v10-0001: same as v9-0001.
0002: not related with our changes, but this fixed a small bug.
The third argument of getopt_long was not correct.
0003: adds start_standby/stop_standby funcitons
0004: tries to refactor some structures.
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/global/
Attachments:
[application/octet-stream] v10-0001-Creates-a-new-logical-replica-from-a-standby-ser.patch (73.2K, ../../TY3PR01MB98891E7735141FE8760CEC4AF57E2@TY3PR01MB9889.jpnprd01.prod.outlook.com/2-v10-0001-Creates-a-new-logical-replica-from-a-standby-ser.patch)
download | inline diff:
From 8eff27951f4b778639e91e0a02f871628af60afd Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email protected]>
Date: Mon, 5 Jun 2023 14:39:40 -0400
Subject: [PATCH v10 1/4] Creates a new logical replica from a standby server
A new tool called pg_subscriber can convert a physical replica into a
logical replica. It runs on the target server and should be able to
connect to the source server (publisher) and the target server
(subscriber).
The conversion requires a few steps. Check if the target data directory
has the same system identifier than the source data directory. Stop the
target server if it is running as a standby server. Create one
replication slot per specified database on the source server. One
additional replication slot is created at the end to get the consistent
LSN (This consistent LSN will be used as (a) a stopping point for the
recovery process and (b) a starting point for the subscriptions). Write
recovery parameters into the target data directory and start the target
server (Wait until the target server is promoted). Create one
publication (FOR ALL TABLES) per specified database on the source
server. Create one subscription per specified database on the target
server (Use replication slot and publication created in a previous step.
Don't enable the subscriptions yet). Sets the replication progress to
the consistent LSN that was got in a previous step. Enable the
subscription for each specified database on the target server. Remove
the additional replication slot that was used to get the consistent LSN.
Stop the target server. Change the system identifier from the target
server.
Depending on your workload and database size, creating a logical replica
couldn't be an option due to resource constraints (WAL backlog should be
available until all table data is synchronized). The initial data copy
and the replication progress tends to be faster on a physical replica.
The purpose of this tool is to speed up a logical replica setup.
---
doc/src/sgml/ref/allfiles.sgml | 1 +
doc/src/sgml/ref/pg_subscriber.sgml | 305 +++
doc/src/sgml/reference.sgml | 1 +
src/bin/pg_basebackup/.gitignore | 1 +
src/bin/pg_basebackup/Makefile | 8 +-
src/bin/pg_basebackup/meson.build | 19 +
src/bin/pg_basebackup/pg_subscriber.c | 1805 +++++++++++++++++
src/bin/pg_basebackup/t/040_pg_subscriber.pl | 44 +
.../t/041_pg_subscriber_standby.pl | 139 ++
9 files changed, 2322 insertions(+), 1 deletion(-)
create mode 100644 doc/src/sgml/ref/pg_subscriber.sgml
create mode 100644 src/bin/pg_basebackup/pg_subscriber.c
create mode 100644 src/bin/pg_basebackup/t/040_pg_subscriber.pl
create mode 100644 src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index 4a42999b18..3862c976d7 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -214,6 +214,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY pgResetwal SYSTEM "pg_resetwal.sgml">
<!ENTITY pgRestore SYSTEM "pg_restore.sgml">
<!ENTITY pgRewind SYSTEM "pg_rewind.sgml">
+<!ENTITY pgSubscriber SYSTEM "pg_subscriber.sgml">
<!ENTITY pgVerifyBackup SYSTEM "pg_verifybackup.sgml">
<!ENTITY pgtestfsync SYSTEM "pgtestfsync.sgml">
<!ENTITY pgtesttiming SYSTEM "pgtesttiming.sgml">
diff --git a/doc/src/sgml/ref/pg_subscriber.sgml b/doc/src/sgml/ref/pg_subscriber.sgml
new file mode 100644
index 0000000000..99d4fcee49
--- /dev/null
+++ b/doc/src/sgml/ref/pg_subscriber.sgml
@@ -0,0 +1,305 @@
+<!--
+doc/src/sgml/ref/pg_subscriber.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="app-pgsubscriber">
+ <indexterm zone="app-pgsubscriber">
+ <primary>pg_subscriber</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle><application>pg_subscriber</application></refentrytitle>
+ <manvolnum>1</manvolnum>
+ <refmiscinfo>Application</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>pg_subscriber</refname>
+ <refpurpose>convert a physical replica into a new logical replica</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_subscriber</command>
+ <arg rep="repeat"><replaceable>option</replaceable></arg>
+ </cmdsynopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+ <para>
+ <application>pg_subscriber</application> takes the publisher and subscriber
+ connection strings, a cluster directory from a physical replica and a list of
+ database names and it sets up a new logical replica using the physical
+ recovery process.
+ </para>
+
+ <para>
+ The <application>pg_subscriber</application> should be run at the target
+ server. The source server (known as publisher server) should accept logical
+ replication connections from the target server (known as subscriber server).
+ The target server should accept local logical replication connection.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Options</title>
+
+ <para>
+ <application>pg_subscriber</application> accepts the following
+ command-line arguments:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-D <replaceable class="parameter">directory</replaceable></option></term>
+ <term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term>
+ <listitem>
+ <para>
+ The target directory that contains a cluster directory from a physical
+ replica.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-P <replaceable class="parameter">conninfo</replaceable></option></term>
+ <term><option>--publisher-conninfo=<replaceable class="parameter">conninfo</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the publisher. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-S <replaceable class="parameter">conninfo</replaceable></option></term>
+ <term><option>--subscriber-conninfo=<replaceable class="parameter">conninfo</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the subscriber. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-d <replaceable class="parameter">dbname</replaceable></option></term>
+ <term><option>--database=<replaceable class="parameter">dbname</replaceable></option></term>
+ <listitem>
+ <para>
+ The database name to create the subscription. Multiple databases can be
+ selected by writing multiple <option>-d</option> switches.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
+ <listitem>
+ <para>
+ Do everything except actually modifying the target directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-r</option></term>
+ <term><option>--retain</option></term>
+ <listitem>
+ <para>
+ Retain log file even after successful completion.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-t <replaceable class="parameter">seconds</replaceable></option></term>
+ <term><option>--timeout=<replaceable class="parameter">seconds</replaceable></option></term>
+ <listitem>
+ <para>
+ The maximum number of seconds to wait for recovery to end. Setting to 0
+ disables. The default is 0.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-v</option></term>
+ <term><option>--verbose</option></term>
+ <listitem>
+ <para>
+ Enables verbose mode. This will cause
+ <application>pg_subscriber</application> to output progress messages
+ and detailed information about each step.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ Other options are also available:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-V</option></term>
+ <term><option>--version</option></term>
+ <listitem>
+ <para>
+ Print the <application>pg_subscriber</application> version and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-?</option></term>
+ <term><option>--help</option></term>
+ <listitem>
+ <para>
+ Show help about <application>pg_subscriber</application> command
+ line arguments, and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Notes</title>
+
+ <para>
+ The transformation proceeds in the following steps:
+ </para>
+
+ <procedure>
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the given target data
+ directory has the same system identifier than the source data directory.
+ Since it uses the recovery process as one of the steps, it starts the
+ target server as a replica from the source server. If the system
+ identifier is not the same, <application>pg_subscriber</application> will
+ terminate with an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the target data
+ directory is used by a physical replica. Stop the physical replica if it is
+ running. One of the next steps is to add some recovery parameters that
+ requires a server start. This step avoids an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one replication slot for
+ each specified database on the source server. The replication slot name
+ contains a <literal>pg_subscriber</literal> prefix. These replication
+ slots will be used by the subscriptions in a future step. Another
+ replication slot is used to get a consistent start location. This
+ consistent LSN will be used as a stopping point in the <xref
+ linkend="guc-recovery-target-lsn"/> parameter and by the
+ subscriptions as a replication starting point. It guarantees that no
+ transaction will be lost.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> writes recovery parameters into
+ the target data directory and start the target server. It specifies a LSN
+ (consistent LSN that was obtained in the previous step) of write-ahead
+ log location up to which recovery will proceed. It also specifies
+ <literal>promote</literal> as the action that the server should take once
+ the recovery target is reached. This step finishes once the server ends
+ standby mode and is accepting read-write operations.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Next, <application>pg_subscriber</application> creates one publication
+ for each specified database on the source server. Each publication
+ replicates changes for all tables in the database. The publication name
+ contains a <literal>pg_subscriber</literal> prefix. These publication
+ will be used by a corresponding subscription in a next step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one subscription for
+ each specified database on the target server. Each subscription name
+ contains a <literal>pg_subscriber</literal> prefix. The replication slot
+ name is identical to the subscription name. It does not copy existing data
+ from the source server. It does not create a replication slot. Instead, it
+ uses the replication slot that was created in a previous step. The
+ subscription is created but it is not enabled yet. The reason is the
+ replication progress must be set to the consistent LSN but replication
+ origin name contains the subscription oid in its name. Hence, the
+ subscription will be enabled in a separate step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> sets the replication progress to
+ the consistent LSN that was obtained in a previous step. When the target
+ server started the recovery process, it caught up to the consistent LSN.
+ This is the exact LSN to be used as a initial location for each
+ subscription.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Finally, <application>pg_subscriber</application> enables the subscription
+ for each specified database on the target server. The subscription starts
+ streaming from the consistent LSN.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> removes the additional replication
+ slot that was used to get the consistent LSN on the source server.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> stops the target server to change
+ its system identifier.
+ </para>
+ </step>
+ </procedure>
+ </refsect1>
+
+ <refsect1>
+ <title>Examples</title>
+
+ <para>
+ To create a logical replica for databases <literal>hr</literal> and
+ <literal>finance</literal> from a physical replica at <literal>foo</literal>:
+<screen>
+<prompt>$</prompt> <userinput>pg_subscriber -D /usr/local/pgsql/data -P "host=foo" -S "host=localhost" -d hr -d finance</userinput>
+</screen>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="app-pgbasebackup"/></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index aa94f6adf6..266f4e515a 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -285,6 +285,7 @@
&pgCtl;
&pgResetwal;
&pgRewind;
+ &pgSubscriber;
&pgtestfsync;
&pgtesttiming;
&pgupgrade;
diff --git a/src/bin/pg_basebackup/.gitignore b/src/bin/pg_basebackup/.gitignore
index 26048bdbd8..0e5384a1d5 100644
--- a/src/bin/pg_basebackup/.gitignore
+++ b/src/bin/pg_basebackup/.gitignore
@@ -1,5 +1,6 @@
/pg_basebackup
/pg_receivewal
/pg_recvlogical
+/pg_subscriber
/tmp_check/
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index abfb6440ec..f6281b7676 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -44,7 +44,7 @@ BBOBJS = \
bbstreamer_tar.o \
bbstreamer_zstd.o
-all: pg_basebackup pg_receivewal pg_recvlogical
+all: pg_basebackup pg_receivewal pg_recvlogical pg_subscriber
pg_basebackup: $(BBOBJS) $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) $(BBOBJS) $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
@@ -55,10 +55,14 @@ pg_receivewal: pg_receivewal.o $(OBJS) | submake-libpq submake-libpgport submake
pg_recvlogical: pg_recvlogical.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_recvlogical.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_subscriber: $(WIN32RES) pg_subscriber.o | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
install: all installdirs
$(INSTALL_PROGRAM) pg_basebackup$(X) '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
$(INSTALL_PROGRAM) pg_receivewal$(X) '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
$(INSTALL_PROGRAM) pg_recvlogical$(X) '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ $(INSTALL_PROGRAM) pg_subscriber$(X) '$(DESTDIR)$(bindir)/pg_subscriber$(X)'
installdirs:
$(MKDIR_P) '$(DESTDIR)$(bindir)'
@@ -67,10 +71,12 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ rm -f '$(DESTDIR)$(bindir)/pg_subscriber$(X)'
clean distclean:
rm -f pg_basebackup$(X) pg_receivewal$(X) pg_recvlogical$(X) \
$(BBOBJS) pg_receivewal.o pg_recvlogical.o \
+ pg_subscriber$(X) pg_subscriber.o \
$(OBJS)
rm -rf tmp_check
diff --git a/src/bin/pg_basebackup/meson.build b/src/bin/pg_basebackup/meson.build
index f7e60e6670..ccfd7bb7a5 100644
--- a/src/bin/pg_basebackup/meson.build
+++ b/src/bin/pg_basebackup/meson.build
@@ -75,6 +75,23 @@ pg_recvlogical = executable('pg_recvlogical',
)
bin_targets += pg_recvlogical
+pg_subscriber_sources = files(
+ 'pg_subscriber.c'
+)
+
+if host_system == 'windows'
+ pg_subscriber_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'pg_subscriber',
+ '--FILEDESC', 'pg_subscriber - create a new logical replica from a standby server',])
+endif
+
+pg_subscriber = executable('pg_subscriber',
+ pg_subscriber_sources,
+ dependencies: [frontend_code, libpq],
+ kwargs: default_bin_args,
+)
+bin_targets += pg_subscriber
+
tests += {
'name': 'pg_basebackup',
'sd': meson.current_source_dir(),
@@ -89,6 +106,8 @@ tests += {
't/011_in_place_tablespace.pl',
't/020_pg_receivewal.pl',
't/030_pg_recvlogical.pl',
+ 't/040_pg_subscriber.pl',
+ 't/041_pg_subscriber_standby.pl',
],
},
}
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
new file mode 100644
index 0000000000..cb97dbda5e
--- /dev/null
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -0,0 +1,1805 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_subscriber.c
+ * Create a new logical replica from a standby server
+ *
+ * Copyright (C) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_basebackup/pg_subscriber.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include <signal.h>
+#include <sys/stat.h>
+#include <sys/time.h>
+#include <sys/wait.h>
+#include <time.h>
+
+#include "access/xlogdefs.h"
+#include "catalog/pg_control.h"
+#include "common/connect.h"
+#include "common/controldata_utils.h"
+#include "common/file_perm.h"
+#include "common/file_utils.h"
+#include "common/logging.h"
+#include "fe_utils/recovery_gen.h"
+#include "fe_utils/simple_list.h"
+#include "getopt_long.h"
+#include "utils/pidfile.h"
+
+#define PGS_OUTPUT_DIR "pg_subscriber_output.d"
+
+typedef struct LogicalRepInfo
+{
+ Oid oid; /* database OID */
+ char *dbname; /* database name */
+ char *pubconninfo; /* publication connection string for logical
+ * replication */
+ char *subconninfo; /* subscription connection string for logical
+ * replication */
+ char *pubname; /* publication name */
+ char *subname; /* subscription name (also replication slot
+ * name) */
+
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepInfo;
+
+static void cleanup_objects_atexit(void);
+static void usage();
+static char *get_base_conninfo(char *conninfo, char *dbname,
+ const char *noderole);
+static bool get_exec_path(const char *path);
+static bool check_data_directory(const char *datadir);
+static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
+static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
+static PGconn *connect_database(const char *conninfo);
+static void disconnect_database(PGconn *conn);
+static uint64 get_sysid_from_conn(const char *conninfo);
+static uint64 get_control_from_datadir(const char *datadir);
+static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
+static bool setup_publisher(LogicalRepInfo *dbinfo);
+static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name);
+static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
+static void wait_for_end_recovery(const char *conninfo);
+static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+
+#define USEC_PER_SEC 1000000
+#define WAIT_INTERVAL 1 /* 1 second */
+
+/* Options */
+static const char *progname;
+
+static char *subscriber_dir = NULL;
+static char *pub_conninfo_str = NULL;
+static char *sub_conninfo_str = NULL;
+static SimpleStringList database_names = {NULL, NULL};
+static char *primary_slot_name = NULL;
+static bool dry_run = false;
+static bool retain = false;
+static int recovery_timeout = 0;
+
+static bool success = false;
+
+static char *pg_ctl_path = NULL;
+static char *pg_resetwal_path = NULL;
+
+static LogicalRepInfo *dbinfo;
+static int num_dbs = 0;
+
+static char temp_replslot[NAMEDATALEN] = {0};
+static bool made_transient_replslot = false;
+
+enum WaitPMResult
+{
+ POSTMASTER_READY,
+ POSTMASTER_STANDBY,
+ POSTMASTER_STILL_STARTING,
+ POSTMASTER_FAILED
+};
+
+
+/*
+ * Cleanup objects that were created by pg_subscriber if there is an error.
+ *
+ * Replication slots, publications and subscriptions are created. Depending on
+ * the step it failed, it should remove the already created objects if it is
+ * possible (sometimes it won't work due to a connection issue).
+ */
+static void
+cleanup_objects_atexit(void)
+{
+ PGconn *conn;
+ int i;
+
+ if (success)
+ return;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ if (dbinfo[i].made_subscription)
+ {
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn != NULL)
+ {
+ drop_subscription(conn, &dbinfo[i]);
+ disconnect_database(conn);
+ }
+ }
+
+ if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ {
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn != NULL)
+ {
+ if (dbinfo[i].made_publication)
+ drop_publication(conn, &dbinfo[i]);
+ if (dbinfo[i].made_replslot)
+ drop_replication_slot(conn, &dbinfo[i], NULL);
+ disconnect_database(conn);
+ }
+ }
+ }
+
+ if (made_transient_replslot)
+ {
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn != NULL)
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ disconnect_database(conn);
+ }
+ }
+}
+
+static void
+usage(void)
+{
+ printf(_("%s creates a new logical replica from a standby server.\n\n"),
+ progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION]...\n"), progname);
+ printf(_("\nOptions:\n"));
+ printf(_(" -D, --pgdata=DATADIR location for the subscriber data directory\n"));
+ printf(_(" -P, --publisher-conninfo=CONNINFO publisher connection string\n"));
+ printf(_(" -S, --subscriber-conninfo=CONNINFO subscriber connection string\n"));
+ printf(_(" -d, --database=DBNAME database to create a subscription\n"));
+ printf(_(" -n, --dry-run stop before modifying anything\n"));
+ printf(_(" -t, --recovery-timeout=SECS seconds to wait for recovery to end\n"));
+ printf(_(" -r, --retain retain log file after success\n"));
+ printf(_(" -v, --verbose output verbose messages\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
+ printf(_("\nReport bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+ printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
+
+/*
+ * Validate a connection string. Returns a base connection string that is a
+ * connection string without a database name.
+ * Since we might process multiple databases, each database name will be
+ * appended to this base connection string to provide a final connection string.
+ * If the second argument (dbname) is not null, returns dbname if the provided
+ * connection string contains it. If option --database is not provided, uses
+ * dbname as the only database to setup the logical replica.
+ * It is the caller's responsibility to free the returned connection string and
+ * dbname.
+ */
+static char *
+get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ PQconninfoOption *conn_opts = NULL;
+ PQconninfoOption *conn_opt;
+ char *errmsg = NULL;
+ char *ret;
+ int i;
+
+ pg_log_info("validating connection string on %s", noderole);
+
+ conn_opts = PQconninfoParse(conninfo, &errmsg);
+ if (conn_opts == NULL)
+ {
+ pg_log_error("could not parse connection string: %s", errmsg);
+ return NULL;
+ }
+
+ i = 0;
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (strcmp(conn_opt->keyword, "dbname") == 0 && conn_opt->val != NULL)
+ {
+ if (dbname)
+ dbname = pg_strdup(conn_opt->val);
+ continue;
+ }
+
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0')
+ {
+ if (i > 0)
+ appendPQExpBufferChar(buf, ' ');
+ appendPQExpBuffer(buf, "%s=%s", conn_opt->keyword, conn_opt->val);
+ i++;
+ }
+ }
+
+ ret = pg_strdup(buf->data);
+
+ destroyPQExpBuffer(buf);
+ PQconninfoFree(conn_opts);
+
+ return ret;
+}
+
+/*
+ * Get the absolute path from other PostgreSQL binaries (pg_ctl and
+ * pg_resetwal) that is used by it.
+ */
+static bool
+get_exec_path(const char *path)
+{
+ int rc;
+
+ pg_ctl_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_ctl",
+ "pg_ctl (PostgreSQL) " PG_VERSION "\n",
+ pg_ctl_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_ctl", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_ctl", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
+
+ pg_resetwal_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_resetwal",
+ "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
+ pg_resetwal_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_resetwal", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_resetwal", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+
+ return true;
+}
+
+/*
+ * Is it a cluster directory? These are preliminary checks. It is far from
+ * making an accurate check. If it is not a clone from the publisher, it will
+ * eventually fail in a future step.
+ */
+static bool
+check_data_directory(const char *datadir)
+{
+ struct stat statbuf;
+ char versionfile[MAXPGPATH];
+
+ pg_log_info("checking if directory \"%s\" is a cluster data directory",
+ datadir);
+
+ if (stat(datadir, &statbuf) != 0)
+ {
+ if (errno == ENOENT)
+ pg_log_error("data directory \"%s\" does not exist", datadir);
+ else
+ pg_log_error("could not access directory \"%s\": %s", datadir, strerror(errno));
+
+ return false;
+ }
+
+ snprintf(versionfile, MAXPGPATH, "%s/PG_VERSION", datadir);
+ if (stat(versionfile, &statbuf) != 0 && errno == ENOENT)
+ {
+ pg_log_error("directory \"%s\" is not a database cluster directory", datadir);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Append database name into a base connection string.
+ *
+ * dbname is the only parameter that changes so it is not included in the base
+ * connection string. This function concatenates dbname to build a "real"
+ * connection string.
+ */
+static char *
+concat_conninfo_dbname(const char *conninfo, const char *dbname)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *ret;
+
+ Assert(conninfo != NULL);
+
+ appendPQExpBufferStr(buf, conninfo);
+ appendPQExpBuffer(buf, " dbname=%s", dbname);
+
+ ret = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+
+ return ret;
+}
+
+/*
+ * Store publication and subscription information.
+ */
+static LogicalRepInfo *
+store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+{
+ LogicalRepInfo *dbinfo;
+ SimpleStringListCell *cell;
+ int i = 0;
+
+ dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+
+ for (cell = database_names.head; cell; cell = cell->next)
+ {
+ char *conninfo;
+
+ /* Publisher. */
+ conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
+ dbinfo[i].pubconninfo = conninfo;
+ dbinfo[i].dbname = cell->val;
+ dbinfo[i].made_replslot = false;
+ dbinfo[i].made_publication = false;
+ dbinfo[i].made_subscription = false;
+ /* other struct fields will be filled later. */
+
+ /* Subscriber. */
+ conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
+ dbinfo[i].subconninfo = conninfo;
+
+ i++;
+ }
+
+ return dbinfo;
+}
+
+static PGconn *
+connect_database(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ const char *rconninfo;
+
+ /* logical replication mode */
+ rconninfo = psprintf("%s replication=database", conninfo);
+
+ conn = PQconnectdb(rconninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ pg_log_error("connection to database failed: %s", PQerrorMessage(conn));
+ return NULL;
+ }
+
+ /* secure search_path */
+ res = PQexec(conn, ALWAYS_SECURE_SEARCH_PATH_SQL);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not clear search_path: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+ PQclear(res);
+
+ return conn;
+}
+
+static void
+disconnect_database(PGconn *conn)
+{
+ Assert(conn != NULL);
+
+ PQfinish(conn);
+}
+
+/*
+ * Obtain the system identifier using the provided connection. It will be used
+ * to compare if a data directory is a clone of another one.
+ */
+static uint64
+get_sysid_from_conn(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from publisher");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "IDENTIFY_SYSTEM");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not send replication command \"%s\": %s",
+ "IDENTIFY_SYSTEM", PQresultErrorMessage(res));
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+ if (PQntuples(res) != 1 || PQnfields(res) < 3)
+ {
+ pg_log_error("could not identify system: got %d rows and %d fields, expected %d rows and %d or more fields",
+ PQntuples(res), PQnfields(res), 1, 3);
+
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+
+ sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+
+ pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+
+ disconnect_database(conn);
+
+ return sysid;
+}
+
+/*
+ * Obtain the system identifier from control file. It will be used to compare
+ * if a data directory is a clone of another one. This routine is used locally
+ * and avoids a replication connection.
+ */
+static uint64
+get_control_from_datadir(const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ sysid = cf->system_identifier;
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+
+ pfree(cf);
+
+ return sysid;
+}
+
+/*
+ * Modify the system identifier. Since a standby server preserves the system
+ * identifier, it makes sense to change it to avoid situations in which WAL
+ * files from one of the systems might be used in the other one.
+ */
+static void
+modify_sysid(const char *pg_resetwal_path, const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ struct timeval tv;
+
+ char *cmd_str;
+ int rc;
+
+ pg_log_info("modifying system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ /*
+ * Select a new system identifier.
+ *
+ * XXX this code was extracted from BootStrapXLOG().
+ */
+ gettimeofday(&tv, NULL);
+ cf->system_identifier = ((uint64) tv.tv_sec) << 32;
+ cf->system_identifier |= ((uint64) tv.tv_usec) << 12;
+ cf->system_identifier |= getpid() & 0xFFF;
+
+ if (!dry_run)
+ update_controlfile(datadir, cf, true);
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) cf->system_identifier);
+
+ pg_log_info("running pg_resetwal on the subscriber");
+
+ cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+
+ pg_log_debug("command is: %s", cmd_str);
+
+ if (!dry_run)
+ {
+ rc = system(cmd_str);
+ if (rc == 0)
+ pg_log_info("subscriber successfully changed the system identifier");
+ else
+ pg_log_error("subscriber failed to change system identifier: exit code: %d", rc);
+ }
+
+ pfree(cf);
+}
+
+/*
+ * Is the source server ready for logical replication? If so, create the
+ * publications and replication slots in preparation for logical replication.
+ */
+static bool
+setup_publisher(LogicalRepInfo *dbinfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ PQExpBuffer str = createPQExpBuffer();
+
+ char *wal_level;
+ int max_repslots;
+ int cur_repslots;
+ int max_walsenders;
+ int cur_walsenders;
+
+ pg_log_info("checking settings on publisher");
+
+ /*
+ * Logical replication requires a few parameters to be set on publisher.
+ * Since these parameters are not a requirement for physical replication,
+ * we should check it to make sure it won't fail.
+ *
+ * wal_level = logical
+ * max_replication_slots >= current + number of dbs to be converted
+ * max_wal_senders >= current + number of dbs to be converted
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "WITH wl AS (SELECT setting AS wallevel FROM pg_settings WHERE name = 'wal_level'),"
+ " total_mrs AS (SELECT setting AS tmrs FROM pg_settings WHERE name = 'max_replication_slots'),"
+ " cur_mrs AS (SELECT count(*) AS cmrs FROM pg_replication_slots),"
+ " total_mws AS (SELECT setting AS tmws FROM pg_settings WHERE name = 'max_wal_senders'),"
+ " cur_mws AS (SELECT count(*) AS cmws FROM pg_stat_activity WHERE backend_type = 'walsender')"
+ "SELECT wallevel, tmrs, cmrs, tmws, cmws FROM wl, total_mrs, cur_mrs, total_mws, cur_mws");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publisher settings: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ wal_level = strdup(PQgetvalue(res, 0, 0));
+ max_repslots = atoi(PQgetvalue(res, 0, 1));
+ cur_repslots = atoi(PQgetvalue(res, 0, 2));
+ max_walsenders = atoi(PQgetvalue(res, 0, 3));
+ cur_walsenders = atoi(PQgetvalue(res, 0, 4));
+
+ PQclear(res);
+
+ pg_log_debug("subscriber: wal_level: %s", wal_level);
+ pg_log_debug("subscriber: max_replication_slots: %d", max_repslots);
+ pg_log_debug("subscriber: current replication slots: %d", cur_repslots);
+ pg_log_debug("subscriber: max_wal_senders: %d", max_walsenders);
+ pg_log_debug("subscriber: current wal senders: %d", cur_walsenders);
+
+ /*
+ * If standby sets primary_slot_name, check if this replication slot is in
+ * use on primary for WAL retention purposes. This replication slot has no
+ * use after the transformation, hence, it will be removed at the end of
+ * this process.
+ */
+ if (primary_slot_name)
+ {
+ appendPQExpBuffer(str,
+ "SELECT 1 FROM pg_replication_slots WHERE active AND slot_name = '%s'", primary_slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain replication slot information: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
+ PQntuples(res), 1);
+ pg_free(primary_slot_name); /* it is not being used. */
+ primary_slot_name = NULL;
+ return false;
+ }
+ else
+ {
+ pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+ }
+
+ PQclear(res);
+ }
+
+ disconnect_database(conn);
+
+ if (strcmp(wal_level, "logical") != 0)
+ {
+ pg_log_error("publisher requires wal_level >= logical");
+ return false;
+ }
+
+ if (max_repslots - cur_repslots < num_dbs)
+ {
+ pg_log_error("publisher requires %d replication slots, but only %d remain", num_dbs, max_repslots - cur_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", cur_repslots + num_dbs);
+ return false;
+ }
+
+ if (max_walsenders - cur_walsenders < num_dbs)
+ {
+ pg_log_error("publisher requires %d wal sender processes, but only %d remain", num_dbs, max_walsenders - cur_walsenders);
+ pg_log_error_hint("Consider increasing max_wal_senders to at least %d.", cur_walsenders + num_dbs);
+ return false;
+ }
+
+ for (int i = 0; i < num_dbs; i++)
+ {
+ char pubname[NAMEDATALEN];
+ char replslotname[NAMEDATALEN];
+
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT oid FROM pg_catalog.pg_database WHERE datname = current_database()");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain database OID: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain database OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ return false;
+ }
+
+ /* Remember database OID. */
+ dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+
+ PQclear(res);
+
+ /*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 35 characters (14 + 10 +
+ * '\0').
+ */
+ snprintf(pubname, sizeof(pubname), "pg_subscriber_%u", dbinfo[i].oid);
+ dbinfo[i].pubname = pg_strdup(pubname);
+
+ /*
+ * Create publication on publisher. This step should be executed
+ * *before* promoting the subscriber to avoid any transactions between
+ * consistent LSN and the new publication rows (such transactions
+ * wouldn't see the new publication rows resulting in an error).
+ */
+ create_publication(conn, &dbinfo[i]);
+
+ /*
+ * Build the replication slot name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 36
+ * characters (14 + 10 + 1 + 10 + '\0'). System identifier is included
+ * to reduce the probability of collision. By default, subscription
+ * name is used as replication slot name.
+ */
+ snprintf(replslotname, sizeof(replslotname),
+ "pg_subscriber_%u_%d",
+ dbinfo[i].oid,
+ (int) getpid());
+ dbinfo[i].subname = pg_strdup(replslotname);
+
+ /* Create replication slot on publisher. */
+ if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
+ pg_log_info("create replication slot \"%s\" on publisher", replslotname);
+ else
+ return false;
+
+ disconnect_database(conn);
+ }
+
+ return true;
+}
+
+/*
+ * Is the target server ready for logical replication?
+ */
+static bool
+setup_subscriber(LogicalRepInfo *dbinfo)
+{
+ PGconn *conn;
+ PGresult *res;
+
+ int max_lrworkers;
+ int max_repslots;
+ int max_wprocs;
+
+ pg_log_info("checking settings on subscriber");
+
+ /*
+ * Logical replication requires a few parameters to be set on subscriber.
+ * Since these parameters are not a requirement for physical replication,
+ * we should check it to make sure it won't fail.
+ *
+ * max_replication_slots >= number of dbs to be converted
+ * max_logical_replication_workers >= number of dbs to be converted
+ * max_worker_processes >= 1 + number of dbs to be converted
+ */
+ conn = connect_database(dbinfo[0].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT setting FROM pg_settings WHERE name IN ('max_logical_replication_workers', 'max_replication_slots', 'max_worker_processes', 'primary_slot_name') ORDER BY name");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscriber settings: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ max_lrworkers = atoi(PQgetvalue(res, 0, 0));
+ max_repslots = atoi(PQgetvalue(res, 1, 0));
+ max_wprocs = atoi(PQgetvalue(res, 2, 0));
+ if (strcmp(PQgetvalue(res, 3, 0), "") != 0)
+ primary_slot_name = pg_strdup(PQgetvalue(res, 3, 0));
+
+ pg_log_debug("subscriber: max_logical_replication_workers: %d", max_lrworkers);
+ pg_log_debug("subscriber: max_replication_slots: %d", max_repslots);
+ pg_log_debug("subscriber: max_worker_processes: %d", max_wprocs);
+ pg_log_debug("subscriber: primary_slot_name: %s", primary_slot_name);
+
+ PQclear(res);
+
+ disconnect_database(conn);
+
+ if (max_repslots < num_dbs)
+ {
+ pg_log_error("subscriber requires %d replication slots, but only %d remain", num_dbs, max_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", num_dbs);
+ return false;
+ }
+
+ if (max_lrworkers < num_dbs)
+ {
+ pg_log_error("subscriber requires %d logical replication workers, but only %d remain", num_dbs, max_lrworkers);
+ pg_log_error_hint("Consider increasing max_logical_replication_workers to at least %d.", num_dbs);
+ return false;
+ }
+
+ if (max_wprocs < num_dbs + 1)
+ {
+ pg_log_error("subscriber requires %d worker processes, but only %d remain", num_dbs + 1, max_wprocs);
+ pg_log_error_hint("Consider increasing max_worker_processes to at least %d.", num_dbs + 1);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Create a logical replication slot and returns a consistent LSN. The returned
+ * LSN might be used to catch up the subscriber up to the required point.
+ *
+ * CreateReplicationSlot() is not used because it does not provide the one-row
+ * result set that contains the consistent LSN.
+ */
+static char *
+create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res = NULL;
+ char *lsn = NULL;
+ bool transient_replslot = false;
+
+ Assert(conn != NULL);
+
+ /*
+ * If no slot name is informed, it is a transient replication slot used
+ * only for catch up purposes.
+ */
+ if (slot_name[0] == '\0')
+ {
+ snprintf(slot_name, NAMEDATALEN, "pg_subscriber_%d_startpoint",
+ (int) getpid());
+ transient_replslot = true;
+ }
+
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
+ appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQresultErrorMessage(res));
+ return lsn;
+ }
+ }
+
+ /* for cleanup purposes */
+ if (transient_replslot)
+ made_transient_replslot = true;
+ else
+ dbinfo->made_replslot = true;
+
+ if (!dry_run)
+ {
+ lsn = pg_strdup(PQgetvalue(res, 0, 1));
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+
+ return lsn;
+}
+
+static void
+drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Reports a suitable message if pg_ctl fails.
+ */
+static void
+pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
+{
+ if (rc != 0)
+ {
+ if (WIFEXITED(rc))
+ {
+ pg_log_error("pg_ctl failed with exit code %d", WEXITSTATUS(rc));
+ }
+ else if (WIFSIGNALED(rc))
+ {
+#if defined(WIN32)
+ pg_log_error("pg_ctl was terminated by exception 0x%X", WTERMSIG(rc));
+ pg_log_error_detail("See C include file \"ntstatus.h\" for a description of the hexadecimal value.");
+#else
+ pg_log_error("pg_ctl was terminated by signal %d: %s",
+ WTERMSIG(rc), pg_strsignal(WTERMSIG(rc)));
+#endif
+ }
+ else
+ {
+ pg_log_error("pg_ctl exited with unrecognized status %d", rc);
+ }
+
+ pg_log_error_detail("The failed command was: %s", pg_ctl_cmd);
+ exit(1);
+ }
+
+ if (action)
+ pg_log_info("postmaster was started");
+ else
+ pg_log_info("postmaster was stopped");
+}
+
+/*
+ * Returns after the server finishes the recovery process.
+ *
+ * If recovery_timeout option is set, terminate abnormally without finishing
+ * the recovery process. By default, it waits forever.
+ */
+static void
+wait_for_end_recovery(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ int status = POSTMASTER_STILL_STARTING;
+ int timer = 0;
+
+ char *pg_ctl_cmd;
+ int rc;
+
+ pg_log_info("waiting the postmaster to reach the consistent state");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ for (;;)
+ {
+ bool in_recovery;
+
+ res = PQexec(conn, "SELECT pg_catalog.pg_is_in_recovery()");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain recovery progress");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("unexpected result from pg_is_in_recovery function");
+ exit(1);
+ }
+
+ in_recovery = (strcmp(PQgetvalue(res, 0, 0), "t") == 0);
+
+ PQclear(res);
+
+ /*
+ * Does the recovery process finish? In dry run mode, there is no
+ * recovery mode. Bail out as the recovery process has ended.
+ */
+ if (!in_recovery || dry_run)
+ {
+ status = POSTMASTER_READY;
+ break;
+ }
+
+ /*
+ * Bail out after recovery_timeout seconds if this option is set.
+ */
+ if (recovery_timeout > 0 && timer >= recovery_timeout)
+ {
+ pg_log_error("recovery timed out");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+ exit(1);
+ }
+
+ /* Keep waiting. */
+ pg_usleep(WAIT_INTERVAL * USEC_PER_SEC);
+
+ timer += WAIT_INTERVAL;
+ }
+
+ disconnect_database(conn);
+
+ if (status == POSTMASTER_STILL_STARTING)
+ {
+ pg_log_error("server did not end recovery");
+ exit(1);
+ }
+
+ pg_log_info("postmaster reached the consistent state");
+}
+
+/*
+ * Create a publication that includes all tables in the database.
+ */
+static void
+create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ /* Check if the publication needs to be created. */
+ appendPQExpBuffer(str,
+ "SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
+ dbinfo->pubname);
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publication information: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) == 1)
+ {
+ /*
+ * If publication name already exists and puballtables is true, let's
+ * use it. A previous run of pg_subscriber must have created this
+ * publication. Bail out.
+ */
+ if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
+ {
+ pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ return;
+ }
+ else
+ {
+ /*
+ * Unfortunately, if it reaches this code path, it will always
+ * fail (unless you decide to change the existing publication
+ * name). That's bad but it is very unlikely that the user will
+ * choose a name with pg_subscriber_ prefix followed by the exact
+ * database oid in which puballtables is false.
+ */
+ pg_log_error("publication \"%s\" does not replicate changes for all tables",
+ dbinfo->pubname);
+ pg_log_error_hint("Consider renaming this publication.");
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(str);
+
+ pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
+ dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_publication = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove publication if it couldn't finish all steps.
+ */
+static void
+drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Create a subscription with some predefined options.
+ *
+ * A replication slot was already created in a previous step. Let's use it. By
+ * default, the subscription name is used as replication slot name. It is
+ * not required to copy data. The subscription will be created but it will not
+ * be enabled now. That's because the replication progress must be set and the
+ * replication origin name (one of the function arguments) contains the
+ * subscription OID in its name. Once the subscription is created,
+ * set_replication_progress() can obtain the chosen origin name and set up its
+ * initial location.
+ */
+static void
+create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str,
+ "CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
+ "WITH (create_slot = false, copy_data = false, enabled = false)",
+ dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
+ dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_subscription = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove subscription if it couldn't finish all steps.
+ */
+static void
+drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Sets the replication progress to the consistent LSN.
+ *
+ * The subscriber caught up to the consistent LSN provided by the temporary
+ * replication slot. The goal is to set up the initial location for the logical
+ * replication that is the exact LSN that the subscriber was promoted. Once the
+ * subscription is enabled it will start streaming from that location onwards.
+ * In dry run mode, the subscription OID and LSN are set to invalid values for
+ * printing purposes.
+ */
+static void
+set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+ Oid suboid;
+ char originname[NAMEDATALEN];
+ char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+
+ Assert(conn != NULL);
+
+ appendPQExpBuffer(str,
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscription OID: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1 && !dry_run)
+ {
+ pg_log_error("could not obtain subscription OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (dry_run)
+ {
+ suboid = InvalidOid;
+ snprintf(lsnstr, sizeof(lsnstr), "%X/%X", LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ suboid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ snprintf(lsnstr, sizeof(lsnstr), "%s", lsn);
+ }
+
+ /*
+ * The origin name is defined as pg_%u. %u is the subscription OID. See
+ * ApplyWorkerMain().
+ */
+ snprintf(originname, sizeof(originname), "pg_%u", suboid);
+
+ PQclear(res);
+
+ pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
+ originname, lsnstr, dbinfo->dbname);
+
+ resetPQExpBuffer(str);
+ appendPQExpBuffer(str,
+ "SELECT pg_catalog.pg_replication_origin_advance('%s', '%s')", originname, lsnstr);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not set replication progress for the subscription \"%s\": %s",
+ dbinfo->subname, PQresultErrorMessage(res));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Enables the subscription.
+ *
+ * The subscription was created in a previous step but it was disabled. After
+ * adjusting the initial location, enabling the subscription is the last step
+ * of this setup.
+ */
+static void
+enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+int
+main(int argc, char **argv)
+{
+ static struct option long_options[] =
+ {
+ {"help", no_argument, NULL, '?'},
+ {"version", no_argument, NULL, 'V'},
+ {"pgdata", required_argument, NULL, 'D'},
+ {"publisher-conninfo", required_argument, NULL, 'P'},
+ {"subscriber-conninfo", required_argument, NULL, 'S'},
+ {"database", required_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"recovery-timeout", required_argument, NULL, 't'},
+ {"retain", no_argument, NULL, 'r'},
+ {"verbose", no_argument, NULL, 'v'},
+ {NULL, 0, NULL, 0}
+ };
+
+ int c;
+ int option_index;
+ int rc;
+
+ char *pg_ctl_cmd;
+
+ char *base_dir;
+ char *server_start_log;
+
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+
+ char *pub_base_conninfo = NULL;
+ char *sub_base_conninfo = NULL;
+ char *dbname_conninfo = NULL;
+
+ uint64 pub_sysid;
+ uint64 sub_sysid;
+ struct stat statbuf;
+
+ PGconn *conn;
+ char *consistent_lsn;
+
+ PQExpBuffer recoveryconfcontents = NULL;
+
+ char pidfile[MAXPGPATH];
+
+ int i;
+
+ pg_logging_init(argv[0]);
+ pg_logging_set_level(PG_LOG_WARNING);
+ progname = get_progname(argv[0]);
+ set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_subscriber"));
+
+ if (argc > 1)
+ {
+ if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+ {
+ usage();
+ exit(0);
+ }
+ else if (strcmp(argv[1], "-V") == 0
+ || strcmp(argv[1], "--version") == 0)
+ {
+ puts("pg_subscriber (PostgreSQL) " PG_VERSION);
+ exit(0);
+ }
+ }
+
+ atexit(cleanup_objects_atexit);
+
+ /*
+ * Don't allow it to be run as root. It uses pg_ctl which does not allow
+ * it either.
+ */
+#ifndef WIN32
+ if (geteuid() == 0)
+ {
+ pg_log_error("cannot be executed by \"root\"");
+ pg_log_error_hint("You must run %s as the PostgreSQL superuser.",
+ progname);
+ exit(1);
+ }
+#endif
+
+ while ((c = getopt_long(argc, argv, "D:P:S:d:nv",
+ long_options, &option_index)) != -1)
+ {
+ switch (c)
+ {
+ case 'D':
+ subscriber_dir = pg_strdup(optarg);
+ break;
+ case 'P':
+ pub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'S':
+ sub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'd':
+ /* Ignore duplicated database names. */
+ if (!simple_string_list_member(&database_names, optarg))
+ {
+ simple_string_list_append(&database_names, optarg);
+ num_dbs++;
+ }
+ break;
+ case 'n':
+ dry_run = true;
+ break;
+ case 'r':
+ retain = true;
+ break;
+ case 't':
+ recovery_timeout = atoi(optarg);
+ break;
+ case 'v':
+ pg_logging_increase_verbosity();
+ break;
+ default:
+ /* getopt_long already emitted a complaint */
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Any non-option arguments?
+ */
+ if (optind < argc)
+ {
+ pg_log_error("too many command-line arguments (first is \"%s\")",
+ argv[optind]);
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Required arguments
+ */
+ if (subscriber_dir == NULL)
+ {
+ pg_log_error("no subscriber data directory specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Parse connection string. Build a base connection string that might be
+ * reused by multiple databases.
+ */
+ if (pub_conninfo_str == NULL)
+ {
+ /*
+ * TODO use primary_conninfo (if available) from subscriber and
+ * extract publisher connection string. Assume that there are
+ * identical entries for physical and logical replication. If there is
+ * not, we would fail anyway.
+ */
+ pg_log_error("no publisher connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
+ "publisher");
+ if (pub_base_conninfo == NULL)
+ exit(1);
+
+ if (sub_conninfo_str == NULL)
+ {
+ pg_log_error("no subscriber connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
+ if (sub_base_conninfo == NULL)
+ exit(1);
+
+ if (database_names.head == NULL)
+ {
+ pg_log_info("no database was specified");
+
+ /*
+ * If --database option is not provided, try to obtain the dbname from
+ * the publisher conninfo. If dbname parameter is not available, error
+ * out.
+ */
+ if (dbname_conninfo)
+ {
+ simple_string_list_append(&database_names, dbname_conninfo);
+ num_dbs++;
+
+ pg_log_info("database \"%s\" was extracted from the publisher connection string",
+ dbname_conninfo);
+ }
+ else
+ {
+ pg_log_error("no database name specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
+ */
+ if (!get_exec_path(argv[0]))
+ exit(1);
+
+ /* rudimentary check for a data directory. */
+ if (!check_data_directory(subscriber_dir))
+ exit(1);
+
+ /* Store database information for publisher and subscriber. */
+ dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+
+ /*
+ * Check if the subscriber data directory has the same system identifier
+ * than the publisher data directory.
+ */
+ pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
+ sub_sysid = get_control_from_datadir(subscriber_dir);
+ if (pub_sysid != sub_sysid)
+ {
+ pg_log_error("subscriber data directory is not a copy of the source database cluster");
+ exit(1);
+ }
+
+ /*
+ * Create the output directory to store any data generated by this tool.
+ */
+ base_dir = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("directory path for subscriber is too long");
+ exit(1);
+ }
+
+ if (mkdir(base_dir, pg_dir_create_mode) < 0 && errno != EEXIST)
+ {
+ pg_log_error("could not create directory \"%s\": %m", base_dir);
+ exit(1);
+ }
+
+ /* subscriber PID file. */
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+
+ /*
+ * The standby server must be running. That's because some checks will be
+ * done (is it ready for a logical replication setup?). After that, stop
+ * the subscriber in preparation to modify some recovery parameters that
+ * require a restart.
+ */
+ if (stat(pidfile, &statbuf) == 0)
+ {
+ /*
+ * Check if the standby server is ready for logical replication.
+ */
+ if (!setup_subscriber(dbinfo))
+ exit(1);
+
+ /*
+ * Check if the primary server is ready for logical replication and
+ * create the required objects for each database on publisher. This
+ * step is here mainly because if we stop the standby we cannot verify
+ * if the primary slot is in use. We could use an extra connection for
+ * it but it doesn't seem worth.
+ */
+ if (!setup_publisher(dbinfo))
+ exit(1);
+
+ pg_log_info("standby is up and running");
+ pg_log_info("stopping the server to start the transformation steps");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+ }
+ else
+ {
+ pg_log_error("standby is not running");
+ pg_log_error_hint("Start the standby and try again.");
+ exit(1);
+ }
+
+ /*
+ * Create a logical replication slot to get a consistent LSN.
+ *
+ * This consistent LSN will be used later to advanced the recently created
+ * replication slots. We cannot use the last created replication slot
+ * because the consistent LSN should be obtained *after* the base backup
+ * finishes (and the base backup should include the logical replication
+ * slots).
+ *
+ * XXX we should probably use the last created replication slot to get a
+ * consistent LSN but it should be changed after adding pg_basebackup
+ * support.
+ *
+ * A temporary replication slot is not used here to avoid keeping a
+ * replication connection open (depending when base backup was taken, the
+ * connection should be open for a few hours).
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+ consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
+ temp_replslot);
+
+ /*
+ * Write recovery parameters.
+ *
+ * Despite of the recovery parameters will be written to the subscriber,
+ * use a publisher connection for the follwing recovery functions. The
+ * connection is only used to check the current server version (physical
+ * replica, same server version). The subscriber is not running yet. In
+ * dry run mode, the recovery parameters *won't* be written. An invalid
+ * LSN is used for printing purposes.
+ */
+ recoveryconfcontents = GenerateRecoveryConfig(conn, NULL);
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_inclusive = true\n");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_action = promote\n");
+
+ if (dry_run)
+ {
+ appendPQExpBuffer(recoveryconfcontents, "# dry run mode");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%X/%X'\n",
+ LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
+ consistent_lsn);
+ WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ }
+ disconnect_database(conn);
+
+ pg_log_debug("recovery parameters:\n%s", recoveryconfcontents->data);
+
+ /*
+ * Start subscriber and wait until accepting connections.
+ */
+ pg_log_info("starting the subscriber");
+
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ server_start_log = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(server_start_log, MAXPGPATH, "%s/%s/server_start_%s.log", subscriber_dir, PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+
+ pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, subscriber_dir, server_start_log);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+
+ /*
+ * Waiting the subscriber to be promoted.
+ */
+ wait_for_end_recovery(dbinfo[0].subconninfo);
+
+ /*
+ * Create a subscription for each database.
+ */
+ for (i = 0; i < num_dbs; i++)
+ {
+ /* Connect to subscriber. */
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ create_subscription(conn, &dbinfo[i]);
+
+ /* Set the replication progress to the correct LSN. */
+ set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+
+ /* Enable subscription. */
+ enable_subscription(conn, &dbinfo[i]);
+
+ disconnect_database(conn);
+ }
+
+ /*
+ * The transient replication slot is no longer required. Drop it.
+ *
+ * If the physical replication slot exists, drop it.
+ *
+ * XXX we might not fail here. Instead, we provide a warning so the user
+ * eventually drops the replication slot later.
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ {
+ if (primary_slot_name != NULL)
+ pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
+ pg_log_warning("could not drop transient replication slot \"%s\" on publisher", temp_replslot);
+ pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
+ }
+ else
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ if (primary_slot_name != NULL)
+ drop_replication_slot(conn, &dbinfo[0], primary_slot_name);
+ disconnect_database(conn);
+ }
+
+ /*
+ * Stop the subscriber.
+ */
+ pg_log_info("stopping the subscriber");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+
+ /*
+ * Change system identifier.
+ */
+ modify_sysid(pg_resetwal_path, subscriber_dir);
+
+ /*
+ * The log file is kept if retain option is specified or this tool does
+ * not run successfully. Otherwise, log file is removed.
+ */
+ if (!retain)
+ unlink(server_start_log);
+
+ success = true;
+
+ pg_log_info("Done!");
+
+ return 0;
+}
diff --git a/src/bin/pg_basebackup/t/040_pg_subscriber.pl b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
new file mode 100644
index 0000000000..4ebff76b2d
--- /dev/null
+++ b/src/bin/pg_basebackup/t/040_pg_subscriber.pl
@@ -0,0 +1,44 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test checking options of pg_subscriber.
+#
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+program_help_ok('pg_subscriber');
+program_version_ok('pg_subscriber');
+program_options_handling_ok('pg_subscriber');
+
+my $datadir = PostgreSQL::Test::Utils::tempdir;
+
+command_fails(['pg_subscriber'],
+ 'no subscriber data directory specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--pgdata', $datadir
+ ],
+ 'no publisher connection string specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--dry-run',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres'
+ ],
+ 'no subscriber connection string specified');
+command_fails(
+ [
+ 'pg_subscriber',
+ '--verbose',
+ '--pgdata', $datadir,
+ '--publisher-conninfo', 'dbname=postgres',
+ '--subscriber-conninfo', 'dbname=postgres'
+ ],
+ 'no database name specified');
+
+done_testing();
diff --git a/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
new file mode 100644
index 0000000000..fbcd0fc82b
--- /dev/null
+++ b/src/bin/pg_basebackup/t/041_pg_subscriber_standby.pl
@@ -0,0 +1,139 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test using a standby server as the subscriber.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node_p;
+my $node_f;
+my $node_s;
+my $result;
+
+# Set up node P as primary
+$node_p = PostgreSQL::Test::Cluster->new('node_p');
+$node_p->init(allows_streaming => 'logical');
+$node_p->start;
+
+# Set up node F as about-to-fail node
+# The extra option forces it to initialize a new cluster instead of copying a
+# previously initdb's cluster.
+$node_f = PostgreSQL::Test::Cluster->new('node_f');
+$node_f->init(allows_streaming => 'logical', extra => [ '--no-instructions' ]);
+$node_f->start;
+
+# On node P
+# - create databases
+# - create test tables
+# - insert a row
+$node_p->safe_psql(
+ 'postgres', q(
+ CREATE DATABASE pg1;
+ CREATE DATABASE pg2;
+));
+$node_p->safe_psql('pg1', 'CREATE TABLE tbl1 (a text)');
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('first row')");
+$node_p->safe_psql('pg2', 'CREATE TABLE tbl2 (a text)');
+
+# Set up node S as standby linking to node P
+$node_p->backup('backup_1');
+$node_s = PostgreSQL::Test::Cluster->new('node_s');
+$node_s->init_from_backup($node_p, 'backup_1', has_streaming => 1);
+$node_s->append_conf('postgresql.conf', 'log_min_messages = debug2');
+$node_s->set_standby_mode();
+$node_s->start;
+
+# Insert another row on node P and wait node S to catch up
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('second row')");
+$node_p->wait_for_replay_catchup($node_s);
+
+# Run pg_subscriber on about-to-fail node F
+command_fails(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $node_f->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_f->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'subscriber data directory is not a copy of the source database cluster');
+
+# dry run mode on node S
+command_ok(
+ [
+ 'pg_subscriber', '--verbose', '--dry-run',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_subscriber --dry-run on node S');
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+# Check if node S is still a standby
+is($node_s->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
+ 't', 'standby is in recovery');
+
+# Run pg_subscriber on node S
+command_ok(
+ [
+ 'pg_subscriber', '--verbose',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-conninfo', $node_p->connstr('pg1'),
+ '--subscriber-conninfo', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_subscriber on node S');
+
+# Insert rows on P
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('third row')");
+$node_p->safe_psql('pg2', "INSERT INTO tbl2 VALUES('row 1')");
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+
+# Get subscription names
+$result = $node_s->safe_psql(
+ 'postgres', qq(
+ SELECT subname FROM pg_subscription WHERE subname ~ '^pg_subscriber_'
+));
+my @subnames = split("\n", $result);
+
+# Wait subscriber to catch up
+$node_s->wait_for_subscription_sync($node_p, $subnames[0]);
+$node_s->wait_for_subscription_sync($node_p, $subnames[1]);
+
+# Check result on database pg1
+$result = $node_s->safe_psql('pg1', 'SELECT * FROM tbl1');
+is( $result, qq(first row
+second row
+third row),
+ 'logical replication works on database pg1');
+
+# Check result on database pg2
+$result = $node_s->safe_psql('pg2', 'SELECT * FROM tbl2');
+is( $result, qq(row 1),
+ 'logical replication works on database pg2');
+
+# Different system identifier?
+my $sysid_p = $node_p->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+my $sysid_s = $node_s->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+ok($sysid_p != $sysid_s, 'system identifier was changed');
+
+# clean up
+$node_p->teardown_node;
+$node_s->teardown_node;
+
+done_testing();
--
2.43.0
[application/octet-stream] v10-0002-fix-getopt-options.patch (761B, ../../TY3PR01MB98891E7735141FE8760CEC4AF57E2@TY3PR01MB9889.jpnprd01.prod.outlook.com/3-v10-0002-fix-getopt-options.patch)
download | inline diff:
From 722a2ba9bb647559def5f6d826fe9886973910d4 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <[email protected]>
Date: Mon, 29 Jan 2024 08:43:11 +0000
Subject: [PATCH v10 2/4] fix getopt options
---
src/bin/pg_basebackup/pg_subscriber.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index cb97dbda5e..df770f0fc8 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -1448,7 +1448,7 @@ main(int argc, char **argv)
}
#endif
- while ((c = getopt_long(argc, argv, "D:P:S:d:nv",
+ while ((c = getopt_long(argc, argv, "D:P:S:d:nrt:v",
long_options, &option_index)) != -1)
{
switch (c)
--
2.43.0
[application/octet-stream] v10-0003-add-start_standby-stop_standby-functions.patch (4.6K, ../../TY3PR01MB98891E7735141FE8760CEC4AF57E2@TY3PR01MB9889.jpnprd01.prod.outlook.com/4-v10-0003-add-start_standby-stop_standby-functions.patch)
download | inline diff:
From e17e392b68011f9173ddaa9bc88009a8e5629c44 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <[email protected]>
Date: Mon, 29 Jan 2024 08:27:48 +0000
Subject: [PATCH v10 3/4] add start_standby/stop_standby functions
---
src/bin/pg_basebackup/pg_subscriber.c | 104 +++++++++++++++++---------
1 file changed, 69 insertions(+), 35 deletions(-)
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index df770f0fc8..f535e7160f 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -75,6 +75,9 @@ static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void start_standby(char *server_start_log);
+static void stop_standby(void);
+
#define USEC_PER_SEC 1000000
#define WAIT_INTERVAL 1 /* 1 second */
@@ -1363,6 +1366,68 @@ enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
destroyPQExpBuffer(str);
}
+/*
+ * Start a standby server with given logfile. This also decides filename if not
+ * determined yet. If the start is failed, the process exits with error
+ * messages.
+ */
+static void
+start_standby(char *server_start_log)
+{
+ int rc;
+ char *pg_ctl_cmd;
+
+ Assert(server_start_log != NULL);
+
+ if (server_start_log[0] == '\0')
+ {
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ len = snprintf(server_start_log, MAXPGPATH,
+ "%s/%s/server_start_%s.log", subscriber_dir,
+ PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+ }
+
+ pg_log_info("starting the standby server");
+ pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path,
+ subscriber_dir, server_start_log);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+}
+
+/*
+ * Stop a standby server. If the stop is failed, the process exits with error
+ * messages.
+ */
+static void
+stop_standby(void)
+{
+ int rc;
+ char *pg_ctl_cmd;
+
+ pg_log_info("stopping the standby server");
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path,
+ subscriber_dir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+}
+
int
main(int argc, char **argv)
{
@@ -1383,16 +1448,10 @@ main(int argc, char **argv)
int c;
int option_index;
- int rc;
-
- char *pg_ctl_cmd;
char *base_dir;
- char *server_start_log;
+ char server_start_log[MAXPGPATH] = {0};
- char timebuf[128];
- struct timeval time;
- time_t tt;
int len;
char *pub_base_conninfo = NULL;
@@ -1638,9 +1697,7 @@ main(int argc, char **argv)
pg_log_info("standby is up and running");
pg_log_info("stopping the server to start the transformation steps");
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
- rc = system(pg_ctl_cmd);
- pg_ctl_status(pg_ctl_cmd, rc, 0);
+ stop_standby();
}
else
{
@@ -1705,26 +1762,7 @@ main(int argc, char **argv)
/*
* Start subscriber and wait until accepting connections.
*/
- pg_log_info("starting the subscriber");
-
- /* append timestamp with ISO 8601 format. */
- gettimeofday(&time, NULL);
- tt = (time_t) time.tv_sec;
- strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
- snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
- ".%03d", (int) (time.tv_usec / 1000));
-
- server_start_log = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(server_start_log, MAXPGPATH, "%s/%s/server_start_%s.log", subscriber_dir, PGS_OUTPUT_DIR, timebuf);
- if (len >= MAXPGPATH)
- {
- pg_log_error("log file path is too long");
- exit(1);
- }
-
- pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, subscriber_dir, server_start_log);
- rc = system(pg_ctl_cmd);
- pg_ctl_status(pg_ctl_cmd, rc, 1);
+ start_standby(server_start_log);
/*
* Waiting the subscriber to be promoted.
@@ -1779,11 +1817,7 @@ main(int argc, char **argv)
/*
* Stop the subscriber.
*/
- pg_log_info("stopping the subscriber");
-
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
- rc = system(pg_ctl_cmd);
- pg_ctl_status(pg_ctl_cmd, rc, 0);
+ stop_standby();
/*
* Change system identifier.
--
2.43.0
[application/octet-stream] v10-0004-Divide-LogicalReplInfo-into-some-strcutures.patch (48.0K, ../../TY3PR01MB98891E7735141FE8760CEC4AF57E2@TY3PR01MB9889.jpnprd01.prod.outlook.com/5-v10-0004-Divide-LogicalReplInfo-into-some-strcutures.patch)
download | inline diff:
From afd8eb318c40e4602268cf5a82b209cb7fa07af2 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <[email protected]>
Date: Mon, 29 Jan 2024 07:03:59 +0000
Subject: [PATCH v10 4/4] Divide LogicalReplInfo into some strcutures
---
src/bin/pg_basebackup/pg_subscriber.c | 674 ++++++++++++++------------
src/tools/pgindent/typedefs.list | 4 +
2 files changed, 374 insertions(+), 304 deletions(-)
diff --git a/src/bin/pg_basebackup/pg_subscriber.c b/src/bin/pg_basebackup/pg_subscriber.c
index f535e7160f..a8df8d5135 100644
--- a/src/bin/pg_basebackup/pg_subscriber.c
+++ b/src/bin/pg_basebackup/pg_subscriber.c
@@ -32,51 +32,72 @@
#define PGS_OUTPUT_DIR "pg_subscriber_output.d"
-typedef struct LogicalRepInfo
+typedef struct LogicalRepPerdbInfo
{
- Oid oid; /* database OID */
- char *dbname; /* database name */
- char *pubconninfo; /* publication connection string for logical
- * replication */
- char *subconninfo; /* subscription connection string for logical
- * replication */
- char *pubname; /* publication name */
- char *subname; /* subscription name (also replication slot
- * name) */
-
- bool made_replslot; /* replication slot was created */
- bool made_publication; /* publication was created */
- bool made_subscription; /* subscription was created */
-} LogicalRepInfo;
+ Oid oid;
+ char *dbname;
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepPerdbInfo;
+
+typedef struct LogicalRepPerdbInfoArr
+{
+ LogicalRepPerdbInfo *perdb; /* array of db infos */
+ int ndbs; /* number of db infos */
+} LogicalRepPerdbInfoArr;
+
+typedef struct PrimaryInfo
+{
+ char *base_conninfo;
+ uint64 sysid;
+ bool made_transient_replslot;
+} PrimaryInfo;
+
+typedef struct StandbyInfo
+{
+ char *base_conninfo;
+ char *bindir;
+ char *pgdata;
+ char *primary_slot_name;
+ char *server_log;
+ uint64 sysid;
+} StandbyInfo;
static void cleanup_objects_atexit(void);
static void usage();
static char *get_base_conninfo(char *conninfo, char *dbname,
const char *noderole);
-static bool get_exec_path(const char *path);
+static bool get_exec_base_path(const char *path);
static bool check_data_directory(const char *datadir);
static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
-static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
-static PGconn *connect_database(const char *conninfo);
+static void store_db_names(LogicalRepPerdbInfo **perdb, int ndbs);
+static PGconn *connect_database(const char *base_conninfo, const char*dbname);
static void disconnect_database(PGconn *conn);
-static uint64 get_sysid_from_conn(const char *conninfo);
-static uint64 get_control_from_datadir(const char *datadir);
-static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
-static bool setup_publisher(LogicalRepInfo *dbinfo);
-static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
- char *slot_name);
-static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static void get_sysid_for_primary(PrimaryInfo *primary, char *dbname);
+static void get_sysid_for_standby(StandbyInfo *standby);
+static void modify_sysid(const char *bindir, const char *datadir);
+static bool setup_publisher(PrimaryInfo *primary, LogicalRepPerdbInfoArr *dbarr);
+static char *create_logical_replication_slot(PGconn *conn, bool temporary,
+ LogicalRepPerdbInfo *perdb);
+static void drop_replication_slot(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *slot_name);
static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
-static void wait_for_end_recovery(const char *conninfo);
-static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
-static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
-static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
-static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-
-static void start_standby(char *server_start_log);
-static void stop_standby(void);
+static void wait_for_end_recovery(StandbyInfo *standby,
+ const char *dbname);
+static void create_publication(PGconn *conn, PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb);
+static void drop_publication(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void create_subscription(PGconn *conn, StandbyInfo *standby,
+ PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb);
+static void drop_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void set_replication_progress(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb);
+
+static void start_standby(StandbyInfo *standby);
+static void stop_standby(StandbyInfo *standby);
#define USEC_PER_SEC 1000000
#define WAIT_INTERVAL 1 /* 1 second */
@@ -84,25 +105,18 @@ static void stop_standby(void);
/* Options */
static const char *progname;
-static char *subscriber_dir = NULL;
static char *pub_conninfo_str = NULL;
static char *sub_conninfo_str = NULL;
static SimpleStringList database_names = {NULL, NULL};
-static char *primary_slot_name = NULL;
static bool dry_run = false;
static bool retain = false;
static int recovery_timeout = 0;
static bool success = false;
-static char *pg_ctl_path = NULL;
-static char *pg_resetwal_path = NULL;
-
-static LogicalRepInfo *dbinfo;
-static int num_dbs = 0;
-
-static char temp_replslot[NAMEDATALEN] = {0};
-static bool made_transient_replslot = false;
+static LogicalRepPerdbInfoArr dbarr;
+static PrimaryInfo primary;
+static StandbyInfo standby;
enum WaitPMResult
{
@@ -112,6 +126,30 @@ enum WaitPMResult
POSTMASTER_FAILED
};
+/*
+ * Build the replication slot and subscription name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 36 characters
+ * (14 + 10 + 1 + 10 + '\0'). System identifier is included to reduce the
+ * probability of collision. By default, subscription name is used as
+ * replication slot name.
+ */
+static inline void
+get_subscription_name(Oid oid, int pid, char *subname, Size szsub)
+{
+ snprintf(subname, szsub, "pg_subscriber_%u_%d", oid, pid);
+}
+
+/*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 35 characters (14 + 10 +
+ * '\0').
+ */
+static inline void
+get_publication_name(Oid oid, char *pubname, Size szpub)
+{
+ snprintf(pubname, szpub, "pg_subscriber_%u", oid);
+}
+
/*
* Cleanup objects that were created by pg_subscriber if there is an error.
@@ -129,38 +167,51 @@ cleanup_objects_atexit(void)
if (success)
return;
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
- if (dbinfo[i].made_subscription)
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
+
+ if (perdb->made_subscription)
{
- conn = connect_database(dbinfo[i].subconninfo);
+ conn = connect_database(standby.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- drop_subscription(conn, &dbinfo[i]);
+ drop_subscription(conn, perdb);
disconnect_database(conn);
}
}
- if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ if (perdb->made_publication || perdb->made_replslot)
{
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- if (dbinfo[i].made_publication)
- drop_publication(conn, &dbinfo[i]);
- if (dbinfo[i].made_replslot)
- drop_replication_slot(conn, &dbinfo[i], NULL);
- disconnect_database(conn);
+ if (perdb->made_publication)
+ drop_publication(conn, perdb);
+ if (perdb->made_replslot)
+ {
+ char replslotname[NAMEDATALEN];
+
+ get_subscription_name(perdb->oid, (int) getpid(),
+ replslotname, NAMEDATALEN);
+ drop_replication_slot(conn, perdb, replslotname);
+ }
}
}
}
- if (made_transient_replslot)
+ if (primary.made_transient_replslot)
{
- conn = connect_database(dbinfo[0].pubconninfo);
+ char transient_replslot[NAMEDATALEN];
+
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
+
if (conn != NULL)
{
- drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ snprintf(transient_replslot, NAMEDATALEN, "pg_subscriber_%d_startpoint",
+ (int) getpid());
+
+ drop_replication_slot(conn, &dbarr.perdb[0], transient_replslot);
disconnect_database(conn);
}
}
@@ -246,15 +297,16 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
}
/*
- * Get the absolute path from other PostgreSQL binaries (pg_ctl and
- * pg_resetwal) that is used by it.
+ * Get the absolute binary path from another PostgreSQL binary (pg_ctl) and set
+ * to StandbyInfo.
*/
static bool
-get_exec_path(const char *path)
+get_exec_base_path(const char *path)
{
- int rc;
+ int rc;
+ char pg_ctl_path[MAXPGPATH];
+ char *p;
- pg_ctl_path = pg_malloc(MAXPGPATH);
rc = find_other_exec(path, "pg_ctl",
"pg_ctl (PostgreSQL) " PG_VERSION "\n",
pg_ctl_path);
@@ -279,30 +331,12 @@ get_exec_path(const char *path)
pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
- pg_resetwal_path = pg_malloc(MAXPGPATH);
- rc = find_other_exec(path, "pg_resetwal",
- "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
- pg_resetwal_path);
- if (rc < 0)
- {
- char full_path[MAXPGPATH];
+ /* Extract the directory part from the path */
+ p = strrchr(pg_ctl_path, 'p');
+ Assert(p);
- if (find_my_exec(path, full_path) < 0)
- strlcpy(full_path, progname, sizeof(full_path));
- if (rc == -1)
- pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
- "same directory as \"%s\".\n"
- "Check your installation.",
- "pg_resetwal", progname, full_path);
- else
- pg_log_error("The program \"%s\" was found by \"%s\"\n"
- "but was not the same version as %s.\n"
- "Check your installation.",
- "pg_resetwal", full_path, progname);
- return false;
- }
-
- pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+ *p = '\0';
+ standby.bindir = pg_strdup(pg_ctl_path);
return true;
}
@@ -366,49 +400,35 @@ concat_conninfo_dbname(const char *conninfo, const char *dbname)
}
/*
- * Store publication and subscription information.
+ * Initialize per-db structure and store the name of databases.
*/
-static LogicalRepInfo *
-store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+static void
+store_db_names(LogicalRepPerdbInfo **perdb, int ndbs)
{
- LogicalRepInfo *dbinfo;
- SimpleStringListCell *cell;
- int i = 0;
+ SimpleStringListCell *cell;
+ int i = 0;
- dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+ dbarr.perdb = (LogicalRepPerdbInfo *) pg_malloc0(ndbs *
+ sizeof(LogicalRepPerdbInfo));
for (cell = database_names.head; cell; cell = cell->next)
{
- char *conninfo;
-
- /* Publisher. */
- conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
- dbinfo[i].pubconninfo = conninfo;
- dbinfo[i].dbname = cell->val;
- dbinfo[i].made_replslot = false;
- dbinfo[i].made_publication = false;
- dbinfo[i].made_subscription = false;
- /* other struct fields will be filled later. */
-
- /* Subscriber. */
- conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
- dbinfo[i].subconninfo = conninfo;
-
+ (*perdb)[i].dbname = pg_strdup(cell->val);
i++;
}
-
- return dbinfo;
}
static PGconn *
-connect_database(const char *conninfo)
+connect_database(const char *base_conninfo, const char*dbname)
{
PGconn *conn;
PGresult *res;
- const char *rconninfo;
+ char *rconninfo;
+ char *concat_conninfo = concat_conninfo_dbname(base_conninfo,
+ dbname);
/* logical replication mode */
- rconninfo = psprintf("%s replication=database", conninfo);
+ rconninfo = psprintf("%s replication=database", concat_conninfo);
conn = PQconnectdb(rconninfo);
if (PQstatus(conn) != CONNECTION_OK)
@@ -426,6 +446,8 @@ connect_database(const char *conninfo)
}
PQclear(res);
+ pfree(rconninfo);
+ pfree(concat_conninfo);
return conn;
}
@@ -438,19 +460,18 @@ disconnect_database(PGconn *conn)
}
/*
- * Obtain the system identifier using the provided connection. It will be used
- * to compare if a data directory is a clone of another one.
+ * Obtain the system identifier from the primary server. It will be used to
+ * compare if a data directory is a clone of another one.
*/
-static uint64
-get_sysid_from_conn(const char *conninfo)
+static void
+get_sysid_for_primary(PrimaryInfo *primary, char *dbname)
{
PGconn *conn;
PGresult *res;
- uint64 sysid;
pg_log_info("getting system identifier from publisher");
- conn = connect_database(conninfo);
+ conn = connect_database(primary->base_conninfo, dbname);
if (conn == NULL)
exit(1);
@@ -473,43 +494,40 @@ get_sysid_from_conn(const char *conninfo)
exit(1);
}
- sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+ primary->sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
- pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+ pg_log_info("system identifier is " UINT64_FORMAT " on publisher",
+ primary->sysid);
disconnect_database(conn);
-
- return sysid;
}
/*
- * Obtain the system identifier from control file. It will be used to compare
- * if a data directory is a clone of another one. This routine is used locally
- * and avoids a replication connection.
+ * Obtain the system identifier from a standby server. It will be used to
+ * compare if a data directory is a clone of another one. This routine is used
+ * locally and avoids a replication connection.
*/
-static uint64
-get_control_from_datadir(const char *datadir)
+static void
+get_sysid_for_standby(StandbyInfo *standby)
{
ControlFileData *cf;
bool crc_ok;
- uint64 sysid;
pg_log_info("getting system identifier from subscriber");
- cf = get_controlfile(datadir, &crc_ok);
+ cf = get_controlfile(standby->pgdata, &crc_ok);
if (!crc_ok)
{
pg_log_error("control file appears to be corrupt");
exit(1);
}
- sysid = cf->system_identifier;
+ standby->sysid = cf->system_identifier;
- pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+ pg_log_info("system identifier is " UINT64_FORMAT " on subscriber",
+ standby->sysid);
pfree(cf);
-
- return sysid;
}
/*
@@ -518,7 +536,7 @@ get_control_from_datadir(const char *datadir)
* files from one of the systems might be used in the other one.
*/
static void
-modify_sysid(const char *pg_resetwal_path, const char *datadir)
+modify_sysid(const char *bindir, const char *datadir)
{
ControlFileData *cf;
bool crc_ok;
@@ -553,7 +571,7 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
pg_log_info("running pg_resetwal on the subscriber");
- cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+ cmd_str = psprintf("\"%s/pg_resetwal\" -D \"%s\"", bindir, datadir);
pg_log_debug("command is: %s", cmd_str);
@@ -574,7 +592,7 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
* publications and replication slots in preparation for logical replication.
*/
static bool
-setup_publisher(LogicalRepInfo *dbinfo)
+setup_publisher(PrimaryInfo *primary, LogicalRepPerdbInfoArr *dbarr)
{
PGconn *conn;
PGresult *res;
@@ -597,7 +615,7 @@ setup_publisher(LogicalRepInfo *dbinfo)
* max_replication_slots >= current + number of dbs to be converted
* max_wal_senders >= current + number of dbs to be converted
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary->base_conninfo, dbarr->perdb[0].dbname);
if (conn == NULL)
exit(1);
@@ -635,10 +653,11 @@ setup_publisher(LogicalRepInfo *dbinfo)
* use after the transformation, hence, it will be removed at the end of
* this process.
*/
- if (primary_slot_name)
+ if (standby.primary_slot_name)
{
appendPQExpBuffer(str,
- "SELECT 1 FROM pg_replication_slots WHERE active AND slot_name = '%s'", primary_slot_name);
+ "SELECT 1 FROM pg_replication_slots WHERE active AND slot_name = '%s'",
+ standby.primary_slot_name);
pg_log_debug("command is: %s", str->data);
@@ -653,13 +672,14 @@ setup_publisher(LogicalRepInfo *dbinfo)
{
pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
PQntuples(res), 1);
- pg_free(primary_slot_name); /* it is not being used. */
- primary_slot_name = NULL;
+ pg_free(standby.primary_slot_name); /* it is not being used. */
+ standby.primary_slot_name = NULL;
return false;
}
else
{
- pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+ pg_log_info("primary has replication slot \"%s\"",
+ standby.primary_slot_name);
}
PQclear(res);
@@ -673,26 +693,29 @@ setup_publisher(LogicalRepInfo *dbinfo)
return false;
}
- if (max_repslots - cur_repslots < num_dbs)
+ if (max_repslots - cur_repslots < dbarr->ndbs)
{
- pg_log_error("publisher requires %d replication slots, but only %d remain", num_dbs, max_repslots - cur_repslots);
- pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", cur_repslots + num_dbs);
+ pg_log_error("publisher requires %d replication slots, but only %d remain",
+ dbarr->ndbs, max_repslots - cur_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.",
+ cur_repslots + dbarr->ndbs);
return false;
}
- if (max_walsenders - cur_walsenders < num_dbs)
+ if (max_walsenders - cur_walsenders < dbarr->ndbs)
{
- pg_log_error("publisher requires %d wal sender processes, but only %d remain", num_dbs, max_walsenders - cur_walsenders);
- pg_log_error_hint("Consider increasing max_wal_senders to at least %d.", cur_walsenders + num_dbs);
+ pg_log_error("publisher requires %d wal sender processes, but only %d remain",
+ dbarr->ndbs, max_walsenders - cur_walsenders);
+ pg_log_error_hint("Consider increasing max_wal_senders to at least %d.",
+ cur_walsenders + dbarr->ndbs);
return false;
}
- for (int i = 0; i < num_dbs; i++)
+ for (int i = 0; i < dbarr->ndbs; i++)
{
- char pubname[NAMEDATALEN];
- char replslotname[NAMEDATALEN];
+ LogicalRepPerdbInfo *perdb = &dbarr->perdb[i];
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -712,43 +735,21 @@ setup_publisher(LogicalRepInfo *dbinfo)
}
/* Remember database OID. */
- dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ perdb->oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
PQclear(res);
- /*
- * Build the publication name. The name must not exceed NAMEDATALEN -
- * 1. This current schema uses a maximum of 35 characters (14 + 10 +
- * '\0').
- */
- snprintf(pubname, sizeof(pubname), "pg_subscriber_%u", dbinfo[i].oid);
- dbinfo[i].pubname = pg_strdup(pubname);
-
/*
* Create publication on publisher. This step should be executed
* *before* promoting the subscriber to avoid any transactions between
* consistent LSN and the new publication rows (such transactions
* wouldn't see the new publication rows resulting in an error).
*/
- create_publication(conn, &dbinfo[i]);
-
- /*
- * Build the replication slot name. The name must not exceed
- * NAMEDATALEN - 1. This current schema uses a maximum of 36
- * characters (14 + 10 + 1 + 10 + '\0'). System identifier is included
- * to reduce the probability of collision. By default, subscription
- * name is used as replication slot name.
- */
- snprintf(replslotname, sizeof(replslotname),
- "pg_subscriber_%u_%d",
- dbinfo[i].oid,
- (int) getpid());
- dbinfo[i].subname = pg_strdup(replslotname);
+ create_publication(conn, primary, perdb);
/* Create replication slot on publisher. */
- if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
- pg_log_info("create replication slot \"%s\" on publisher", replslotname);
- else
+ if (create_logical_replication_slot(conn, false, perdb) == NULL &&
+ !dry_run)
return false;
disconnect_database(conn);
@@ -761,7 +762,7 @@ setup_publisher(LogicalRepInfo *dbinfo)
* Is the target server ready for logical replication?
*/
static bool
-setup_subscriber(LogicalRepInfo *dbinfo)
+setup_subscriber(StandbyInfo *standby, LogicalRepPerdbInfoArr *dbarr)
{
PGconn *conn;
PGresult *res;
@@ -781,7 +782,7 @@ setup_subscriber(LogicalRepInfo *dbinfo)
* max_logical_replication_workers >= number of dbs to be converted
* max_worker_processes >= 1 + number of dbs to be converted
*/
- conn = connect_database(dbinfo[0].subconninfo);
+ conn = connect_database(standby->base_conninfo, dbarr->perdb[0].dbname);
if (conn == NULL)
exit(1);
@@ -798,35 +799,41 @@ setup_subscriber(LogicalRepInfo *dbinfo)
max_repslots = atoi(PQgetvalue(res, 1, 0));
max_wprocs = atoi(PQgetvalue(res, 2, 0));
if (strcmp(PQgetvalue(res, 3, 0), "") != 0)
- primary_slot_name = pg_strdup(PQgetvalue(res, 3, 0));
+ standby->primary_slot_name = pg_strdup(PQgetvalue(res, 3, 0));
pg_log_debug("subscriber: max_logical_replication_workers: %d", max_lrworkers);
pg_log_debug("subscriber: max_replication_slots: %d", max_repslots);
pg_log_debug("subscriber: max_worker_processes: %d", max_wprocs);
- pg_log_debug("subscriber: primary_slot_name: %s", primary_slot_name);
+ pg_log_debug("subscriber: primary_slot_name: %s", standby->primary_slot_name);
PQclear(res);
disconnect_database(conn);
- if (max_repslots < num_dbs)
+ if (max_repslots < dbarr->ndbs)
{
- pg_log_error("subscriber requires %d replication slots, but only %d remain", num_dbs, max_repslots);
- pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", num_dbs);
+ pg_log_error("subscriber requires %d replication slots, but only %d remain",
+ dbarr->ndbs, max_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.",
+ dbarr->ndbs);
return false;
}
- if (max_lrworkers < num_dbs)
+ if (max_lrworkers < dbarr->ndbs)
{
- pg_log_error("subscriber requires %d logical replication workers, but only %d remain", num_dbs, max_lrworkers);
- pg_log_error_hint("Consider increasing max_logical_replication_workers to at least %d.", num_dbs);
+ pg_log_error("subscriber requires %d logical replication workers, but only %d remain",
+ dbarr->ndbs, max_lrworkers);
+ pg_log_error_hint("Consider increasing max_logical_replication_workers to at least %d.",
+ dbarr->ndbs);
return false;
}
- if (max_wprocs < num_dbs + 1)
+ if (max_wprocs < dbarr->ndbs + 1)
{
- pg_log_error("subscriber requires %d worker processes, but only %d remain", num_dbs + 1, max_wprocs);
- pg_log_error_hint("Consider increasing max_worker_processes to at least %d.", num_dbs + 1);
+ pg_log_error("subscriber requires %d worker processes, but only %d remain",
+ dbarr->ndbs + 1, max_wprocs);
+ pg_log_error_hint("Consider increasing max_worker_processes to at least %d.",
+ dbarr->ndbs + 1);
return false;
}
@@ -841,28 +848,32 @@ setup_subscriber(LogicalRepInfo *dbinfo)
* result set that contains the consistent LSN.
*/
static char *
-create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
- char *slot_name)
+create_logical_replication_slot(PGconn *conn, bool temporary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res = NULL;
char *lsn = NULL;
- bool transient_replslot = false;
+ char slot_name[NAMEDATALEN];
Assert(conn != NULL);
/*
- * If no slot name is informed, it is a transient replication slot used
- * only for catch up purposes.
+ * Construct a name of logical replication slot. The formatting is
+ * different depends on its persistency.
+ *
+ * For persistent slots: the name must be same as the subscription.
+ * For temporary slots: OID is not needed, but another string is added.
*/
- if (slot_name[0] == '\0')
- {
+ if (temporary)
snprintf(slot_name, NAMEDATALEN, "pg_subscriber_%d_startpoint",
(int) getpid());
- transient_replslot = true;
- }
+ else
+ get_subscription_name(perdb->oid, (int) getpid(), slot_name,
+ NAMEDATALEN);
- pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"",
+ slot_name, perdb->dbname);
appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
@@ -874,17 +885,19 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
- pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
- PQresultErrorMessage(res));
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s",
+ slot_name, perdb->dbname, PQresultErrorMessage(res));
return lsn;
}
+
+ pg_log_info("create replication slot \"%s\" on publisher", slot_name);
}
/* for cleanup purposes */
- if (transient_replslot)
- made_transient_replslot = true;
+ if (temporary)
+ primary.made_transient_replslot = true;
else
- dbinfo->made_replslot = true;
+ perdb->made_replslot = true;
if (!dry_run)
{
@@ -898,14 +911,16 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
}
static void
-drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+drop_replication_slot(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *slot_name)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
Assert(conn != NULL);
- pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"",
+ slot_name, perdb->dbname);
appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
@@ -915,7 +930,8 @@ drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_nam
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s",
+ slot_name, perdb->dbname,
PQerrorMessage(conn));
PQclear(res);
@@ -968,19 +984,17 @@ pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
* the recovery process. By default, it waits forever.
*/
static void
-wait_for_end_recovery(const char *conninfo)
+wait_for_end_recovery(StandbyInfo *standby,
+ const char *dbname)
{
PGconn *conn;
PGresult *res;
int status = POSTMASTER_STILL_STARTING;
int timer = 0;
- char *pg_ctl_cmd;
- int rc;
-
pg_log_info("waiting the postmaster to reach the consistent state");
- conn = connect_database(conninfo);
+ conn = connect_database(standby->base_conninfo, dbname);
if (conn == NULL)
exit(1);
@@ -1021,9 +1035,13 @@ wait_for_end_recovery(const char *conninfo)
*/
if (recovery_timeout > 0 && timer >= recovery_timeout)
{
+ char *pg_ctl_cmd;
+ int rc;
+
pg_log_error("recovery timed out");
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, subscriber_dir);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s",
+ standby->bindir, standby->pgdata);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
@@ -1051,17 +1069,22 @@ wait_for_end_recovery(const char *conninfo)
* Create a publication that includes all tables in the database.
*/
static void
-create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+create_publication(PGconn *conn, PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
+
/* Check if the publication needs to be created. */
appendPQExpBuffer(str,
"SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
- dbinfo->pubname);
+ pubname);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
@@ -1081,7 +1104,7 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
*/
if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
{
- pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ pg_log_info("publication \"%s\" already exists", pubname);
return;
}
else
@@ -1094,7 +1117,7 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* database oid in which puballtables is false.
*/
pg_log_error("publication \"%s\" does not replicate changes for all tables",
- dbinfo->pubname);
+ pubname);
pg_log_error_hint("Consider renaming this publication.");
PQclear(res);
PQfinish(conn);
@@ -1105,9 +1128,9 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
PQclear(res);
resetPQExpBuffer(str);
- pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+ pg_log_info("creating publication \"%s\" on database \"%s\"", pubname, perdb->dbname);
- appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", pubname);
pg_log_debug("command is: %s", str->data);
@@ -1117,14 +1140,14 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
- dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ pubname, perdb->dbname, PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
}
/* for cleanup purposes */
- dbinfo->made_publication = true;
+ perdb->made_publication = true;
if (!dry_run)
PQclear(res);
@@ -1136,24 +1159,28 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* Remove publication if it couldn't finish all steps.
*/
static void
-drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+drop_publication(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
- appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+ pg_log_info("dropping publication \"%s\" on database \"%s\"",
+ pubname, perdb->dbname);
+
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", pubname);
- pg_log_debug("command is: %s", str->data);
if (!dry_run)
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s",
+ pubname, perdb->dbname, PQerrorMessage(conn));
PQclear(res);
}
@@ -1174,19 +1201,30 @@ drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* initial location.
*/
static void
-create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+create_subscription(PGconn *conn, StandbyInfo *standby,
+ PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", subname,
+ perdb->dbname);
appendPQExpBuffer(str,
"CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
"WITH (create_slot = false, copy_data = false, enabled = false)",
- dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+ subname,
+ concat_conninfo_dbname(primary->base_conninfo,
+ perdb->dbname),
+ pubname);
pg_log_debug("command is: %s", str->data);
@@ -1196,14 +1234,14 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
- dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ subname, perdb->dbname, PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
}
/* for cleanup purposes */
- dbinfo->made_subscription = true;
+ perdb->made_subscription = true;
if (!dry_run)
PQclear(res);
@@ -1215,16 +1253,20 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* Remove subscription if it couldn't finish all steps.
*/
static void
-drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+drop_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"",
+ subname, perdb->dbname);
- appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", subname);
pg_log_debug("command is: %s", str->data);
@@ -1232,7 +1274,8 @@ drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
{
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s",
+ subname, perdb->dbname, PQerrorMessage(conn));
PQclear(res);
}
@@ -1251,18 +1294,23 @@ drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* printing purposes.
*/
static void
-set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+set_replication_progress(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *lsn)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
Oid suboid;
char originname[NAMEDATALEN];
char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
appendPQExpBuffer(str,
- "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'",
+ subname);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
@@ -1303,7 +1351,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
PQclear(res);
pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
- originname, lsnstr, dbinfo->dbname);
+ originname, lsnstr, perdb->dbname);
resetPQExpBuffer(str);
appendPQExpBuffer(str,
@@ -1317,7 +1365,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
pg_log_error("could not set replication progress for the subscription \"%s\": %s",
- dbinfo->subname, PQresultErrorMessage(res));
+ subname, PQresultErrorMessage(res));
PQfinish(conn);
exit(1);
}
@@ -1336,16 +1384,19 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
* of this setup.
*/
static void
-enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+enable_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", subname,
+ perdb->dbname);
- appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", subname);
pg_log_debug("command is: %s", str->data);
@@ -1354,7 +1405,7 @@ enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
- pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ pg_log_error("could not enable subscription \"%s\": %s", subname,
PQerrorMessage(conn));
PQfinish(conn);
exit(1);
@@ -1372,20 +1423,20 @@ enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* messages.
*/
static void
-start_standby(char *server_start_log)
+start_standby(StandbyInfo *standby)
{
int rc;
char *pg_ctl_cmd;
- Assert(server_start_log != NULL);
-
- if (server_start_log[0] == '\0')
+ if (standby->server_log == NULL)
{
char timebuf[128];
struct timeval time;
time_t tt;
int len;
+ standby->server_log = (char *) pg_malloc0(MAXPGPATH);
+
/* append timestamp with ISO 8601 format. */
gettimeofday(&time, NULL);
tt = (time_t) time.tv_sec;
@@ -1393,8 +1444,8 @@ start_standby(char *server_start_log)
snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
".%03d", (int) (time.tv_usec / 1000));
- len = snprintf(server_start_log, MAXPGPATH,
- "%s/%s/server_start_%s.log", subscriber_dir,
+ len = snprintf(standby->server_log, MAXPGPATH,
+ "%s/%s/server_start_%s.log", standby->pgdata,
PGS_OUTPUT_DIR, timebuf);
if (len >= MAXPGPATH)
{
@@ -1404,8 +1455,8 @@ start_standby(char *server_start_log)
}
pg_log_info("starting the standby server");
- pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path,
- subscriber_dir, server_start_log);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" start -D \"%s\" -s -l \"%s\"",
+ standby->bindir, standby->pgdata, standby->server_log);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 1);
}
@@ -1415,15 +1466,15 @@ start_standby(char *server_start_log)
* messages.
*/
static void
-stop_standby(void)
+stop_standby(StandbyInfo *standby)
{
int rc;
char *pg_ctl_cmd;
pg_log_info("stopping the standby server");
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path,
- subscriber_dir);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s", standby->bindir,
+ standby->pgdata);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
}
@@ -1454,12 +1505,8 @@ main(int argc, char **argv)
int len;
- char *pub_base_conninfo = NULL;
- char *sub_base_conninfo = NULL;
char *dbname_conninfo = NULL;
- uint64 pub_sysid;
- uint64 sub_sysid;
struct stat statbuf;
PGconn *conn;
@@ -1513,7 +1560,7 @@ main(int argc, char **argv)
switch (c)
{
case 'D':
- subscriber_dir = pg_strdup(optarg);
+ standby.pgdata = pg_strdup(optarg);
break;
case 'P':
pub_conninfo_str = pg_strdup(optarg);
@@ -1526,7 +1573,7 @@ main(int argc, char **argv)
if (!simple_string_list_member(&database_names, optarg))
{
simple_string_list_append(&database_names, optarg);
- num_dbs++;
+ dbarr.ndbs++;
}
break;
case 'n':
@@ -1562,7 +1609,7 @@ main(int argc, char **argv)
/*
* Required arguments
*/
- if (subscriber_dir == NULL)
+ if (standby.pgdata == NULL)
{
pg_log_error("no subscriber data directory specified");
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1585,9 +1632,10 @@ main(int argc, char **argv)
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
exit(1);
}
- pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
- "publisher");
- if (pub_base_conninfo == NULL)
+ primary.base_conninfo = get_base_conninfo(pub_conninfo_str,
+ dbname_conninfo,
+ "publisher");
+ if (primary.base_conninfo == NULL)
exit(1);
if (sub_conninfo_str == NULL)
@@ -1596,8 +1644,12 @@ main(int argc, char **argv)
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
exit(1);
}
- sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
- if (sub_base_conninfo == NULL)
+
+ standby.base_conninfo = get_base_conninfo(sub_conninfo_str, NULL,
+ "subscriber");
+
+
+ if (standby.base_conninfo == NULL)
exit(1);
if (database_names.head == NULL)
@@ -1612,7 +1664,7 @@ main(int argc, char **argv)
if (dbname_conninfo)
{
simple_string_list_append(&database_names, dbname_conninfo);
- num_dbs++;
+ dbarr.ndbs++;
pg_log_info("database \"%s\" was extracted from the publisher connection string",
dbname_conninfo);
@@ -1628,23 +1680,23 @@ main(int argc, char **argv)
/*
* Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
*/
- if (!get_exec_path(argv[0]))
+ if (!get_exec_base_path(argv[0]))
exit(1);
/* rudimentary check for a data directory. */
- if (!check_data_directory(subscriber_dir))
+ if (!check_data_directory(standby.pgdata))
exit(1);
- /* Store database information for publisher and subscriber. */
- dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+ /* Store database information to dbarr */
+ store_db_names(&dbarr.perdb, dbarr.ndbs);
/*
* Check if the subscriber data directory has the same system identifier
* than the publisher data directory.
*/
- pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
- sub_sysid = get_control_from_datadir(subscriber_dir);
- if (pub_sysid != sub_sysid)
+ get_sysid_for_primary(&primary, dbarr.perdb[0].dbname);
+ get_sysid_for_standby(&standby);
+ if (primary.sysid != standby.sysid)
{
pg_log_error("subscriber data directory is not a copy of the source database cluster");
exit(1);
@@ -1654,7 +1706,7 @@ main(int argc, char **argv)
* Create the output directory to store any data generated by this tool.
*/
base_dir = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s", standby.pgdata, PGS_OUTPUT_DIR);
if (len >= MAXPGPATH)
{
pg_log_error("directory path for subscriber is too long");
@@ -1668,7 +1720,7 @@ main(int argc, char **argv)
}
/* subscriber PID file. */
- snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", standby.pgdata);
/*
* The standby server must be running. That's because some checks will be
@@ -1681,7 +1733,7 @@ main(int argc, char **argv)
/*
* Check if the standby server is ready for logical replication.
*/
- if (!setup_subscriber(dbinfo))
+ if (!setup_subscriber(&standby, &dbarr))
exit(1);
/*
@@ -1691,13 +1743,13 @@ main(int argc, char **argv)
* if the primary slot is in use. We could use an extra connection for
* it but it doesn't seem worth.
*/
- if (!setup_publisher(dbinfo))
+ if (!setup_publisher(&primary, &dbarr))
exit(1);
pg_log_info("standby is up and running");
pg_log_info("stopping the server to start the transformation steps");
- stop_standby();
+ stop_standby(&standby);
}
else
{
@@ -1723,11 +1775,11 @@ main(int argc, char **argv)
* replication connection open (depending when base backup was taken, the
* connection should be open for a few hours).
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
if (conn == NULL)
exit(1);
- consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
- temp_replslot);
+ consistent_lsn = create_logical_replication_slot(conn, true, &dbarr.perdb[0]);
+
/*
* Write recovery parameters.
@@ -1753,7 +1805,7 @@ main(int argc, char **argv)
{
appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
consistent_lsn);
- WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ WriteRecoveryConfig(conn, standby.pgdata, recoveryconfcontents);
}
disconnect_database(conn);
@@ -1762,32 +1814,32 @@ main(int argc, char **argv)
/*
* Start subscriber and wait until accepting connections.
*/
- start_standby(server_start_log);
+ start_standby(&standby);
/*
* Waiting the subscriber to be promoted.
*/
- wait_for_end_recovery(dbinfo[0].subconninfo);
+ wait_for_end_recovery(&standby, dbarr.perdb[0].dbname);
/*
* Create a subscription for each database.
*/
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
+
/* Connect to subscriber. */
- conn = connect_database(dbinfo[i].subconninfo);
+ conn = connect_database(standby.base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
- create_subscription(conn, &dbinfo[i]);
+ create_subscription(conn, &standby, &primary, perdb);
/* Set the replication progress to the correct LSN. */
- set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+ set_replication_progress(conn, perdb, consistent_lsn);
/* Enable subscription. */
- enable_subscription(conn, &dbinfo[i]);
-
- disconnect_database(conn);
+ enable_subscription(conn, perdb);
}
/*
@@ -1798,31 +1850,45 @@ main(int argc, char **argv)
* XXX we might not fail here. Instead, we provide a warning so the user
* eventually drops the replication slot later.
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
if (conn == NULL)
{
- if (primary_slot_name != NULL)
- pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
- pg_log_warning("could not drop transient replication slot \"%s\" on publisher", temp_replslot);
+ char transient_replslot[NAMEDATALEN];
+
+ snprintf(transient_replslot, NAMEDATALEN, "pg_subscriber_%d_startpoint",
+ (int) getpid());
+
+ if (standby.primary_slot_name != NULL)
+ pg_log_warning("could not drop replication slot \"%s\" on primary",
+ standby.primary_slot_name);
+ pg_log_warning("could not drop transient replication slot \"%s\" on publisher",
+ transient_replslot);
pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
}
else
{
- drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[0];
+ char *primary_slot_name = standby.primary_slot_name;
+ char transient_replslot[NAMEDATALEN];
+
if (primary_slot_name != NULL)
- drop_replication_slot(conn, &dbinfo[0], primary_slot_name);
+ drop_replication_slot(conn, perdb, primary_slot_name);
+
+ snprintf(transient_replslot, NAMEDATALEN, "pg_subscriber_%d_startpoint",
+ (int) getpid());
+ drop_replication_slot(conn, perdb, transient_replslot);
disconnect_database(conn);
}
/*
* Stop the subscriber.
*/
- stop_standby();
+ stop_standby(&standby);
/*
* Change system identifier.
*/
- modify_sysid(pg_resetwal_path, subscriber_dir);
+ modify_sysid(standby.bindir, standby.pgdata);
/*
* The log file is kept if retain option is specified or this tool does
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 7e866e3c3d..e9e3db9ffc 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1506,6 +1506,8 @@ LogicalRepCommitPreparedTxnData
LogicalRepCtxStruct
LogicalRepMsgType
LogicalRepPartMapEntry
+LogicalRepPerdbInfo
+LogicalRepPerdbInfoArr
LogicalRepPreparedTxnData
LogicalRepRelId
LogicalRepRelMapEntry
@@ -1884,6 +1886,7 @@ PREDICATELOCK
PREDICATELOCKTAG
PREDICATELOCKTARGET
PREDICATELOCKTARGETTAG
+PrimaryInfo
PROCESS_INFORMATION
PROCLOCK
PROCLOCKTAG
@@ -2460,6 +2463,7 @@ SQLValueFunctionOp
SSL
SSLExtensionInfoContext
SSL_CTX
+StandbyInfo
STARTUPINFO
STRLEN
SV
--
2.43.0
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
@ 2024-01-30 02:01 ` Euler Taveira <[email protected]>
2024-01-30 09:26 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
1 sibling, 1 reply; 77+ messages in thread
From: Euler Taveira @ 2024-01-30 02:01 UTC (permalink / raw)
To: [email protected] <[email protected]>; [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Shlok Kyal <[email protected]>
On Sun, Jan 28, 2024, at 10:10 PM, Euler Taveira wrote:
> On Fri, Jan 26, 2024, at 4:55 AM, Hayato Kuroda (Fujitsu) wrote:
>> Again, thanks for updating the patch! There are my random comments for v9.
>
> Thanks for checking v9. I already incorporated some of the points below into
> the next patch. Give me a couple of hours to include some important points.
Here it is another patch that includes the following changes:
* rename the tool to pg_createsubscriber: it was the name with the most votes
[1].
* fix recovery-timeout option [2]
* refactor: split setup_publisher into check_publisher (that contains only GUC
checks) and setup_publisher (that does what the name suggests) [2]
* doc: verbose option can be specified multiple times [2]
* typedefs.list: add LogicalRepInfo [2]
* fix: register cleanup routine after the data structure (dbinfo) is assigned
[2]
* doc: add mandatory options to synopsis [3]
* refactor: rename some options (such as publisher-conninfo to
publisher-server) to use the same pattern as pg_rewind.
* feat: remove publications from subscriber [2]
* feat: use temporary replication slot to get consistent LSN [3]
* refactor: move subscriber setup to its own function
* refactor: stop standby server to its own function [2]
* refactor: start standby server to its own function [2]
* fix: getopt options [4]
There is a few open items in my list. I included v10-0002. I had already
included a refactor to include start/stop functions so I didn't include
v10-0003. I'll check v10-0004 tomorrow.
One open item that is worrying me is how to handle the pg_ctl timeout. This
patch does nothing and the user should use PGCTLTIMEOUT environment variable to
avoid that the execution is canceled after 60 seconds (default for pg_ctl).
Even if you set a high value, it might not be enough for cases like
time-delayed replica. Maybe pg_ctl should accept no timeout as --timeout
option. I'll include this caveat into the documentation but I'm afraid it is
not sufficient and we should provide a better way to handle this situation.
[1] https://www.postgresql.org/message-id/b315c7da-7ab1-4014-a2a9-8ab6ae26017c%40app.fastmail.com
[2] https://www.postgresql.org/message-id/TY3PR01MB98895A551923953B3DA3C7C8F5792%40TY3PR01MB9889.jpnprd0...
[3] https://www.postgresql.org/message-id/[email protected]....
[4] https://www.postgresql.org/message-id/TY3PR01MB98891E7735141FE8760CEC4AF57E2%40TY3PR01MB9889.jpnprd0...
--
Euler Taveira
EDB https://www.enterprisedb.com/
Attachments:
[text/x-patch] v11-0001-Creates-a-new-logical-replica-from-a-standby-ser.patch (76.2K, ../../[email protected]/3-v11-0001-Creates-a-new-logical-replica-from-a-standby-ser.patch)
download | inline diff:
From 67c6e63cf34885bdd687c06e1e071d15f42cdf2a Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email protected]>
Date: Mon, 5 Jun 2023 14:39:40 -0400
Subject: [PATCH v11] Creates a new logical replica from a standby server
A new tool called pg_createsubscriber can convert a physical replica
into a logical replica. It runs on the target server and should be able
to connect to the source server (publisher) and the target server
(subscriber).
The conversion requires a few steps. Check if the target data directory
has the same system identifier than the source data directory. Stop the
target server if it is running as a standby server. Create one
replication slot per specified database on the source server. One
additional replication slot is created at the end to get the consistent
LSN (This consistent LSN will be used as (a) a stopping point for the
recovery process and (b) a starting point for the subscriptions). Write
recovery parameters into the target data directory and start the target
server (Wait until the target server is promoted). Create one
publication (FOR ALL TABLES) per specified database on the source
server. Create one subscription per specified database on the target
server (Use replication slot and publication created in a previous step.
Don't enable the subscriptions yet). Sets the replication progress to
the consistent LSN that was got in a previous step. Enable the
subscription for each specified database on the target server. Stop the
target server. Change the system identifier from the target server.
Depending on your workload and database size, creating a logical replica
couldn't be an option due to resource constraints (WAL backlog should be
available until all table data is synchronized). The initial data copy
and the replication progress tends to be faster on a physical replica.
The purpose of this tool is to speed up a logical replica setup.
---
doc/src/sgml/ref/allfiles.sgml | 1 +
doc/src/sgml/ref/pg_createsubscriber.sgml | 322 +++
doc/src/sgml/reference.sgml | 1 +
src/bin/pg_basebackup/.gitignore | 1 +
src/bin/pg_basebackup/Makefile | 8 +-
src/bin/pg_basebackup/meson.build | 19 +
src/bin/pg_basebackup/pg_createsubscriber.c | 1852 +++++++++++++++++
.../t/040_pg_createsubscriber.pl | 44 +
.../t/041_pg_createsubscriber_standby.pl | 139 ++
src/tools/pgindent/typedefs.list | 1 +
10 files changed, 2387 insertions(+), 1 deletion(-)
create mode 100644 doc/src/sgml/ref/pg_createsubscriber.sgml
create mode 100644 src/bin/pg_basebackup/pg_createsubscriber.c
create mode 100644 src/bin/pg_basebackup/t/040_pg_createsubscriber.pl
create mode 100644 src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index 4a42999b18..a2b5eea0e0 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -214,6 +214,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY pgResetwal SYSTEM "pg_resetwal.sgml">
<!ENTITY pgRestore SYSTEM "pg_restore.sgml">
<!ENTITY pgRewind SYSTEM "pg_rewind.sgml">
+<!ENTITY pgCreateSubscriber SYSTEM "pg_createsubscriber.sgml">
<!ENTITY pgVerifyBackup SYSTEM "pg_verifybackup.sgml">
<!ENTITY pgtestfsync SYSTEM "pgtestfsync.sgml">
<!ENTITY pgtesttiming SYSTEM "pgtesttiming.sgml">
diff --git a/doc/src/sgml/ref/pg_createsubscriber.sgml b/doc/src/sgml/ref/pg_createsubscriber.sgml
new file mode 100644
index 0000000000..1c78ff92e0
--- /dev/null
+++ b/doc/src/sgml/ref/pg_createsubscriber.sgml
@@ -0,0 +1,322 @@
+<!--
+doc/src/sgml/ref/pg_createsubscriber.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="app-pgcreatesubscriber">
+ <indexterm zone="app-pgcreatesubscriber">
+ <primary>pg_createsubscriber</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle><application>pg_createsubscriber</application></refentrytitle>
+ <manvolnum>1</manvolnum>
+ <refmiscinfo>Application</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>pg_createsubscriber</refname>
+ <refpurpose>convert a physical replica into a new logical replica</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_createsubscriber</command>
+ <arg rep="repeat"><replaceable>option</replaceable></arg>
+ <group choice="plain">
+ <group choice="req">
+ <arg choice="plain"><option>-D</option> </arg>
+ <arg choice="plain"><option>--pgdata</option></arg>
+ </group>
+ <replaceable>datadir</replaceable>
+ <group choice="req">
+ <arg choice="plain"><option>-P</option></arg>
+ <arg choice="plain"><option>--publisher-server</option></arg>
+ </group>
+ <replaceable>connstr</replaceable>
+ <group choice="req">
+ <arg choice="plain"><option>-S</option></arg>
+ <arg choice="plain"><option>--subscriber-server</option></arg>
+ </group>
+ <replaceable>connstr</replaceable>
+ <group choice="req">
+ <arg choice="plain"><option>-d</option></arg>
+ <arg choice="plain"><option>--database</option></arg>
+ </group>
+ <replaceable>dbname</replaceable>
+ </group>
+ </cmdsynopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+ <para>
+ <application>pg_createsubscriber</application> takes the publisher and subscriber
+ connection strings, a cluster directory from a physical replica and a list of
+ database names and it sets up a new logical replica using the physical
+ recovery process.
+ </para>
+
+ <para>
+ The <application>pg_createsubscriber</application> should be run at the target
+ server. The source server (known as publisher server) should accept logical
+ replication connections from the target server (known as subscriber server).
+ The target server should accept local logical replication connection.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Options</title>
+
+ <para>
+ <application>pg_createsubscriber</application> accepts the following
+ command-line arguments:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-D <replaceable class="parameter">directory</replaceable></option></term>
+ <term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term>
+ <listitem>
+ <para>
+ The target directory that contains a cluster directory from a physical
+ replica.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-P <replaceable class="parameter">connstr</replaceable></option></term>
+ <term><option>--publisher-server=<replaceable class="parameter">connstr</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the publisher. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-S <replaceable class="parameter">connstr</replaceable></option></term>
+ <term><option>--subscriber-server=<replaceable class="parameter">connstr</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the subscriber. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-d <replaceable class="parameter">dbname</replaceable></option></term>
+ <term><option>--database=<replaceable class="parameter">dbname</replaceable></option></term>
+ <listitem>
+ <para>
+ The database name to create the subscription. Multiple databases can be
+ selected by writing multiple <option>-d</option> switches.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
+ <listitem>
+ <para>
+ Do everything except actually modifying the target directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-r</option></term>
+ <term><option>--retain</option></term>
+ <listitem>
+ <para>
+ Retain log file even after successful completion.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-t <replaceable class="parameter">seconds</replaceable></option></term>
+ <term><option>--recovery-timeout=<replaceable class="parameter">seconds</replaceable></option></term>
+ <listitem>
+ <para>
+ The maximum number of seconds to wait for recovery to end. Setting to 0
+ disables. The default is 0.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-v</option></term>
+ <term><option>--verbose</option></term>
+ <listitem>
+ <para>
+ Enables verbose mode. This will cause
+ <application>pg_createsubscriber</application> to output progress messages
+ and detailed information about each step to standard error.
+ Repeating the option causes additional debug-level messages to appear on
+ standard error.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ Other options are also available:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-V</option></term>
+ <term><option>--version</option></term>
+ <listitem>
+ <para>
+ Print the <application>pg_createsubscriber</application> version and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-?</option></term>
+ <term><option>--help</option></term>
+ <listitem>
+ <para>
+ Show help about <application>pg_createsubscriber</application> command
+ line arguments, and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Notes</title>
+
+ <para>
+ The transformation proceeds in the following steps:
+ </para>
+
+ <procedure>
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> checks if the given target data
+ directory has the same system identifier than the source data directory.
+ Since it uses the recovery process as one of the steps, it starts the
+ target server as a replica from the source server. If the system
+ identifier is not the same, <application>pg_createsubscriber</application> will
+ terminate with an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> checks if the target data
+ directory is used by a physical replica. Stop the physical replica if it is
+ running. One of the next steps is to add some recovery parameters that
+ requires a server start. This step avoids an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> creates one replication slot for
+ each specified database on the source server. The replication slot name
+ contains a <literal>pg_createsubscriber</literal> prefix. These replication
+ slots will be used by the subscriptions in a future step. A temporary
+ replication slot is used to get a consistent start location. This
+ consistent LSN will be used as a stopping point in the <xref
+ linkend="guc-recovery-target-lsn"/> parameter and by the
+ subscriptions as a replication starting point. It guarantees that no
+ transaction will be lost.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> writes recovery parameters into
+ the target data directory and start the target server. It specifies a LSN
+ (consistent LSN that was obtained in the previous step) of write-ahead
+ log location up to which recovery will proceed. It also specifies
+ <literal>promote</literal> as the action that the server should take once
+ the recovery target is reached. This step finishes once the server ends
+ standby mode and is accepting read-write operations.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Next, <application>pg_createsubscriber</application> creates one publication
+ for each specified database on the source server. Each publication
+ replicates changes for all tables in the database. The publication name
+ contains a <literal>pg_createsubscriber</literal> prefix. These publication
+ will be used by a corresponding subscription in a next step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> creates one subscription for
+ each specified database on the target server. Each subscription name
+ contains a <literal>pg_createsubscriber</literal> prefix. The replication slot
+ name is identical to the subscription name. It does not copy existing data
+ from the source server. It does not create a replication slot. Instead, it
+ uses the replication slot that was created in a previous step. The
+ subscription is created but it is not enabled yet. The reason is the
+ replication progress must be set to the consistent LSN but replication
+ origin name contains the subscription oid in its name. Hence, the
+ subscription will be enabled in a separate step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> sets the replication progress to
+ the consistent LSN that was obtained in a previous step. When the target
+ server started the recovery process, it caught up to the consistent LSN.
+ This is the exact LSN to be used as a initial location for each
+ subscription.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Finally, <application>pg_createsubscriber</application> enables the subscription
+ for each specified database on the target server. The subscription starts
+ streaming from the consistent LSN.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> stops the target server to change
+ its system identifier.
+ </para>
+ </step>
+ </procedure>
+ </refsect1>
+
+ <refsect1>
+ <title>Examples</title>
+
+ <para>
+ To create a logical replica for databases <literal>hr</literal> and
+ <literal>finance</literal> from a physical replica at <literal>foo</literal>:
+<screen>
+<prompt>$</prompt> <userinput>pg_createsubscriber -D /usr/local/pgsql/data -P "host=foo" -S "host=localhost" -d hr -d finance</userinput>
+</screen>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="app-pgbasebackup"/></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index aa94f6adf6..c5edd244ef 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -285,6 +285,7 @@
&pgCtl;
&pgResetwal;
&pgRewind;
+ &pgCreateSubscriber;
&pgtestfsync;
&pgtesttiming;
&pgupgrade;
diff --git a/src/bin/pg_basebackup/.gitignore b/src/bin/pg_basebackup/.gitignore
index 26048bdbd8..b3a6f5a2fe 100644
--- a/src/bin/pg_basebackup/.gitignore
+++ b/src/bin/pg_basebackup/.gitignore
@@ -1,5 +1,6 @@
/pg_basebackup
/pg_receivewal
/pg_recvlogical
+/pg_createsubscriber
/tmp_check/
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index abfb6440ec..ded434b683 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -44,7 +44,7 @@ BBOBJS = \
bbstreamer_tar.o \
bbstreamer_zstd.o
-all: pg_basebackup pg_receivewal pg_recvlogical
+all: pg_basebackup pg_receivewal pg_recvlogical pg_createsubscriber
pg_basebackup: $(BBOBJS) $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) $(BBOBJS) $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
@@ -55,10 +55,14 @@ pg_receivewal: pg_receivewal.o $(OBJS) | submake-libpq submake-libpgport submake
pg_recvlogical: pg_recvlogical.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_recvlogical.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_createsubscriber: $(WIN32RES) pg_createsubscriber.o | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
install: all installdirs
$(INSTALL_PROGRAM) pg_basebackup$(X) '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
$(INSTALL_PROGRAM) pg_receivewal$(X) '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
$(INSTALL_PROGRAM) pg_recvlogical$(X) '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ $(INSTALL_PROGRAM) pg_createsubscriber$(X) '$(DESTDIR)$(bindir)/pg_createsubscriber$(X)'
installdirs:
$(MKDIR_P) '$(DESTDIR)$(bindir)'
@@ -67,10 +71,12 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ rm -f '$(DESTDIR)$(bindir)/pg_createsubscriber$(X)'
clean distclean:
rm -f pg_basebackup$(X) pg_receivewal$(X) pg_recvlogical$(X) \
$(BBOBJS) pg_receivewal.o pg_recvlogical.o \
+ pg_createsubscriber$(X) pg_createsubscriber.o \
$(OBJS)
rm -rf tmp_check
diff --git a/src/bin/pg_basebackup/meson.build b/src/bin/pg_basebackup/meson.build
index f7e60e6670..345a2d6fcd 100644
--- a/src/bin/pg_basebackup/meson.build
+++ b/src/bin/pg_basebackup/meson.build
@@ -75,6 +75,23 @@ pg_recvlogical = executable('pg_recvlogical',
)
bin_targets += pg_recvlogical
+pg_createsubscriber_sources = files(
+ 'pg_createsubscriber.c'
+)
+
+if host_system == 'windows'
+ pg_createsubscriber_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'pg_createsubscriber',
+ '--FILEDESC', 'pg_createsubscriber - create a new logical replica from a standby server',])
+endif
+
+pg_createsubscriber = executable('pg_createsubscriber',
+ pg_createsubscriber_sources,
+ dependencies: [frontend_code, libpq],
+ kwargs: default_bin_args,
+)
+bin_targets += pg_createsubscriber
+
tests += {
'name': 'pg_basebackup',
'sd': meson.current_source_dir(),
@@ -89,6 +106,8 @@ tests += {
't/011_in_place_tablespace.pl',
't/020_pg_receivewal.pl',
't/030_pg_recvlogical.pl',
+ 't/040_pg_createsubscriber.pl',
+ 't/041_pg_createsubscriber_standby.pl',
],
},
}
diff --git a/src/bin/pg_basebackup/pg_createsubscriber.c b/src/bin/pg_basebackup/pg_createsubscriber.c
new file mode 100644
index 0000000000..478560b3e4
--- /dev/null
+++ b/src/bin/pg_basebackup/pg_createsubscriber.c
@@ -0,0 +1,1852 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_createsubscriber.c
+ * Create a new logical replica from a standby server
+ *
+ * Copyright (C) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_basebackup/pg_createsubscriber.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include <signal.h>
+#include <sys/stat.h>
+#include <sys/time.h>
+#include <sys/wait.h>
+#include <time.h>
+
+#include "access/xlogdefs.h"
+#include "catalog/pg_control.h"
+#include "common/connect.h"
+#include "common/controldata_utils.h"
+#include "common/file_perm.h"
+#include "common/file_utils.h"
+#include "common/logging.h"
+#include "fe_utils/recovery_gen.h"
+#include "fe_utils/simple_list.h"
+#include "getopt_long.h"
+#include "utils/pidfile.h"
+
+#define PGS_OUTPUT_DIR "pg_createsubscriber_output.d"
+
+typedef struct LogicalRepInfo
+{
+ Oid oid; /* database OID */
+ char *dbname; /* database name */
+ char *pubconninfo; /* publication connection string for logical
+ * replication */
+ char *subconninfo; /* subscription connection string for logical
+ * replication */
+ char *pubname; /* publication name */
+ char *subname; /* subscription name (also replication slot
+ * name) */
+
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepInfo;
+
+static void cleanup_objects_atexit(void);
+static void usage();
+static char *get_base_conninfo(char *conninfo, char *dbname,
+ const char *noderole);
+static bool get_exec_path(const char *path);
+static bool check_data_directory(const char *datadir);
+static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
+static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
+static PGconn *connect_database(const char *conninfo);
+static void disconnect_database(PGconn *conn);
+static uint64 get_sysid_from_conn(const char *conninfo);
+static uint64 get_control_from_datadir(const char *datadir);
+static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
+static bool check_publisher(LogicalRepInfo *dbinfo);
+static bool setup_publisher(LogicalRepInfo *dbinfo);
+static bool check_subscriber(LogicalRepInfo *dbinfo);
+static bool setup_subscriber(LogicalRepInfo *dbinfo, const char *consistent_lsn);
+static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name);
+static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static char *server_logfile_name(const char *datadir);
+static void start_standby_server(const char *pg_ctl_path, const char *datadir, const char *logfile);
+static void stop_standby_server(const char *pg_ctl_path, const char *datadir);
+static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
+static void wait_for_end_recovery(const char *conninfo);
+static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+
+#define USEC_PER_SEC 1000000
+#define WAIT_INTERVAL 1 /* 1 second */
+
+/* Options */
+static const char *progname;
+
+static char *subscriber_dir = NULL;
+static char *pub_conninfo_str = NULL;
+static char *sub_conninfo_str = NULL;
+static SimpleStringList database_names = {NULL, NULL};
+static char *primary_slot_name = NULL;
+static bool dry_run = false;
+static bool retain = false;
+static int recovery_timeout = 0;
+
+static bool success = false;
+
+static char *pg_ctl_path = NULL;
+static char *pg_resetwal_path = NULL;
+
+static LogicalRepInfo *dbinfo;
+static int num_dbs = 0;
+
+enum WaitPMResult
+{
+ POSTMASTER_READY,
+ POSTMASTER_STANDBY,
+ POSTMASTER_STILL_STARTING,
+ POSTMASTER_FAILED
+};
+
+
+/*
+ * Cleanup objects that were created by pg_createsubscriber if there is an error.
+ *
+ * Replication slots, publications and subscriptions are created. Depending on
+ * the step it failed, it should remove the already created objects if it is
+ * possible (sometimes it won't work due to a connection issue).
+ */
+static void
+cleanup_objects_atexit(void)
+{
+ PGconn *conn;
+ int i;
+
+ if (success)
+ return;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ if (dbinfo[i].made_subscription)
+ {
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn != NULL)
+ {
+ drop_subscription(conn, &dbinfo[i]);
+ if (dbinfo[i].made_publication)
+ drop_publication(conn, &dbinfo[i]);
+ disconnect_database(conn);
+ }
+ }
+
+ if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ {
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn != NULL)
+ {
+ if (dbinfo[i].made_publication)
+ drop_publication(conn, &dbinfo[i]);
+ if (dbinfo[i].made_replslot)
+ drop_replication_slot(conn, &dbinfo[i], NULL);
+ disconnect_database(conn);
+ }
+ }
+ }
+}
+
+static void
+usage(void)
+{
+ printf(_("%s creates a new logical replica from a standby server.\n\n"),
+ progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION]...\n"), progname);
+ printf(_("\nOptions:\n"));
+ printf(_(" -D, --pgdata=DATADIR location for the subscriber data directory\n"));
+ printf(_(" -P, --publisher-server=CONNSTR publisher connection string\n"));
+ printf(_(" -S, --subscriber-server=CONNSTR subscriber connection string\n"));
+ printf(_(" -d, --database=DBNAME database to create a subscription\n"));
+ printf(_(" -n, --dry-run stop before modifying anything\n"));
+ printf(_(" -t, --recovery-timeout=SECS seconds to wait for recovery to end\n"));
+ printf(_(" -r, --retain retain log file after success\n"));
+ printf(_(" -v, --verbose output verbose messages\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
+ printf(_("\nReport bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+ printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
+
+/*
+ * Validate a connection string. Returns a base connection string that is a
+ * connection string without a database name.
+ * Since we might process multiple databases, each database name will be
+ * appended to this base connection string to provide a final connection string.
+ * If the second argument (dbname) is not null, returns dbname if the provided
+ * connection string contains it. If option --database is not provided, uses
+ * dbname as the only database to setup the logical replica.
+ * It is the caller's responsibility to free the returned connection string and
+ * dbname.
+ */
+static char *
+get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ PQconninfoOption *conn_opts = NULL;
+ PQconninfoOption *conn_opt;
+ char *errmsg = NULL;
+ char *ret;
+ int i;
+
+ pg_log_info("validating connection string on %s", noderole);
+
+ conn_opts = PQconninfoParse(conninfo, &errmsg);
+ if (conn_opts == NULL)
+ {
+ pg_log_error("could not parse connection string: %s", errmsg);
+ return NULL;
+ }
+
+ i = 0;
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (strcmp(conn_opt->keyword, "dbname") == 0 && conn_opt->val != NULL)
+ {
+ if (dbname)
+ dbname = pg_strdup(conn_opt->val);
+ continue;
+ }
+
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0')
+ {
+ if (i > 0)
+ appendPQExpBufferChar(buf, ' ');
+ appendPQExpBuffer(buf, "%s=%s", conn_opt->keyword, conn_opt->val);
+ i++;
+ }
+ }
+
+ ret = pg_strdup(buf->data);
+
+ destroyPQExpBuffer(buf);
+ PQconninfoFree(conn_opts);
+
+ return ret;
+}
+
+/*
+ * Get the absolute path from other PostgreSQL binaries (pg_ctl and
+ * pg_resetwal) that is used by it.
+ */
+static bool
+get_exec_path(const char *path)
+{
+ int rc;
+
+ pg_ctl_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_ctl",
+ "pg_ctl (PostgreSQL) " PG_VERSION "\n",
+ pg_ctl_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_ctl", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_ctl", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
+
+ pg_resetwal_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_resetwal",
+ "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
+ pg_resetwal_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_resetwal", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_resetwal", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+
+ return true;
+}
+
+/*
+ * Is it a cluster directory? These are preliminary checks. It is far from
+ * making an accurate check. If it is not a clone from the publisher, it will
+ * eventually fail in a future step.
+ */
+static bool
+check_data_directory(const char *datadir)
+{
+ struct stat statbuf;
+ char versionfile[MAXPGPATH];
+
+ pg_log_info("checking if directory \"%s\" is a cluster data directory",
+ datadir);
+
+ if (stat(datadir, &statbuf) != 0)
+ {
+ if (errno == ENOENT)
+ pg_log_error("data directory \"%s\" does not exist", datadir);
+ else
+ pg_log_error("could not access directory \"%s\": %s", datadir, strerror(errno));
+
+ return false;
+ }
+
+ snprintf(versionfile, MAXPGPATH, "%s/PG_VERSION", datadir);
+ if (stat(versionfile, &statbuf) != 0 && errno == ENOENT)
+ {
+ pg_log_error("directory \"%s\" is not a database cluster directory", datadir);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Append database name into a base connection string.
+ *
+ * dbname is the only parameter that changes so it is not included in the base
+ * connection string. This function concatenates dbname to build a "real"
+ * connection string.
+ */
+static char *
+concat_conninfo_dbname(const char *conninfo, const char *dbname)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *ret;
+
+ Assert(conninfo != NULL);
+
+ appendPQExpBufferStr(buf, conninfo);
+ appendPQExpBuffer(buf, " dbname=%s", dbname);
+
+ ret = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+
+ return ret;
+}
+
+/*
+ * Store publication and subscription information.
+ */
+static LogicalRepInfo *
+store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+{
+ LogicalRepInfo *dbinfo;
+ SimpleStringListCell *cell;
+ int i = 0;
+
+ dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+
+ for (cell = database_names.head; cell; cell = cell->next)
+ {
+ char *conninfo;
+
+ /* Publisher. */
+ conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
+ dbinfo[i].pubconninfo = conninfo;
+ dbinfo[i].dbname = cell->val;
+ dbinfo[i].made_replslot = false;
+ dbinfo[i].made_publication = false;
+ dbinfo[i].made_subscription = false;
+ /* other struct fields will be filled later. */
+
+ /* Subscriber. */
+ conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
+ dbinfo[i].subconninfo = conninfo;
+
+ i++;
+ }
+
+ return dbinfo;
+}
+
+static PGconn *
+connect_database(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ const char *rconninfo;
+
+ /* logical replication mode */
+ rconninfo = psprintf("%s replication=database", conninfo);
+
+ conn = PQconnectdb(rconninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ pg_log_error("connection to database failed: %s", PQerrorMessage(conn));
+ return NULL;
+ }
+
+ /* secure search_path */
+ res = PQexec(conn, ALWAYS_SECURE_SEARCH_PATH_SQL);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not clear search_path: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+ PQclear(res);
+
+ return conn;
+}
+
+static void
+disconnect_database(PGconn *conn)
+{
+ Assert(conn != NULL);
+
+ PQfinish(conn);
+}
+
+/*
+ * Obtain the system identifier using the provided connection. It will be used
+ * to compare if a data directory is a clone of another one.
+ */
+static uint64
+get_sysid_from_conn(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from publisher");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "IDENTIFY_SYSTEM");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not send replication command \"%s\": %s",
+ "IDENTIFY_SYSTEM", PQresultErrorMessage(res));
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+ if (PQntuples(res) != 1 || PQnfields(res) < 3)
+ {
+ pg_log_error("could not identify system: got %d rows and %d fields, expected %d rows and %d or more fields",
+ PQntuples(res), PQnfields(res), 1, 3);
+
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+
+ sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+
+ pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+
+ disconnect_database(conn);
+
+ return sysid;
+}
+
+/*
+ * Obtain the system identifier from control file. It will be used to compare
+ * if a data directory is a clone of another one. This routine is used locally
+ * and avoids a replication connection.
+ */
+static uint64
+get_control_from_datadir(const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ sysid = cf->system_identifier;
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+
+ pfree(cf);
+
+ return sysid;
+}
+
+/*
+ * Modify the system identifier. Since a standby server preserves the system
+ * identifier, it makes sense to change it to avoid situations in which WAL
+ * files from one of the systems might be used in the other one.
+ */
+static void
+modify_sysid(const char *pg_resetwal_path, const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ struct timeval tv;
+
+ char *cmd_str;
+ int rc;
+
+ pg_log_info("modifying system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ /*
+ * Select a new system identifier.
+ *
+ * XXX this code was extracted from BootStrapXLOG().
+ */
+ gettimeofday(&tv, NULL);
+ cf->system_identifier = ((uint64) tv.tv_sec) << 32;
+ cf->system_identifier |= ((uint64) tv.tv_usec) << 12;
+ cf->system_identifier |= getpid() & 0xFFF;
+
+ if (!dry_run)
+ update_controlfile(datadir, cf, true);
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) cf->system_identifier);
+
+ pg_log_info("running pg_resetwal on the subscriber");
+
+ cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+
+ pg_log_debug("command is: %s", cmd_str);
+
+ if (!dry_run)
+ {
+ rc = system(cmd_str);
+ if (rc == 0)
+ pg_log_info("subscriber successfully changed the system identifier");
+ else
+ pg_log_error("subscriber failed to change system identifier: exit code: %d", rc);
+ }
+
+ pfree(cf);
+}
+
+/*
+ * Create the publications and replication slots in preparation for logical
+ * replication.
+ */
+static bool
+setup_publisher(LogicalRepInfo *dbinfo)
+{
+ PGconn *conn;
+ PGresult *res;
+
+ for (int i = 0; i < num_dbs; i++)
+ {
+ char pubname[NAMEDATALEN];
+ char replslotname[NAMEDATALEN];
+
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT oid FROM pg_catalog.pg_database WHERE datname = current_database()");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain database OID: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain database OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ return false;
+ }
+
+ /* Remember database OID. */
+ dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+
+ PQclear(res);
+
+ /*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 31 characters (20 + 10 +
+ * '\0').
+ */
+ snprintf(pubname, sizeof(pubname), "pg_createsubscriber_%u", dbinfo[i].oid);
+ dbinfo[i].pubname = pg_strdup(pubname);
+
+ /*
+ * Create publication on publisher. This step should be executed
+ * *before* promoting the subscriber to avoid any transactions between
+ * consistent LSN and the new publication rows (such transactions
+ * wouldn't see the new publication rows resulting in an error).
+ */
+ create_publication(conn, &dbinfo[i]);
+
+ /*
+ * Build the replication slot name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 42
+ * characters (20 + 10 + 1 + 10 + '\0'). PID is included to reduce the
+ * probability of collision. By default, subscription name is used as
+ * replication slot name.
+ */
+ snprintf(replslotname, sizeof(replslotname),
+ "pg_createsubscriber_%u_%d",
+ dbinfo[i].oid,
+ (int) getpid());
+ dbinfo[i].subname = pg_strdup(replslotname);
+
+ /* Create replication slot on publisher. */
+ if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
+ pg_log_info("create replication slot \"%s\" on publisher", replslotname);
+ else
+ return false;
+
+ disconnect_database(conn);
+ }
+
+ return true;
+}
+
+/*
+ * Is the primary server ready for logical replication?
+ */
+static bool
+check_publisher(LogicalRepInfo *dbinfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ PQExpBuffer str = createPQExpBuffer();
+
+ char *wal_level;
+ int max_repslots;
+ int cur_repslots;
+ int max_walsenders;
+ int cur_walsenders;
+
+ pg_log_info("checking settings on publisher");
+
+ /*
+ * Logical replication requires a few parameters to be set on publisher.
+ * Since these parameters are not a requirement for physical replication,
+ * we should check it to make sure it won't fail.
+ *
+ * wal_level = logical
+ * max_replication_slots >= current + number of dbs to be converted
+ * max_wal_senders >= current + number of dbs to be converted
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "WITH wl AS (SELECT setting AS wallevel FROM pg_settings WHERE name = 'wal_level'),"
+ " total_mrs AS (SELECT setting AS tmrs FROM pg_settings WHERE name = 'max_replication_slots'),"
+ " cur_mrs AS (SELECT count(*) AS cmrs FROM pg_replication_slots),"
+ " total_mws AS (SELECT setting AS tmws FROM pg_settings WHERE name = 'max_wal_senders'),"
+ " cur_mws AS (SELECT count(*) AS cmws FROM pg_stat_activity WHERE backend_type = 'walsender')"
+ "SELECT wallevel, tmrs, cmrs, tmws, cmws FROM wl, total_mrs, cur_mrs, total_mws, cur_mws");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publisher settings: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ wal_level = strdup(PQgetvalue(res, 0, 0));
+ max_repslots = atoi(PQgetvalue(res, 0, 1));
+ cur_repslots = atoi(PQgetvalue(res, 0, 2));
+ max_walsenders = atoi(PQgetvalue(res, 0, 3));
+ cur_walsenders = atoi(PQgetvalue(res, 0, 4));
+
+ PQclear(res);
+
+ pg_log_debug("subscriber: wal_level: %s", wal_level);
+ pg_log_debug("subscriber: max_replication_slots: %d", max_repslots);
+ pg_log_debug("subscriber: current replication slots: %d", cur_repslots);
+ pg_log_debug("subscriber: max_wal_senders: %d", max_walsenders);
+ pg_log_debug("subscriber: current wal senders: %d", cur_walsenders);
+
+ /*
+ * If standby sets primary_slot_name, check if this replication slot is in
+ * use on primary for WAL retention purposes. This replication slot has no
+ * use after the transformation, hence, it will be removed at the end of
+ * this process.
+ */
+ if (primary_slot_name)
+ {
+ appendPQExpBuffer(str,
+ "SELECT 1 FROM pg_replication_slots WHERE active AND slot_name = '%s'", primary_slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain replication slot information: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
+ PQntuples(res), 1);
+ pg_free(primary_slot_name); /* it is not being used. */
+ primary_slot_name = NULL;
+ return false;
+ }
+ else
+ {
+ pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+ }
+
+ PQclear(res);
+ }
+
+ disconnect_database(conn);
+
+ if (strcmp(wal_level, "logical") != 0)
+ {
+ pg_log_error("publisher requires wal_level >= logical");
+ return false;
+ }
+
+ if (max_repslots - cur_repslots < num_dbs)
+ {
+ pg_log_error("publisher requires %d replication slots, but only %d remain", num_dbs, max_repslots - cur_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", cur_repslots + num_dbs);
+ return false;
+ }
+
+ if (max_walsenders - cur_walsenders < num_dbs)
+ {
+ pg_log_error("publisher requires %d wal sender processes, but only %d remain", num_dbs, max_walsenders - cur_walsenders);
+ pg_log_error_hint("Consider increasing max_wal_senders to at least %d.", cur_walsenders + num_dbs);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Is the standby server ready for logical replication?
+ */
+static bool
+check_subscriber(LogicalRepInfo *dbinfo)
+{
+ PGconn *conn;
+ PGresult *res;
+
+ int max_lrworkers;
+ int max_repslots;
+ int max_wprocs;
+
+ pg_log_info("checking settings on subscriber");
+
+ /*
+ * Logical replication requires a few parameters to be set on subscriber.
+ * Since these parameters are not a requirement for physical replication,
+ * we should check it to make sure it won't fail.
+ *
+ * max_replication_slots >= number of dbs to be converted
+ * max_logical_replication_workers >= number of dbs to be converted
+ * max_worker_processes >= 1 + number of dbs to be converted
+ */
+ conn = connect_database(dbinfo[0].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT setting FROM pg_settings WHERE name IN ('max_logical_replication_workers', 'max_replication_slots', 'max_worker_processes', 'primary_slot_name') ORDER BY name");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscriber settings: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ max_lrworkers = atoi(PQgetvalue(res, 0, 0));
+ max_repslots = atoi(PQgetvalue(res, 1, 0));
+ max_wprocs = atoi(PQgetvalue(res, 2, 0));
+ if (strcmp(PQgetvalue(res, 3, 0), "") != 0)
+ primary_slot_name = pg_strdup(PQgetvalue(res, 3, 0));
+
+ pg_log_debug("subscriber: max_logical_replication_workers: %d", max_lrworkers);
+ pg_log_debug("subscriber: max_replication_slots: %d", max_repslots);
+ pg_log_debug("subscriber: max_worker_processes: %d", max_wprocs);
+ pg_log_debug("subscriber: primary_slot_name: %s", primary_slot_name);
+
+ PQclear(res);
+
+ disconnect_database(conn);
+
+ if (max_repslots < num_dbs)
+ {
+ pg_log_error("subscriber requires %d replication slots, but only %d remain", num_dbs, max_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", num_dbs);
+ return false;
+ }
+
+ if (max_lrworkers < num_dbs)
+ {
+ pg_log_error("subscriber requires %d logical replication workers, but only %d remain", num_dbs, max_lrworkers);
+ pg_log_error_hint("Consider increasing max_logical_replication_workers to at least %d.", num_dbs);
+ return false;
+ }
+
+ if (max_wprocs < num_dbs + 1)
+ {
+ pg_log_error("subscriber requires %d worker processes, but only %d remain", num_dbs + 1, max_wprocs);
+ pg_log_error_hint("Consider increasing max_worker_processes to at least %d.", num_dbs + 1);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Create the subscriptions, adjust the initial location for logical replication and
+ * enable the subscriptions. That's the last step for logical repliation setup.
+ */
+static bool
+setup_subscriber(LogicalRepInfo *dbinfo, const char *consistent_lsn)
+{
+ PGconn *conn;
+
+ for (int i = 0; i < num_dbs; i++)
+ {
+ /* Connect to subscriber. */
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ /*
+ * Since the publication was created before the consistent LSN, it is
+ * available on the subscriber when the physical replica is promoted.
+ * Remove publications from the subscriber because it has no use.
+ */
+ drop_publication(conn, &dbinfo[i]);
+
+ create_subscription(conn, &dbinfo[i]);
+
+ /* Set the replication progress to the correct LSN. */
+ set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+
+ /* Enable subscription. */
+ enable_subscription(conn, &dbinfo[i]);
+
+ disconnect_database(conn);
+ }
+
+ return true;
+}
+
+/*
+ * Create a logical replication slot and returns a consistent LSN. The returned
+ * LSN might be used to catch up the subscriber up to the required point.
+ *
+ * CreateReplicationSlot() is not used because it does not provide the one-row
+ * result set that contains the consistent LSN.
+ */
+static char *
+create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res = NULL;
+ char *lsn = NULL;
+ bool transient_replslot = false;
+
+ Assert(conn != NULL);
+
+ /*
+ * If no slot name is informed, it is a transient replication slot used
+ * only for catch up purposes.
+ */
+ if (slot_name[0] == '\0')
+ {
+ snprintf(slot_name, NAMEDATALEN, "pg_createsubscriber_%d_startpoint",
+ (int) getpid());
+ transient_replslot = true;
+ }
+
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
+ if (transient_replslot)
+ appendPQExpBufferStr(str, " TEMPORARY");
+ appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQresultErrorMessage(res));
+ return lsn;
+ }
+ }
+
+ /* for cleanup purposes */
+ if (!transient_replslot)
+ dbinfo->made_replslot = true;
+
+ if (!dry_run)
+ {
+ lsn = pg_strdup(PQgetvalue(res, 0, 1));
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+
+ return lsn;
+}
+
+static void
+drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+static char *
+server_logfile_name(const char *datadir)
+{
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+ char *filename;
+
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ filename = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(filename, MAXPGPATH, "%s/%s/server_start_%s.log", datadir, PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+
+ return filename;
+}
+
+static void
+start_standby_server(const char *pg_ctl_path, const char *datadir, const char *logfile)
+{
+ char *pg_ctl_cmd;
+ int rc;
+
+ pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, datadir, logfile);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+}
+
+static void
+stop_standby_server(const char *pg_ctl_path, const char *datadir)
+{
+ char *pg_ctl_cmd;
+ int rc;
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, datadir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+}
+
+/*
+ * Reports a suitable message if pg_ctl fails.
+ */
+static void
+pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
+{
+ if (rc != 0)
+ {
+ if (WIFEXITED(rc))
+ {
+ pg_log_error("pg_ctl failed with exit code %d", WEXITSTATUS(rc));
+ }
+ else if (WIFSIGNALED(rc))
+ {
+#if defined(WIN32)
+ pg_log_error("pg_ctl was terminated by exception 0x%X", WTERMSIG(rc));
+ pg_log_error_detail("See C include file \"ntstatus.h\" for a description of the hexadecimal value.");
+#else
+ pg_log_error("pg_ctl was terminated by signal %d: %s",
+ WTERMSIG(rc), pg_strsignal(WTERMSIG(rc)));
+#endif
+ }
+ else
+ {
+ pg_log_error("pg_ctl exited with unrecognized status %d", rc);
+ }
+
+ pg_log_error_detail("The failed command was: %s", pg_ctl_cmd);
+ exit(1);
+ }
+
+ if (action)
+ pg_log_info("postmaster was started");
+ else
+ pg_log_info("postmaster was stopped");
+}
+
+/*
+ * Returns after the server finishes the recovery process.
+ *
+ * If recovery_timeout option is set, terminate abnormally without finishing
+ * the recovery process. By default, it waits forever.
+ */
+static void
+wait_for_end_recovery(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ int status = POSTMASTER_STILL_STARTING;
+ int timer = 0;
+
+ pg_log_info("waiting the postmaster to reach the consistent state");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ for (;;)
+ {
+ bool in_recovery;
+
+ res = PQexec(conn, "SELECT pg_catalog.pg_is_in_recovery()");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain recovery progress");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("unexpected result from pg_is_in_recovery function");
+ exit(1);
+ }
+
+ in_recovery = (strcmp(PQgetvalue(res, 0, 0), "t") == 0);
+
+ PQclear(res);
+
+ /*
+ * Does the recovery process finish? In dry run mode, there is no
+ * recovery mode. Bail out as the recovery process has ended.
+ */
+ if (!in_recovery || dry_run)
+ {
+ status = POSTMASTER_READY;
+ break;
+ }
+
+ /*
+ * Bail out after recovery_timeout seconds if this option is set.
+ */
+ if (recovery_timeout > 0 && timer >= recovery_timeout)
+ {
+ pg_log_error("recovery timed out");
+ stop_standby_server(pg_ctl_path, subscriber_dir);
+ exit(1);
+ }
+
+ /* Keep waiting. */
+ pg_usleep(WAIT_INTERVAL * USEC_PER_SEC);
+
+ timer += WAIT_INTERVAL;
+ }
+
+ disconnect_database(conn);
+
+ if (status == POSTMASTER_STILL_STARTING)
+ {
+ pg_log_error("server did not end recovery");
+ exit(1);
+ }
+
+ pg_log_info("postmaster reached the consistent state");
+}
+
+/*
+ * Create a publication that includes all tables in the database.
+ */
+static void
+create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ /* Check if the publication needs to be created. */
+ appendPQExpBuffer(str,
+ "SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
+ dbinfo->pubname);
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publication information: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) == 1)
+ {
+ /*
+ * If publication name already exists and puballtables is true, let's
+ * use it. A previous run of pg_createsubscriber must have created
+ * this publication. Bail out.
+ */
+ if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
+ {
+ pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ return;
+ }
+ else
+ {
+ /*
+ * Unfortunately, if it reaches this code path, it will always
+ * fail (unless you decide to change the existing publication
+ * name). That's bad but it is very unlikely that the user will
+ * choose a name with pg_createsubscriber_ prefix followed by the
+ * exact database oid in which puballtables is false.
+ */
+ pg_log_error("publication \"%s\" does not replicate changes for all tables",
+ dbinfo->pubname);
+ pg_log_error_hint("Consider renaming this publication.");
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(str);
+
+ pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
+ dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_publication = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove publication if it couldn't finish all steps.
+ */
+static void
+drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Create a subscription with some predefined options.
+ *
+ * A replication slot was already created in a previous step. Let's use it. By
+ * default, the subscription name is used as replication slot name. It is
+ * not required to copy data. The subscription will be created but it will not
+ * be enabled now. That's because the replication progress must be set and the
+ * replication origin name (one of the function arguments) contains the
+ * subscription OID in its name. Once the subscription is created,
+ * set_replication_progress() can obtain the chosen origin name and set up its
+ * initial location.
+ */
+static void
+create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str,
+ "CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
+ "WITH (create_slot = false, copy_data = false, enabled = false)",
+ dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
+ dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_subscription = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove subscription if it couldn't finish all steps.
+ */
+static void
+drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Sets the replication progress to the consistent LSN.
+ *
+ * The subscriber caught up to the consistent LSN provided by the temporary
+ * replication slot. The goal is to set up the initial location for the logical
+ * replication that is the exact LSN that the subscriber was promoted. Once the
+ * subscription is enabled it will start streaming from that location onwards.
+ * In dry run mode, the subscription OID and LSN are set to invalid values for
+ * printing purposes.
+ */
+static void
+set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+ Oid suboid;
+ char originname[NAMEDATALEN];
+ char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+
+ Assert(conn != NULL);
+
+ appendPQExpBuffer(str,
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscription OID: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1 && !dry_run)
+ {
+ pg_log_error("could not obtain subscription OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (dry_run)
+ {
+ suboid = InvalidOid;
+ snprintf(lsnstr, sizeof(lsnstr), "%X/%X", LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ suboid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ snprintf(lsnstr, sizeof(lsnstr), "%s", lsn);
+ }
+
+ /*
+ * The origin name is defined as pg_%u. %u is the subscription OID. See
+ * ApplyWorkerMain().
+ */
+ snprintf(originname, sizeof(originname), "pg_%u", suboid);
+
+ PQclear(res);
+
+ pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
+ originname, lsnstr, dbinfo->dbname);
+
+ resetPQExpBuffer(str);
+ appendPQExpBuffer(str,
+ "SELECT pg_catalog.pg_replication_origin_advance('%s', '%s')", originname, lsnstr);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not set replication progress for the subscription \"%s\": %s",
+ dbinfo->subname, PQresultErrorMessage(res));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Enables the subscription.
+ *
+ * The subscription was created in a previous step but it was disabled. After
+ * adjusting the initial location, enabling the subscription is the last step
+ * of this setup.
+ */
+static void
+enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+int
+main(int argc, char **argv)
+{
+ static struct option long_options[] =
+ {
+ {"help", no_argument, NULL, '?'},
+ {"version", no_argument, NULL, 'V'},
+ {"pgdata", required_argument, NULL, 'D'},
+ {"publisher-server", required_argument, NULL, 'P'},
+ {"subscriber-server", required_argument, NULL, 'S'},
+ {"database", required_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"recovery-timeout", required_argument, NULL, 't'},
+ {"retain", no_argument, NULL, 'r'},
+ {"verbose", no_argument, NULL, 'v'},
+ {NULL, 0, NULL, 0}
+ };
+
+ int c;
+ int option_index;
+
+ char *base_dir;
+ char *server_start_log;
+ int len;
+
+ char *pub_base_conninfo = NULL;
+ char *sub_base_conninfo = NULL;
+ char *dbname_conninfo = NULL;
+ char temp_replslot[NAMEDATALEN] = {0};
+
+ uint64 pub_sysid;
+ uint64 sub_sysid;
+ struct stat statbuf;
+
+ PGconn *conn;
+ char *consistent_lsn;
+
+ PQExpBuffer recoveryconfcontents = NULL;
+
+ char pidfile[MAXPGPATH];
+
+ pg_logging_init(argv[0]);
+ pg_logging_set_level(PG_LOG_WARNING);
+ progname = get_progname(argv[0]);
+ set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_createsubscriber"));
+
+ if (argc > 1)
+ {
+ if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+ {
+ usage();
+ exit(0);
+ }
+ else if (strcmp(argv[1], "-V") == 0
+ || strcmp(argv[1], "--version") == 0)
+ {
+ puts("pg_createsubscriber (PostgreSQL) " PG_VERSION);
+ exit(0);
+ }
+ }
+
+ /*
+ * Don't allow it to be run as root. It uses pg_ctl which does not allow
+ * it either.
+ */
+#ifndef WIN32
+ if (geteuid() == 0)
+ {
+ pg_log_error("cannot be executed by \"root\"");
+ pg_log_error_hint("You must run %s as the PostgreSQL superuser.",
+ progname);
+ exit(1);
+ }
+#endif
+
+ while ((c = getopt_long(argc, argv, "D:P:S:d:nrt:v",
+ long_options, &option_index)) != -1)
+ {
+ switch (c)
+ {
+ case 'D':
+ subscriber_dir = pg_strdup(optarg);
+ break;
+ case 'P':
+ pub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'S':
+ sub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'd':
+ /* Ignore duplicated database names. */
+ if (!simple_string_list_member(&database_names, optarg))
+ {
+ simple_string_list_append(&database_names, optarg);
+ num_dbs++;
+ }
+ break;
+ case 'n':
+ dry_run = true;
+ break;
+ case 'r':
+ retain = true;
+ break;
+ case 't':
+ recovery_timeout = atoi(optarg);
+ break;
+ case 'v':
+ pg_logging_increase_verbosity();
+ break;
+ default:
+ /* getopt_long already emitted a complaint */
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Any non-option arguments?
+ */
+ if (optind < argc)
+ {
+ pg_log_error("too many command-line arguments (first is \"%s\")",
+ argv[optind]);
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Required arguments
+ */
+ if (subscriber_dir == NULL)
+ {
+ pg_log_error("no subscriber data directory specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Parse connection string. Build a base connection string that might be
+ * reused by multiple databases.
+ */
+ if (pub_conninfo_str == NULL)
+ {
+ /*
+ * TODO use primary_conninfo (if available) from subscriber and
+ * extract publisher connection string. Assume that there are
+ * identical entries for physical and logical replication. If there is
+ * not, we would fail anyway.
+ */
+ pg_log_error("no publisher connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
+ "publisher");
+ if (pub_base_conninfo == NULL)
+ exit(1);
+
+ if (sub_conninfo_str == NULL)
+ {
+ pg_log_error("no subscriber connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
+ if (sub_base_conninfo == NULL)
+ exit(1);
+
+ if (database_names.head == NULL)
+ {
+ pg_log_info("no database was specified");
+
+ /*
+ * If --database option is not provided, try to obtain the dbname from
+ * the publisher conninfo. If dbname parameter is not available, error
+ * out.
+ */
+ if (dbname_conninfo)
+ {
+ simple_string_list_append(&database_names, dbname_conninfo);
+ num_dbs++;
+
+ pg_log_info("database \"%s\" was extracted from the publisher connection string",
+ dbname_conninfo);
+ }
+ else
+ {
+ pg_log_error("no database name specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
+ */
+ if (!get_exec_path(argv[0]))
+ exit(1);
+
+ /* rudimentary check for a data directory. */
+ if (!check_data_directory(subscriber_dir))
+ exit(1);
+
+ /* Store database information for publisher and subscriber. */
+ dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+
+ /* Register a function to clean up objects in case of failure. */
+ atexit(cleanup_objects_atexit);
+
+ /*
+ * Check if the subscriber data directory has the same system identifier
+ * than the publisher data directory.
+ */
+ pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
+ sub_sysid = get_control_from_datadir(subscriber_dir);
+ if (pub_sysid != sub_sysid)
+ {
+ pg_log_error("subscriber data directory is not a copy of the source database cluster");
+ exit(1);
+ }
+
+ /*
+ * Create the output directory to store any data generated by this tool.
+ */
+ base_dir = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("directory path for subscriber is too long");
+ exit(1);
+ }
+
+ if (mkdir(base_dir, pg_dir_create_mode) < 0 && errno != EEXIST)
+ {
+ pg_log_error("could not create directory \"%s\": %m", base_dir);
+ exit(1);
+ }
+
+ server_start_log = server_logfile_name(subscriber_dir);
+
+ /* subscriber PID file. */
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+
+ /*
+ * The standby server must be running. That's because some checks will be
+ * done (is it ready for a logical replication setup?). After that, stop
+ * the subscriber in preparation to modify some recovery parameters that
+ * require a restart.
+ */
+ if (stat(pidfile, &statbuf) == 0)
+ {
+ /*
+ * Check if the standby server is ready for logical replication.
+ */
+ if (!check_subscriber(dbinfo))
+ exit(1);
+
+ /*
+ * Check if the primary server is ready for logical replication. This
+ * routine checks if a replication slot is in use on primary so it
+ * relies on check_subscriber() to obtain the primary_slot_name.
+ * That's why it is called after it.
+ */
+ if (!check_publisher(dbinfo))
+ exit(1);
+
+ /*
+ * Create the required objects for each database on publisher. This
+ * step is here mainly because if we stop the standby we cannot verify
+ * if the primary slot is in use. We could use an extra connection for
+ * it but it doesn't seem worth.
+ */
+ if (!setup_publisher(dbinfo))
+ exit(1);
+
+ /* Stop the standby server. */
+ pg_log_info("standby is up and running");
+ pg_log_info("stopping the server to start the transformation steps");
+ stop_standby_server(pg_ctl_path, subscriber_dir);
+ }
+ else
+ {
+ pg_log_error("standby is not running");
+ pg_log_error_hint("Start the standby and try again.");
+ exit(1);
+ }
+
+ /*
+ * Create a temporary logical replication slot to get a consistent LSN.
+ *
+ * This consistent LSN will be used later to advanced the recently created
+ * replication slots. It is ok to use a temporary replication slot here
+ * because it will have a short lifetime and it is only used as a mark to
+ * start the logical replication.
+ *
+ * XXX we should probably use the last created replication slot to get a
+ * consistent LSN but it should be changed after adding pg_basebackup
+ * support.
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+ consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
+ temp_replslot);
+
+ /*
+ * Write recovery parameters.
+ *
+ * Despite of the recovery parameters will be written to the subscriber,
+ * use a publisher connection for the follwing recovery functions. The
+ * connection is only used to check the current server version (physical
+ * replica, same server version). The subscriber is not running yet. In
+ * dry run mode, the recovery parameters *won't* be written. An invalid
+ * LSN is used for printing purposes.
+ */
+ recoveryconfcontents = GenerateRecoveryConfig(conn, NULL);
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_inclusive = true\n");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_action = promote\n");
+
+ if (dry_run)
+ {
+ appendPQExpBuffer(recoveryconfcontents, "# dry run mode");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%X/%X'\n",
+ LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
+ consistent_lsn);
+ WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ }
+ disconnect_database(conn);
+
+ pg_log_debug("recovery parameters:\n%s", recoveryconfcontents->data);
+
+ /*
+ * Start subscriber and wait until accepting connections.
+ */
+ pg_log_info("starting the subscriber");
+ start_standby_server(pg_ctl_path, subscriber_dir, server_start_log);
+
+ /*
+ * Waiting the subscriber to be promoted.
+ */
+ wait_for_end_recovery(dbinfo[0].subconninfo);
+
+ /*
+ * Create the subscription for each database on subscriber. It does not
+ * enable it immediately because it needs to adjust the logical
+ * replication start point to the LSN reported by consistent_lsn (see
+ * set_replication_progress). It also cleans up publications created by
+ * this tool and replication to the standby.
+ */
+ if (!setup_subscriber(dbinfo, consistent_lsn))
+ exit(1);
+
+ /*
+ * If the primary_slot_name exists on primary, drop it.
+ *
+ * XXX we might not fail here. Instead, we provide a warning so the user
+ * eventually drops this replication slot later.
+ */
+ if (primary_slot_name != NULL)
+ {
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn != NULL)
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ }
+ else
+ {
+ pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
+ pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
+ }
+ disconnect_database(conn);
+ }
+
+ /*
+ * Stop the subscriber.
+ */
+ pg_log_info("stopping the subscriber");
+ stop_standby_server(pg_ctl_path, subscriber_dir);
+
+ /*
+ * Change system identifier.
+ */
+ modify_sysid(pg_resetwal_path, subscriber_dir);
+
+ /*
+ * The log file is kept if retain option is specified or this tool does
+ * not run successfully. Otherwise, log file is removed.
+ */
+ if (!retain)
+ unlink(server_start_log);
+
+ success = true;
+
+ pg_log_info("Done!");
+
+ return 0;
+}
diff --git a/src/bin/pg_basebackup/t/040_pg_createsubscriber.pl b/src/bin/pg_basebackup/t/040_pg_createsubscriber.pl
new file mode 100644
index 0000000000..0f02b1bfac
--- /dev/null
+++ b/src/bin/pg_basebackup/t/040_pg_createsubscriber.pl
@@ -0,0 +1,44 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test checking options of pg_createsubscriber.
+#
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+program_help_ok('pg_createsubscriber');
+program_version_ok('pg_createsubscriber');
+program_options_handling_ok('pg_createsubscriber');
+
+my $datadir = PostgreSQL::Test::Utils::tempdir;
+
+command_fails(['pg_createsubscriber'],
+ 'no subscriber data directory specified');
+command_fails(
+ [
+ 'pg_createsubscriber',
+ '--pgdata', $datadir
+ ],
+ 'no publisher connection string specified');
+command_fails(
+ [
+ 'pg_createsubscriber',
+ '--dry-run',
+ '--pgdata', $datadir,
+ '--publisher-server', 'dbname=postgres'
+ ],
+ 'no subscriber connection string specified');
+command_fails(
+ [
+ 'pg_createsubscriber',
+ '--verbose',
+ '--pgdata', $datadir,
+ '--publisher-server', 'dbname=postgres',
+ '--subscriber-server', 'dbname=postgres'
+ ],
+ 'no database name specified');
+
+done_testing();
diff --git a/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
new file mode 100644
index 0000000000..534bc53a76
--- /dev/null
+++ b/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
@@ -0,0 +1,139 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test using a standby server as the subscriber.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node_p;
+my $node_f;
+my $node_s;
+my $result;
+
+# Set up node P as primary
+$node_p = PostgreSQL::Test::Cluster->new('node_p');
+$node_p->init(allows_streaming => 'logical');
+$node_p->start;
+
+# Set up node F as about-to-fail node
+# The extra option forces it to initialize a new cluster instead of copying a
+# previously initdb's cluster.
+$node_f = PostgreSQL::Test::Cluster->new('node_f');
+$node_f->init(allows_streaming => 'logical', extra => [ '--no-instructions' ]);
+$node_f->start;
+
+# On node P
+# - create databases
+# - create test tables
+# - insert a row
+$node_p->safe_psql(
+ 'postgres', q(
+ CREATE DATABASE pg1;
+ CREATE DATABASE pg2;
+));
+$node_p->safe_psql('pg1', 'CREATE TABLE tbl1 (a text)');
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('first row')");
+$node_p->safe_psql('pg2', 'CREATE TABLE tbl2 (a text)');
+
+# Set up node S as standby linking to node P
+$node_p->backup('backup_1');
+$node_s = PostgreSQL::Test::Cluster->new('node_s');
+$node_s->init_from_backup($node_p, 'backup_1', has_streaming => 1);
+$node_s->append_conf('postgresql.conf', 'log_min_messages = debug2');
+$node_s->set_standby_mode();
+$node_s->start;
+
+# Insert another row on node P and wait node S to catch up
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('second row')");
+$node_p->wait_for_replay_catchup($node_s);
+
+# Run pg_createsubscriber on about-to-fail node F
+command_fails(
+ [
+ 'pg_createsubscriber', '--verbose',
+ '--pgdata', $node_f->data_dir,
+ '--publisher-server', $node_p->connstr('pg1'),
+ '--subscriber-server', $node_f->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'subscriber data directory is not a copy of the source database cluster');
+
+# dry run mode on node S
+command_ok(
+ [
+ 'pg_createsubscriber', '--verbose', '--dry-run',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-server', $node_p->connstr('pg1'),
+ '--subscriber-server', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_createsubscriber --dry-run on node S');
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+# Check if node S is still a standby
+is($node_s->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
+ 't', 'standby is in recovery');
+
+# Run pg_createsubscriber on node S
+command_ok(
+ [
+ 'pg_createsubscriber', '--verbose',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-server', $node_p->connstr('pg1'),
+ '--subscriber-server', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_createsubscriber on node S');
+
+# Insert rows on P
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('third row')");
+$node_p->safe_psql('pg2', "INSERT INTO tbl2 VALUES('row 1')");
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+
+# Get subscription names
+$result = $node_s->safe_psql(
+ 'postgres', qq(
+ SELECT subname FROM pg_subscription WHERE subname ~ '^pg_createsubscriber_'
+));
+my @subnames = split("\n", $result);
+
+# Wait subscriber to catch up
+$node_s->wait_for_subscription_sync($node_p, $subnames[0]);
+$node_s->wait_for_subscription_sync($node_p, $subnames[1]);
+
+# Check result on database pg1
+$result = $node_s->safe_psql('pg1', 'SELECT * FROM tbl1');
+is( $result, qq(first row
+second row
+third row),
+ 'logical replication works on database pg1');
+
+# Check result on database pg2
+$result = $node_s->safe_psql('pg2', 'SELECT * FROM tbl2');
+is( $result, qq(row 1),
+ 'logical replication works on database pg2');
+
+# Different system identifier?
+my $sysid_p = $node_p->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+my $sysid_s = $node_s->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+ok($sysid_p != $sysid_s, 'system identifier was changed');
+
+# clean up
+$node_p->teardown_node;
+$node_s->teardown_node;
+
+done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 90b37b919c..2c5725e18d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1505,6 +1505,7 @@ LogicalRepBeginData
LogicalRepCommitData
LogicalRepCommitPreparedTxnData
LogicalRepCtxStruct
+LogicalRepInfo
LogicalRepMsgType
LogicalRepPartMapEntry
LogicalRepPreparedTxnData
--
2.30.2
^ permalink raw reply [nested|flat] 77+ messages in thread
* RE: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 02:01 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
@ 2024-01-30 09:26 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 12:52 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-30 09:26 UTC (permalink / raw)
To: 'Euler Taveira' <[email protected]>; [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Shlok Kyal <[email protected]>
Dear Euler,
Thanks for updating the patch!
> One open item that is worrying me is how to handle the pg_ctl timeout. This
> patch does nothing and the user should use PGCTLTIMEOUT environment variable to
> avoid that the execution is canceled after 60 seconds (default for pg_ctl).
> Even if you set a high value, it might not be enough for cases like
> time-delayed replica. Maybe pg_ctl should accept no timeout as --timeout
> option. I'll include this caveat into the documentation but I'm afraid it is
> not sufficient and we should provide a better way to handle this situation.
I felt you might be confused a bit. Even if the recovery_min_apply_delay is set,
e.g., 10h., the pg_ctl can start and stop the server. This is because the
walreceiver serialize changes as soon as they received. The delay is done by the
startup process. There are no unflushed data, so server instance can be turned off.
If you meant the combination of recovery-timeout and time-delayed replica, yes,
it would be likely to occur. But in the case, using like --no-timeout option is
dangerous. I think we should overwrite recovery_min_apply_delay to zero. Thought?
Below part contains my comments for v11-0001. Note that the ordering is random.
01. doc
```
<group choice="req">
<arg choice="plain"><option>-D</option> </arg>
<arg choice="plain"><option>--pgdata</option></arg>
</group>
```
According to other documentation like pg_upgrade, we do not write both longer
and shorter options in the synopsis section.
02. doc
```
<para>
<application>pg_createsubscriber</application> takes the publisher and subscriber
connection strings, a cluster directory from a physical replica and a list of
database names and it sets up a new logical replica using the physical
recovery process.
</para>
```
I found that you did not include my suggestion without saying [1]. Do you dislike
the comment or still considering?
03. doc
```
<term><option>-P <replaceable class="parameter">connstr</replaceable></option></term>
```
Too many blank after -P.
04. doc
```
<para>
<application>pg_createsubscriber</application> checks if the target data
directory is used by a physical replica. Stop the physical replica if it is
running. One of the next steps is to add some recovery parameters that
requires a server start. This step avoids an error.
</para>
```
I think doc is not updated. Currently (not sure it is good) the physical replica
must be running before doing pg_createsubscriber.
05. doc
```
each specified database on the source server. The replication slot name
contains a <literal>pg_createsubscriber</literal> prefix. These replication
slots will be used by the subscriptions in a future step. A temporary
```
According to the "Replication Slot Management" subsection in logical-replication.sgml,
the format of names should be described expressly, e.g.:
```
These replication slots have generated names:"pg_createsubscriber_%u_%d" (parameters: Subscription oid, Process id pid).
```
Publications, a temporary slot, and subscriptions should be described as well.
06. doc
```
Next, <application>pg_createsubscriber</application> creates one publication
for each specified database on the source server. Each publication
replicates changes for all tables in the database. The publication name
```
Missing update. Publications are created before creating replication slots.
07. general
I think there are some commenting conversions in PG, but this file breaks it.
* Comments must be one-line as much as possible
* Periods must not be added when the comment is one-line.
* Initial character must be capital.
08. general
Some pg_log_error() + exit(1) can be replaced with pg_fatal().
09. LogicalRepInfo
```
char *subconninfo; /* subscription connection string for logical
* replication */
```
As I posted in comment#8[2], I don't think it is "subscription connection". Also,
"for logical replication" is bit misreading because it would not be passed to
workers.
10. get_base_conninfo
```
static char *
get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
...
/*
* If --database option is not provided, try to obtain the dbname from
* the publisher conninfo. If dbname parameter is not available, error
* out.
*/
```
I'm not sure getting dbname from the conninfo improves user-experience. I felt
it may trigger an unintended targeting.
(I still think the publisher-server should be removed)
11. check_data_directory
```
/*
* Is it a cluster directory? These are preliminary checks. It is far from
* making an accurate check. If it is not a clone from the publisher, it will
* eventually fail in a future step.
*/
static bool
check_data_directory(const char *datadir)
```
We shoud also check whether pg_createsubscriber can create a file and a directory.
GetDataDirectoryCreatePerm() verifies it.
12. main
```
/* Register a function to clean up objects in case of failure. */
atexit(cleanup_objects_atexit);
```
According to the manpage, callback functions would not be called when it exits
due to signals:
> Functions registered using atexit() (and on_exit(3)) are not called if a
> process terminates abnormally because of the delivery of a signal.
Do you have a good way to handle the case? One solution is to output created
objects in any log level, but the consideration may be too much. Thought?
13, main
```
/*
* Create a temporary logical replication slot to get a consistent LSN.
```
Just to clarify - I still think the process exits before here in case of dry run.
In case of pg_resetwal, the process exits before doing actual works like
RewriteControlFile().
14. main
```
* XXX we might not fail here. Instead, we provide a warning so the user
* eventually drops this replication slot later.
```
But there are possibilities to exit(1) in drop_replication_slot(). Is it acceptable?
15. wait_for_end_recovery
```
/*
* Bail out after recovery_timeout seconds if this option is set.
*/
if (recovery_timeout > 0 && timer >= recovery_timeout)
```
Hmm, IIUC, it should be enabled by default [3]. Do you have anything in your mind?
16. main
```
/*
* Create the subscription for each database on subscriber. It does not
* enable it immediately because it needs to adjust the logical
* replication start point to the LSN reported by consistent_lsn (see
* set_replication_progress). It also cleans up publications created by
* this tool and replication to the standby.
*/
if (!setup_subscriber(dbinfo, consistent_lsn))
```
Subscriptions would be created and replication origin would be moved forward here,
but latter one can be done only by the superuser. I felt that this should be
checked in check_subscriber().
17. main
```
/*
* Change system identifier.
*/
modify_sysid(pg_resetwal_path, subscriber_dir);
```
Even if I executed without -v option, an output from pg_resetwal command appears.
It seems bit strange.
```
$ pg_createsubscriber -D data_N2/ -P "port=5431 user=postgres" -S "port=5432 user=postgres" -d postgres
Write-ahead log reset
$
```
[1]: https://www.postgresql.org/message-id/TY3PR01MB9889C362FF76102C88FA1C29F56F2%40TY3PR01MB9889.jpnprd0...
[2]: https://www.postgresql.org/message-id/TY3PR01MB9889C362FF76102C88FA1C29F56F2%40TY3PR01MB9889.jpnprd0...
[3]: https://www.postgresql.org/message-id/CAA4eK1JRgnhv_ySzuFjN7UaX9qxz5Hqcwew7Fk%3D%2BAbJbu0Kd9w%40mail...
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/global/
^ permalink raw reply [nested|flat] 77+ messages in thread
* RE: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 02:01 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 09:26 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
@ 2024-01-31 12:52 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 14:25 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-31 12:52 UTC (permalink / raw)
To: 'Euler Taveira' <[email protected]>; [email protected] <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Shlok Kyal <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>
Dear Euler,
I extracted some review comments which may require many efforts. I hope this makes them
easy to review.
0001: not changed from yours.
0002: avoid to use replication connections. Source: comment #3[1]
0003: Remove -P option and use primary_conninfo instead. Source: [2]
0004: Exit earlier when dry_run is specified. Source: [3]
0005: Refactor data structures. Source: [4]
[1]: https://www.postgresql.org/message-id/TY3PR01MB9889593399165B9A04106741F5662%40TY3PR01MB9889.jpnprd0...
[2]: https://www.postgresql.org/message-id/TY3PR01MB98897C85700C6DF942D2D0A3F5792%40TY3PR01MB9889.jpnprd0...
[3]: https://www.postgresql.org/message-id/TY3PR01MB98897C85700C6DF942D2D0A3F5792%40TY3PR01MB9889.jpnprd0...
[4]: https://www.postgresql.org/message-id/TY3PR01MB9889C362FF76102C88FA1C29F56F2%40TY3PR01MB9889.jpnprd0...
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/
Attachments:
[application/octet-stream] v12-0001-Creates-a-new-logical-replica-from-a-standby-ser.patch (76.2K, ../../TY3PR01MB9889320DDAF153FC8A532630F57C2@TY3PR01MB9889.jpnprd01.prod.outlook.com/2-v12-0001-Creates-a-new-logical-replica-from-a-standby-ser.patch)
download | inline diff:
From 14e74d8df45fcc25d0ba9d8a8417a1c9bb95b6aa Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email protected]>
Date: Mon, 5 Jun 2023 14:39:40 -0400
Subject: [PATCH v12 1/5] Creates a new logical replica from a standby server
A new tool called pg_createsubscriber can convert a physical replica
into a logical replica. It runs on the target server and should be able
to connect to the source server (publisher) and the target server
(subscriber).
The conversion requires a few steps. Check if the target data directory
has the same system identifier than the source data directory. Stop the
target server if it is running as a standby server. Create one
replication slot per specified database on the source server. One
additional replication slot is created at the end to get the consistent
LSN (This consistent LSN will be used as (a) a stopping point for the
recovery process and (b) a starting point for the subscriptions). Write
recovery parameters into the target data directory and start the target
server (Wait until the target server is promoted). Create one
publication (FOR ALL TABLES) per specified database on the source
server. Create one subscription per specified database on the target
server (Use replication slot and publication created in a previous step.
Don't enable the subscriptions yet). Sets the replication progress to
the consistent LSN that was got in a previous step. Enable the
subscription for each specified database on the target server. Stop the
target server. Change the system identifier from the target server.
Depending on your workload and database size, creating a logical replica
couldn't be an option due to resource constraints (WAL backlog should be
available until all table data is synchronized). The initial data copy
and the replication progress tends to be faster on a physical replica.
The purpose of this tool is to speed up a logical replica setup.
---
doc/src/sgml/ref/allfiles.sgml | 1 +
doc/src/sgml/ref/pg_createsubscriber.sgml | 322 +++
doc/src/sgml/reference.sgml | 1 +
src/bin/pg_basebackup/.gitignore | 1 +
src/bin/pg_basebackup/Makefile | 8 +-
src/bin/pg_basebackup/meson.build | 19 +
src/bin/pg_basebackup/pg_createsubscriber.c | 1852 +++++++++++++++++
.../t/040_pg_createsubscriber.pl | 44 +
.../t/041_pg_createsubscriber_standby.pl | 139 ++
src/tools/pgindent/typedefs.list | 1 +
10 files changed, 2387 insertions(+), 1 deletion(-)
create mode 100644 doc/src/sgml/ref/pg_createsubscriber.sgml
create mode 100644 src/bin/pg_basebackup/pg_createsubscriber.c
create mode 100644 src/bin/pg_basebackup/t/040_pg_createsubscriber.pl
create mode 100644 src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index 4a42999b18..a2b5eea0e0 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -214,6 +214,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY pgResetwal SYSTEM "pg_resetwal.sgml">
<!ENTITY pgRestore SYSTEM "pg_restore.sgml">
<!ENTITY pgRewind SYSTEM "pg_rewind.sgml">
+<!ENTITY pgCreateSubscriber SYSTEM "pg_createsubscriber.sgml">
<!ENTITY pgVerifyBackup SYSTEM "pg_verifybackup.sgml">
<!ENTITY pgtestfsync SYSTEM "pgtestfsync.sgml">
<!ENTITY pgtesttiming SYSTEM "pgtesttiming.sgml">
diff --git a/doc/src/sgml/ref/pg_createsubscriber.sgml b/doc/src/sgml/ref/pg_createsubscriber.sgml
new file mode 100644
index 0000000000..1c78ff92e0
--- /dev/null
+++ b/doc/src/sgml/ref/pg_createsubscriber.sgml
@@ -0,0 +1,322 @@
+<!--
+doc/src/sgml/ref/pg_createsubscriber.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="app-pgcreatesubscriber">
+ <indexterm zone="app-pgcreatesubscriber">
+ <primary>pg_createsubscriber</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle><application>pg_createsubscriber</application></refentrytitle>
+ <manvolnum>1</manvolnum>
+ <refmiscinfo>Application</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>pg_createsubscriber</refname>
+ <refpurpose>convert a physical replica into a new logical replica</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_createsubscriber</command>
+ <arg rep="repeat"><replaceable>option</replaceable></arg>
+ <group choice="plain">
+ <group choice="req">
+ <arg choice="plain"><option>-D</option> </arg>
+ <arg choice="plain"><option>--pgdata</option></arg>
+ </group>
+ <replaceable>datadir</replaceable>
+ <group choice="req">
+ <arg choice="plain"><option>-P</option></arg>
+ <arg choice="plain"><option>--publisher-server</option></arg>
+ </group>
+ <replaceable>connstr</replaceable>
+ <group choice="req">
+ <arg choice="plain"><option>-S</option></arg>
+ <arg choice="plain"><option>--subscriber-server</option></arg>
+ </group>
+ <replaceable>connstr</replaceable>
+ <group choice="req">
+ <arg choice="plain"><option>-d</option></arg>
+ <arg choice="plain"><option>--database</option></arg>
+ </group>
+ <replaceable>dbname</replaceable>
+ </group>
+ </cmdsynopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+ <para>
+ <application>pg_createsubscriber</application> takes the publisher and subscriber
+ connection strings, a cluster directory from a physical replica and a list of
+ database names and it sets up a new logical replica using the physical
+ recovery process.
+ </para>
+
+ <para>
+ The <application>pg_createsubscriber</application> should be run at the target
+ server. The source server (known as publisher server) should accept logical
+ replication connections from the target server (known as subscriber server).
+ The target server should accept local logical replication connection.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Options</title>
+
+ <para>
+ <application>pg_createsubscriber</application> accepts the following
+ command-line arguments:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-D <replaceable class="parameter">directory</replaceable></option></term>
+ <term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term>
+ <listitem>
+ <para>
+ The target directory that contains a cluster directory from a physical
+ replica.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-P <replaceable class="parameter">connstr</replaceable></option></term>
+ <term><option>--publisher-server=<replaceable class="parameter">connstr</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the publisher. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-S <replaceable class="parameter">connstr</replaceable></option></term>
+ <term><option>--subscriber-server=<replaceable class="parameter">connstr</replaceable></option></term>
+ <listitem>
+ <para>
+ The connection string to the subscriber. For details see <xref linkend="libpq-connstring"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-d <replaceable class="parameter">dbname</replaceable></option></term>
+ <term><option>--database=<replaceable class="parameter">dbname</replaceable></option></term>
+ <listitem>
+ <para>
+ The database name to create the subscription. Multiple databases can be
+ selected by writing multiple <option>-d</option> switches.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
+ <listitem>
+ <para>
+ Do everything except actually modifying the target directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-r</option></term>
+ <term><option>--retain</option></term>
+ <listitem>
+ <para>
+ Retain log file even after successful completion.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-t <replaceable class="parameter">seconds</replaceable></option></term>
+ <term><option>--recovery-timeout=<replaceable class="parameter">seconds</replaceable></option></term>
+ <listitem>
+ <para>
+ The maximum number of seconds to wait for recovery to end. Setting to 0
+ disables. The default is 0.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-v</option></term>
+ <term><option>--verbose</option></term>
+ <listitem>
+ <para>
+ Enables verbose mode. This will cause
+ <application>pg_createsubscriber</application> to output progress messages
+ and detailed information about each step to standard error.
+ Repeating the option causes additional debug-level messages to appear on
+ standard error.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ Other options are also available:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-V</option></term>
+ <term><option>--version</option></term>
+ <listitem>
+ <para>
+ Print the <application>pg_createsubscriber</application> version and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-?</option></term>
+ <term><option>--help</option></term>
+ <listitem>
+ <para>
+ Show help about <application>pg_createsubscriber</application> command
+ line arguments, and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Notes</title>
+
+ <para>
+ The transformation proceeds in the following steps:
+ </para>
+
+ <procedure>
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> checks if the given target data
+ directory has the same system identifier than the source data directory.
+ Since it uses the recovery process as one of the steps, it starts the
+ target server as a replica from the source server. If the system
+ identifier is not the same, <application>pg_createsubscriber</application> will
+ terminate with an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> checks if the target data
+ directory is used by a physical replica. Stop the physical replica if it is
+ running. One of the next steps is to add some recovery parameters that
+ requires a server start. This step avoids an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> creates one replication slot for
+ each specified database on the source server. The replication slot name
+ contains a <literal>pg_createsubscriber</literal> prefix. These replication
+ slots will be used by the subscriptions in a future step. A temporary
+ replication slot is used to get a consistent start location. This
+ consistent LSN will be used as a stopping point in the <xref
+ linkend="guc-recovery-target-lsn"/> parameter and by the
+ subscriptions as a replication starting point. It guarantees that no
+ transaction will be lost.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> writes recovery parameters into
+ the target data directory and start the target server. It specifies a LSN
+ (consistent LSN that was obtained in the previous step) of write-ahead
+ log location up to which recovery will proceed. It also specifies
+ <literal>promote</literal> as the action that the server should take once
+ the recovery target is reached. This step finishes once the server ends
+ standby mode and is accepting read-write operations.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Next, <application>pg_createsubscriber</application> creates one publication
+ for each specified database on the source server. Each publication
+ replicates changes for all tables in the database. The publication name
+ contains a <literal>pg_createsubscriber</literal> prefix. These publication
+ will be used by a corresponding subscription in a next step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> creates one subscription for
+ each specified database on the target server. Each subscription name
+ contains a <literal>pg_createsubscriber</literal> prefix. The replication slot
+ name is identical to the subscription name. It does not copy existing data
+ from the source server. It does not create a replication slot. Instead, it
+ uses the replication slot that was created in a previous step. The
+ subscription is created but it is not enabled yet. The reason is the
+ replication progress must be set to the consistent LSN but replication
+ origin name contains the subscription oid in its name. Hence, the
+ subscription will be enabled in a separate step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> sets the replication progress to
+ the consistent LSN that was obtained in a previous step. When the target
+ server started the recovery process, it caught up to the consistent LSN.
+ This is the exact LSN to be used as a initial location for each
+ subscription.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Finally, <application>pg_createsubscriber</application> enables the subscription
+ for each specified database on the target server. The subscription starts
+ streaming from the consistent LSN.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_createsubscriber</application> stops the target server to change
+ its system identifier.
+ </para>
+ </step>
+ </procedure>
+ </refsect1>
+
+ <refsect1>
+ <title>Examples</title>
+
+ <para>
+ To create a logical replica for databases <literal>hr</literal> and
+ <literal>finance</literal> from a physical replica at <literal>foo</literal>:
+<screen>
+<prompt>$</prompt> <userinput>pg_createsubscriber -D /usr/local/pgsql/data -P "host=foo" -S "host=localhost" -d hr -d finance</userinput>
+</screen>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="app-pgbasebackup"/></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index aa94f6adf6..c5edd244ef 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -285,6 +285,7 @@
&pgCtl;
&pgResetwal;
&pgRewind;
+ &pgCreateSubscriber;
&pgtestfsync;
&pgtesttiming;
&pgupgrade;
diff --git a/src/bin/pg_basebackup/.gitignore b/src/bin/pg_basebackup/.gitignore
index 26048bdbd8..b3a6f5a2fe 100644
--- a/src/bin/pg_basebackup/.gitignore
+++ b/src/bin/pg_basebackup/.gitignore
@@ -1,5 +1,6 @@
/pg_basebackup
/pg_receivewal
/pg_recvlogical
+/pg_createsubscriber
/tmp_check/
diff --git a/src/bin/pg_basebackup/Makefile b/src/bin/pg_basebackup/Makefile
index abfb6440ec..ded434b683 100644
--- a/src/bin/pg_basebackup/Makefile
+++ b/src/bin/pg_basebackup/Makefile
@@ -44,7 +44,7 @@ BBOBJS = \
bbstreamer_tar.o \
bbstreamer_zstd.o
-all: pg_basebackup pg_receivewal pg_recvlogical
+all: pg_basebackup pg_receivewal pg_recvlogical pg_createsubscriber
pg_basebackup: $(BBOBJS) $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) $(BBOBJS) $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
@@ -55,10 +55,14 @@ pg_receivewal: pg_receivewal.o $(OBJS) | submake-libpq submake-libpgport submake
pg_recvlogical: pg_recvlogical.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_recvlogical.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_createsubscriber: $(WIN32RES) pg_createsubscriber.o | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
install: all installdirs
$(INSTALL_PROGRAM) pg_basebackup$(X) '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
$(INSTALL_PROGRAM) pg_receivewal$(X) '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
$(INSTALL_PROGRAM) pg_recvlogical$(X) '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ $(INSTALL_PROGRAM) pg_createsubscriber$(X) '$(DESTDIR)$(bindir)/pg_createsubscriber$(X)'
installdirs:
$(MKDIR_P) '$(DESTDIR)$(bindir)'
@@ -67,10 +71,12 @@ uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_receivewal$(X)'
rm -f '$(DESTDIR)$(bindir)/pg_recvlogical$(X)'
+ rm -f '$(DESTDIR)$(bindir)/pg_createsubscriber$(X)'
clean distclean:
rm -f pg_basebackup$(X) pg_receivewal$(X) pg_recvlogical$(X) \
$(BBOBJS) pg_receivewal.o pg_recvlogical.o \
+ pg_createsubscriber$(X) pg_createsubscriber.o \
$(OBJS)
rm -rf tmp_check
diff --git a/src/bin/pg_basebackup/meson.build b/src/bin/pg_basebackup/meson.build
index f7e60e6670..345a2d6fcd 100644
--- a/src/bin/pg_basebackup/meson.build
+++ b/src/bin/pg_basebackup/meson.build
@@ -75,6 +75,23 @@ pg_recvlogical = executable('pg_recvlogical',
)
bin_targets += pg_recvlogical
+pg_createsubscriber_sources = files(
+ 'pg_createsubscriber.c'
+)
+
+if host_system == 'windows'
+ pg_createsubscriber_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'pg_createsubscriber',
+ '--FILEDESC', 'pg_createsubscriber - create a new logical replica from a standby server',])
+endif
+
+pg_createsubscriber = executable('pg_createsubscriber',
+ pg_createsubscriber_sources,
+ dependencies: [frontend_code, libpq],
+ kwargs: default_bin_args,
+)
+bin_targets += pg_createsubscriber
+
tests += {
'name': 'pg_basebackup',
'sd': meson.current_source_dir(),
@@ -89,6 +106,8 @@ tests += {
't/011_in_place_tablespace.pl',
't/020_pg_receivewal.pl',
't/030_pg_recvlogical.pl',
+ 't/040_pg_createsubscriber.pl',
+ 't/041_pg_createsubscriber_standby.pl',
],
},
}
diff --git a/src/bin/pg_basebackup/pg_createsubscriber.c b/src/bin/pg_basebackup/pg_createsubscriber.c
new file mode 100644
index 0000000000..478560b3e4
--- /dev/null
+++ b/src/bin/pg_basebackup/pg_createsubscriber.c
@@ -0,0 +1,1852 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_createsubscriber.c
+ * Create a new logical replica from a standby server
+ *
+ * Copyright (C) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_basebackup/pg_createsubscriber.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include <signal.h>
+#include <sys/stat.h>
+#include <sys/time.h>
+#include <sys/wait.h>
+#include <time.h>
+
+#include "access/xlogdefs.h"
+#include "catalog/pg_control.h"
+#include "common/connect.h"
+#include "common/controldata_utils.h"
+#include "common/file_perm.h"
+#include "common/file_utils.h"
+#include "common/logging.h"
+#include "fe_utils/recovery_gen.h"
+#include "fe_utils/simple_list.h"
+#include "getopt_long.h"
+#include "utils/pidfile.h"
+
+#define PGS_OUTPUT_DIR "pg_createsubscriber_output.d"
+
+typedef struct LogicalRepInfo
+{
+ Oid oid; /* database OID */
+ char *dbname; /* database name */
+ char *pubconninfo; /* publication connection string for logical
+ * replication */
+ char *subconninfo; /* subscription connection string for logical
+ * replication */
+ char *pubname; /* publication name */
+ char *subname; /* subscription name (also replication slot
+ * name) */
+
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepInfo;
+
+static void cleanup_objects_atexit(void);
+static void usage();
+static char *get_base_conninfo(char *conninfo, char *dbname,
+ const char *noderole);
+static bool get_exec_path(const char *path);
+static bool check_data_directory(const char *datadir);
+static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
+static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
+static PGconn *connect_database(const char *conninfo);
+static void disconnect_database(PGconn *conn);
+static uint64 get_sysid_from_conn(const char *conninfo);
+static uint64 get_control_from_datadir(const char *datadir);
+static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
+static bool check_publisher(LogicalRepInfo *dbinfo);
+static bool setup_publisher(LogicalRepInfo *dbinfo);
+static bool check_subscriber(LogicalRepInfo *dbinfo);
+static bool setup_subscriber(LogicalRepInfo *dbinfo, const char *consistent_lsn);
+static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name);
+static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static char *server_logfile_name(const char *datadir);
+static void start_standby_server(const char *pg_ctl_path, const char *datadir, const char *logfile);
+static void stop_standby_server(const char *pg_ctl_path, const char *datadir);
+static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
+static void wait_for_end_recovery(const char *conninfo);
+static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
+static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+
+#define USEC_PER_SEC 1000000
+#define WAIT_INTERVAL 1 /* 1 second */
+
+/* Options */
+static const char *progname;
+
+static char *subscriber_dir = NULL;
+static char *pub_conninfo_str = NULL;
+static char *sub_conninfo_str = NULL;
+static SimpleStringList database_names = {NULL, NULL};
+static char *primary_slot_name = NULL;
+static bool dry_run = false;
+static bool retain = false;
+static int recovery_timeout = 0;
+
+static bool success = false;
+
+static char *pg_ctl_path = NULL;
+static char *pg_resetwal_path = NULL;
+
+static LogicalRepInfo *dbinfo;
+static int num_dbs = 0;
+
+enum WaitPMResult
+{
+ POSTMASTER_READY,
+ POSTMASTER_STANDBY,
+ POSTMASTER_STILL_STARTING,
+ POSTMASTER_FAILED
+};
+
+
+/*
+ * Cleanup objects that were created by pg_createsubscriber if there is an error.
+ *
+ * Replication slots, publications and subscriptions are created. Depending on
+ * the step it failed, it should remove the already created objects if it is
+ * possible (sometimes it won't work due to a connection issue).
+ */
+static void
+cleanup_objects_atexit(void)
+{
+ PGconn *conn;
+ int i;
+
+ if (success)
+ return;
+
+ for (i = 0; i < num_dbs; i++)
+ {
+ if (dbinfo[i].made_subscription)
+ {
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn != NULL)
+ {
+ drop_subscription(conn, &dbinfo[i]);
+ if (dbinfo[i].made_publication)
+ drop_publication(conn, &dbinfo[i]);
+ disconnect_database(conn);
+ }
+ }
+
+ if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ {
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn != NULL)
+ {
+ if (dbinfo[i].made_publication)
+ drop_publication(conn, &dbinfo[i]);
+ if (dbinfo[i].made_replslot)
+ drop_replication_slot(conn, &dbinfo[i], NULL);
+ disconnect_database(conn);
+ }
+ }
+ }
+}
+
+static void
+usage(void)
+{
+ printf(_("%s creates a new logical replica from a standby server.\n\n"),
+ progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION]...\n"), progname);
+ printf(_("\nOptions:\n"));
+ printf(_(" -D, --pgdata=DATADIR location for the subscriber data directory\n"));
+ printf(_(" -P, --publisher-server=CONNSTR publisher connection string\n"));
+ printf(_(" -S, --subscriber-server=CONNSTR subscriber connection string\n"));
+ printf(_(" -d, --database=DBNAME database to create a subscription\n"));
+ printf(_(" -n, --dry-run stop before modifying anything\n"));
+ printf(_(" -t, --recovery-timeout=SECS seconds to wait for recovery to end\n"));
+ printf(_(" -r, --retain retain log file after success\n"));
+ printf(_(" -v, --verbose output verbose messages\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
+ printf(_("\nReport bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+ printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
+
+/*
+ * Validate a connection string. Returns a base connection string that is a
+ * connection string without a database name.
+ * Since we might process multiple databases, each database name will be
+ * appended to this base connection string to provide a final connection string.
+ * If the second argument (dbname) is not null, returns dbname if the provided
+ * connection string contains it. If option --database is not provided, uses
+ * dbname as the only database to setup the logical replica.
+ * It is the caller's responsibility to free the returned connection string and
+ * dbname.
+ */
+static char *
+get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ PQconninfoOption *conn_opts = NULL;
+ PQconninfoOption *conn_opt;
+ char *errmsg = NULL;
+ char *ret;
+ int i;
+
+ pg_log_info("validating connection string on %s", noderole);
+
+ conn_opts = PQconninfoParse(conninfo, &errmsg);
+ if (conn_opts == NULL)
+ {
+ pg_log_error("could not parse connection string: %s", errmsg);
+ return NULL;
+ }
+
+ i = 0;
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (strcmp(conn_opt->keyword, "dbname") == 0 && conn_opt->val != NULL)
+ {
+ if (dbname)
+ dbname = pg_strdup(conn_opt->val);
+ continue;
+ }
+
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0')
+ {
+ if (i > 0)
+ appendPQExpBufferChar(buf, ' ');
+ appendPQExpBuffer(buf, "%s=%s", conn_opt->keyword, conn_opt->val);
+ i++;
+ }
+ }
+
+ ret = pg_strdup(buf->data);
+
+ destroyPQExpBuffer(buf);
+ PQconninfoFree(conn_opts);
+
+ return ret;
+}
+
+/*
+ * Get the absolute path from other PostgreSQL binaries (pg_ctl and
+ * pg_resetwal) that is used by it.
+ */
+static bool
+get_exec_path(const char *path)
+{
+ int rc;
+
+ pg_ctl_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_ctl",
+ "pg_ctl (PostgreSQL) " PG_VERSION "\n",
+ pg_ctl_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_ctl", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_ctl", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
+
+ pg_resetwal_path = pg_malloc(MAXPGPATH);
+ rc = find_other_exec(path, "pg_resetwal",
+ "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
+ pg_resetwal_path);
+ if (rc < 0)
+ {
+ char full_path[MAXPGPATH];
+
+ if (find_my_exec(path, full_path) < 0)
+ strlcpy(full_path, progname, sizeof(full_path));
+ if (rc == -1)
+ pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
+ "same directory as \"%s\".\n"
+ "Check your installation.",
+ "pg_resetwal", progname, full_path);
+ else
+ pg_log_error("The program \"%s\" was found by \"%s\"\n"
+ "but was not the same version as %s.\n"
+ "Check your installation.",
+ "pg_resetwal", full_path, progname);
+ return false;
+ }
+
+ pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+
+ return true;
+}
+
+/*
+ * Is it a cluster directory? These are preliminary checks. It is far from
+ * making an accurate check. If it is not a clone from the publisher, it will
+ * eventually fail in a future step.
+ */
+static bool
+check_data_directory(const char *datadir)
+{
+ struct stat statbuf;
+ char versionfile[MAXPGPATH];
+
+ pg_log_info("checking if directory \"%s\" is a cluster data directory",
+ datadir);
+
+ if (stat(datadir, &statbuf) != 0)
+ {
+ if (errno == ENOENT)
+ pg_log_error("data directory \"%s\" does not exist", datadir);
+ else
+ pg_log_error("could not access directory \"%s\": %s", datadir, strerror(errno));
+
+ return false;
+ }
+
+ snprintf(versionfile, MAXPGPATH, "%s/PG_VERSION", datadir);
+ if (stat(versionfile, &statbuf) != 0 && errno == ENOENT)
+ {
+ pg_log_error("directory \"%s\" is not a database cluster directory", datadir);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Append database name into a base connection string.
+ *
+ * dbname is the only parameter that changes so it is not included in the base
+ * connection string. This function concatenates dbname to build a "real"
+ * connection string.
+ */
+static char *
+concat_conninfo_dbname(const char *conninfo, const char *dbname)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *ret;
+
+ Assert(conninfo != NULL);
+
+ appendPQExpBufferStr(buf, conninfo);
+ appendPQExpBuffer(buf, " dbname=%s", dbname);
+
+ ret = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+
+ return ret;
+}
+
+/*
+ * Store publication and subscription information.
+ */
+static LogicalRepInfo *
+store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+{
+ LogicalRepInfo *dbinfo;
+ SimpleStringListCell *cell;
+ int i = 0;
+
+ dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+
+ for (cell = database_names.head; cell; cell = cell->next)
+ {
+ char *conninfo;
+
+ /* Publisher. */
+ conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
+ dbinfo[i].pubconninfo = conninfo;
+ dbinfo[i].dbname = cell->val;
+ dbinfo[i].made_replslot = false;
+ dbinfo[i].made_publication = false;
+ dbinfo[i].made_subscription = false;
+ /* other struct fields will be filled later. */
+
+ /* Subscriber. */
+ conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
+ dbinfo[i].subconninfo = conninfo;
+
+ i++;
+ }
+
+ return dbinfo;
+}
+
+static PGconn *
+connect_database(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ const char *rconninfo;
+
+ /* logical replication mode */
+ rconninfo = psprintf("%s replication=database", conninfo);
+
+ conn = PQconnectdb(rconninfo);
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ pg_log_error("connection to database failed: %s", PQerrorMessage(conn));
+ return NULL;
+ }
+
+ /* secure search_path */
+ res = PQexec(conn, ALWAYS_SECURE_SEARCH_PATH_SQL);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not clear search_path: %s", PQresultErrorMessage(res));
+ return NULL;
+ }
+ PQclear(res);
+
+ return conn;
+}
+
+static void
+disconnect_database(PGconn *conn)
+{
+ Assert(conn != NULL);
+
+ PQfinish(conn);
+}
+
+/*
+ * Obtain the system identifier using the provided connection. It will be used
+ * to compare if a data directory is a clone of another one.
+ */
+static uint64
+get_sysid_from_conn(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from publisher");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "IDENTIFY_SYSTEM");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not send replication command \"%s\": %s",
+ "IDENTIFY_SYSTEM", PQresultErrorMessage(res));
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+ if (PQntuples(res) != 1 || PQnfields(res) < 3)
+ {
+ pg_log_error("could not identify system: got %d rows and %d fields, expected %d rows and %d or more fields",
+ PQntuples(res), PQnfields(res), 1, 3);
+
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+
+ sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+
+ pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+
+ disconnect_database(conn);
+
+ return sysid;
+}
+
+/*
+ * Obtain the system identifier from control file. It will be used to compare
+ * if a data directory is a clone of another one. This routine is used locally
+ * and avoids a replication connection.
+ */
+static uint64
+get_control_from_datadir(const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ uint64 sysid;
+
+ pg_log_info("getting system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ sysid = cf->system_identifier;
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+
+ pfree(cf);
+
+ return sysid;
+}
+
+/*
+ * Modify the system identifier. Since a standby server preserves the system
+ * identifier, it makes sense to change it to avoid situations in which WAL
+ * files from one of the systems might be used in the other one.
+ */
+static void
+modify_sysid(const char *pg_resetwal_path, const char *datadir)
+{
+ ControlFileData *cf;
+ bool crc_ok;
+ struct timeval tv;
+
+ char *cmd_str;
+ int rc;
+
+ pg_log_info("modifying system identifier from subscriber");
+
+ cf = get_controlfile(datadir, &crc_ok);
+ if (!crc_ok)
+ {
+ pg_log_error("control file appears to be corrupt");
+ exit(1);
+ }
+
+ /*
+ * Select a new system identifier.
+ *
+ * XXX this code was extracted from BootStrapXLOG().
+ */
+ gettimeofday(&tv, NULL);
+ cf->system_identifier = ((uint64) tv.tv_sec) << 32;
+ cf->system_identifier |= ((uint64) tv.tv_usec) << 12;
+ cf->system_identifier |= getpid() & 0xFFF;
+
+ if (!dry_run)
+ update_controlfile(datadir, cf, true);
+
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) cf->system_identifier);
+
+ pg_log_info("running pg_resetwal on the subscriber");
+
+ cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+
+ pg_log_debug("command is: %s", cmd_str);
+
+ if (!dry_run)
+ {
+ rc = system(cmd_str);
+ if (rc == 0)
+ pg_log_info("subscriber successfully changed the system identifier");
+ else
+ pg_log_error("subscriber failed to change system identifier: exit code: %d", rc);
+ }
+
+ pfree(cf);
+}
+
+/*
+ * Create the publications and replication slots in preparation for logical
+ * replication.
+ */
+static bool
+setup_publisher(LogicalRepInfo *dbinfo)
+{
+ PGconn *conn;
+ PGresult *res;
+
+ for (int i = 0; i < num_dbs; i++)
+ {
+ char pubname[NAMEDATALEN];
+ char replslotname[NAMEDATALEN];
+
+ conn = connect_database(dbinfo[i].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT oid FROM pg_catalog.pg_database WHERE datname = current_database()");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain database OID: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain database OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ return false;
+ }
+
+ /* Remember database OID. */
+ dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+
+ PQclear(res);
+
+ /*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 31 characters (20 + 10 +
+ * '\0').
+ */
+ snprintf(pubname, sizeof(pubname), "pg_createsubscriber_%u", dbinfo[i].oid);
+ dbinfo[i].pubname = pg_strdup(pubname);
+
+ /*
+ * Create publication on publisher. This step should be executed
+ * *before* promoting the subscriber to avoid any transactions between
+ * consistent LSN and the new publication rows (such transactions
+ * wouldn't see the new publication rows resulting in an error).
+ */
+ create_publication(conn, &dbinfo[i]);
+
+ /*
+ * Build the replication slot name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 42
+ * characters (20 + 10 + 1 + 10 + '\0'). PID is included to reduce the
+ * probability of collision. By default, subscription name is used as
+ * replication slot name.
+ */
+ snprintf(replslotname, sizeof(replslotname),
+ "pg_createsubscriber_%u_%d",
+ dbinfo[i].oid,
+ (int) getpid());
+ dbinfo[i].subname = pg_strdup(replslotname);
+
+ /* Create replication slot on publisher. */
+ if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
+ pg_log_info("create replication slot \"%s\" on publisher", replslotname);
+ else
+ return false;
+
+ disconnect_database(conn);
+ }
+
+ return true;
+}
+
+/*
+ * Is the primary server ready for logical replication?
+ */
+static bool
+check_publisher(LogicalRepInfo *dbinfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ PQExpBuffer str = createPQExpBuffer();
+
+ char *wal_level;
+ int max_repslots;
+ int cur_repslots;
+ int max_walsenders;
+ int cur_walsenders;
+
+ pg_log_info("checking settings on publisher");
+
+ /*
+ * Logical replication requires a few parameters to be set on publisher.
+ * Since these parameters are not a requirement for physical replication,
+ * we should check it to make sure it won't fail.
+ *
+ * wal_level = logical
+ * max_replication_slots >= current + number of dbs to be converted
+ * max_wal_senders >= current + number of dbs to be converted
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "WITH wl AS (SELECT setting AS wallevel FROM pg_settings WHERE name = 'wal_level'),"
+ " total_mrs AS (SELECT setting AS tmrs FROM pg_settings WHERE name = 'max_replication_slots'),"
+ " cur_mrs AS (SELECT count(*) AS cmrs FROM pg_replication_slots),"
+ " total_mws AS (SELECT setting AS tmws FROM pg_settings WHERE name = 'max_wal_senders'),"
+ " cur_mws AS (SELECT count(*) AS cmws FROM pg_stat_activity WHERE backend_type = 'walsender')"
+ "SELECT wallevel, tmrs, cmrs, tmws, cmws FROM wl, total_mrs, cur_mrs, total_mws, cur_mws");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publisher settings: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ wal_level = strdup(PQgetvalue(res, 0, 0));
+ max_repslots = atoi(PQgetvalue(res, 0, 1));
+ cur_repslots = atoi(PQgetvalue(res, 0, 2));
+ max_walsenders = atoi(PQgetvalue(res, 0, 3));
+ cur_walsenders = atoi(PQgetvalue(res, 0, 4));
+
+ PQclear(res);
+
+ pg_log_debug("subscriber: wal_level: %s", wal_level);
+ pg_log_debug("subscriber: max_replication_slots: %d", max_repslots);
+ pg_log_debug("subscriber: current replication slots: %d", cur_repslots);
+ pg_log_debug("subscriber: max_wal_senders: %d", max_walsenders);
+ pg_log_debug("subscriber: current wal senders: %d", cur_walsenders);
+
+ /*
+ * If standby sets primary_slot_name, check if this replication slot is in
+ * use on primary for WAL retention purposes. This replication slot has no
+ * use after the transformation, hence, it will be removed at the end of
+ * this process.
+ */
+ if (primary_slot_name)
+ {
+ appendPQExpBuffer(str,
+ "SELECT 1 FROM pg_replication_slots WHERE active AND slot_name = '%s'", primary_slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain replication slot information: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
+ PQntuples(res), 1);
+ pg_free(primary_slot_name); /* it is not being used. */
+ primary_slot_name = NULL;
+ return false;
+ }
+ else
+ {
+ pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+ }
+
+ PQclear(res);
+ }
+
+ disconnect_database(conn);
+
+ if (strcmp(wal_level, "logical") != 0)
+ {
+ pg_log_error("publisher requires wal_level >= logical");
+ return false;
+ }
+
+ if (max_repslots - cur_repslots < num_dbs)
+ {
+ pg_log_error("publisher requires %d replication slots, but only %d remain", num_dbs, max_repslots - cur_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", cur_repslots + num_dbs);
+ return false;
+ }
+
+ if (max_walsenders - cur_walsenders < num_dbs)
+ {
+ pg_log_error("publisher requires %d wal sender processes, but only %d remain", num_dbs, max_walsenders - cur_walsenders);
+ pg_log_error_hint("Consider increasing max_wal_senders to at least %d.", cur_walsenders + num_dbs);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Is the standby server ready for logical replication?
+ */
+static bool
+check_subscriber(LogicalRepInfo *dbinfo)
+{
+ PGconn *conn;
+ PGresult *res;
+
+ int max_lrworkers;
+ int max_repslots;
+ int max_wprocs;
+
+ pg_log_info("checking settings on subscriber");
+
+ /*
+ * Logical replication requires a few parameters to be set on subscriber.
+ * Since these parameters are not a requirement for physical replication,
+ * we should check it to make sure it won't fail.
+ *
+ * max_replication_slots >= number of dbs to be converted
+ * max_logical_replication_workers >= number of dbs to be converted
+ * max_worker_processes >= 1 + number of dbs to be converted
+ */
+ conn = connect_database(dbinfo[0].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn,
+ "SELECT setting FROM pg_settings WHERE name IN ('max_logical_replication_workers', 'max_replication_slots', 'max_worker_processes', 'primary_slot_name') ORDER BY name");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscriber settings: %s", PQresultErrorMessage(res));
+ return false;
+ }
+
+ max_lrworkers = atoi(PQgetvalue(res, 0, 0));
+ max_repslots = atoi(PQgetvalue(res, 1, 0));
+ max_wprocs = atoi(PQgetvalue(res, 2, 0));
+ if (strcmp(PQgetvalue(res, 3, 0), "") != 0)
+ primary_slot_name = pg_strdup(PQgetvalue(res, 3, 0));
+
+ pg_log_debug("subscriber: max_logical_replication_workers: %d", max_lrworkers);
+ pg_log_debug("subscriber: max_replication_slots: %d", max_repslots);
+ pg_log_debug("subscriber: max_worker_processes: %d", max_wprocs);
+ pg_log_debug("subscriber: primary_slot_name: %s", primary_slot_name);
+
+ PQclear(res);
+
+ disconnect_database(conn);
+
+ if (max_repslots < num_dbs)
+ {
+ pg_log_error("subscriber requires %d replication slots, but only %d remain", num_dbs, max_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", num_dbs);
+ return false;
+ }
+
+ if (max_lrworkers < num_dbs)
+ {
+ pg_log_error("subscriber requires %d logical replication workers, but only %d remain", num_dbs, max_lrworkers);
+ pg_log_error_hint("Consider increasing max_logical_replication_workers to at least %d.", num_dbs);
+ return false;
+ }
+
+ if (max_wprocs < num_dbs + 1)
+ {
+ pg_log_error("subscriber requires %d worker processes, but only %d remain", num_dbs + 1, max_wprocs);
+ pg_log_error_hint("Consider increasing max_worker_processes to at least %d.", num_dbs + 1);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Create the subscriptions, adjust the initial location for logical replication and
+ * enable the subscriptions. That's the last step for logical repliation setup.
+ */
+static bool
+setup_subscriber(LogicalRepInfo *dbinfo, const char *consistent_lsn)
+{
+ PGconn *conn;
+
+ for (int i = 0; i < num_dbs; i++)
+ {
+ /* Connect to subscriber. */
+ conn = connect_database(dbinfo[i].subconninfo);
+ if (conn == NULL)
+ exit(1);
+
+ /*
+ * Since the publication was created before the consistent LSN, it is
+ * available on the subscriber when the physical replica is promoted.
+ * Remove publications from the subscriber because it has no use.
+ */
+ drop_publication(conn, &dbinfo[i]);
+
+ create_subscription(conn, &dbinfo[i]);
+
+ /* Set the replication progress to the correct LSN. */
+ set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+
+ /* Enable subscription. */
+ enable_subscription(conn, &dbinfo[i]);
+
+ disconnect_database(conn);
+ }
+
+ return true;
+}
+
+/*
+ * Create a logical replication slot and returns a consistent LSN. The returned
+ * LSN might be used to catch up the subscriber up to the required point.
+ *
+ * CreateReplicationSlot() is not used because it does not provide the one-row
+ * result set that contains the consistent LSN.
+ */
+static char *
+create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
+ char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res = NULL;
+ char *lsn = NULL;
+ bool transient_replslot = false;
+
+ Assert(conn != NULL);
+
+ /*
+ * If no slot name is informed, it is a transient replication slot used
+ * only for catch up purposes.
+ */
+ if (slot_name[0] == '\0')
+ {
+ snprintf(slot_name, NAMEDATALEN, "pg_createsubscriber_%d_startpoint",
+ (int) getpid());
+ transient_replslot = true;
+ }
+
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
+ if (transient_replslot)
+ appendPQExpBufferStr(str, " TEMPORARY");
+ appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQresultErrorMessage(res));
+ return lsn;
+ }
+ }
+
+ /* for cleanup purposes */
+ if (!transient_replslot)
+ dbinfo->made_replslot = true;
+
+ if (!dry_run)
+ {
+ lsn = pg_strdup(PQgetvalue(res, 0, 1));
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+
+ return lsn;
+}
+
+static void
+drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+static char *
+server_logfile_name(const char *datadir)
+{
+ char timebuf[128];
+ struct timeval time;
+ time_t tt;
+ int len;
+ char *filename;
+
+ /* append timestamp with ISO 8601 format. */
+ gettimeofday(&time, NULL);
+ tt = (time_t) time.tv_sec;
+ strftime(timebuf, sizeof(timebuf), "%Y%m%dT%H%M%S", localtime(&tt));
+ snprintf(timebuf + strlen(timebuf), sizeof(timebuf) - strlen(timebuf),
+ ".%03d", (int) (time.tv_usec / 1000));
+
+ filename = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(filename, MAXPGPATH, "%s/%s/server_start_%s.log", datadir, PGS_OUTPUT_DIR, timebuf);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("log file path is too long");
+ exit(1);
+ }
+
+ return filename;
+}
+
+static void
+start_standby_server(const char *pg_ctl_path, const char *datadir, const char *logfile)
+{
+ char *pg_ctl_cmd;
+ int rc;
+
+ pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, datadir, logfile);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 1);
+}
+
+static void
+stop_standby_server(const char *pg_ctl_path, const char *datadir)
+{
+ char *pg_ctl_cmd;
+ int rc;
+
+ pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, datadir);
+ rc = system(pg_ctl_cmd);
+ pg_ctl_status(pg_ctl_cmd, rc, 0);
+}
+
+/*
+ * Reports a suitable message if pg_ctl fails.
+ */
+static void
+pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
+{
+ if (rc != 0)
+ {
+ if (WIFEXITED(rc))
+ {
+ pg_log_error("pg_ctl failed with exit code %d", WEXITSTATUS(rc));
+ }
+ else if (WIFSIGNALED(rc))
+ {
+#if defined(WIN32)
+ pg_log_error("pg_ctl was terminated by exception 0x%X", WTERMSIG(rc));
+ pg_log_error_detail("See C include file \"ntstatus.h\" for a description of the hexadecimal value.");
+#else
+ pg_log_error("pg_ctl was terminated by signal %d: %s",
+ WTERMSIG(rc), pg_strsignal(WTERMSIG(rc)));
+#endif
+ }
+ else
+ {
+ pg_log_error("pg_ctl exited with unrecognized status %d", rc);
+ }
+
+ pg_log_error_detail("The failed command was: %s", pg_ctl_cmd);
+ exit(1);
+ }
+
+ if (action)
+ pg_log_info("postmaster was started");
+ else
+ pg_log_info("postmaster was stopped");
+}
+
+/*
+ * Returns after the server finishes the recovery process.
+ *
+ * If recovery_timeout option is set, terminate abnormally without finishing
+ * the recovery process. By default, it waits forever.
+ */
+static void
+wait_for_end_recovery(const char *conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ int status = POSTMASTER_STILL_STARTING;
+ int timer = 0;
+
+ pg_log_info("waiting the postmaster to reach the consistent state");
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ for (;;)
+ {
+ bool in_recovery;
+
+ res = PQexec(conn, "SELECT pg_catalog.pg_is_in_recovery()");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain recovery progress");
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ pg_log_error("unexpected result from pg_is_in_recovery function");
+ exit(1);
+ }
+
+ in_recovery = (strcmp(PQgetvalue(res, 0, 0), "t") == 0);
+
+ PQclear(res);
+
+ /*
+ * Does the recovery process finish? In dry run mode, there is no
+ * recovery mode. Bail out as the recovery process has ended.
+ */
+ if (!in_recovery || dry_run)
+ {
+ status = POSTMASTER_READY;
+ break;
+ }
+
+ /*
+ * Bail out after recovery_timeout seconds if this option is set.
+ */
+ if (recovery_timeout > 0 && timer >= recovery_timeout)
+ {
+ pg_log_error("recovery timed out");
+ stop_standby_server(pg_ctl_path, subscriber_dir);
+ exit(1);
+ }
+
+ /* Keep waiting. */
+ pg_usleep(WAIT_INTERVAL * USEC_PER_SEC);
+
+ timer += WAIT_INTERVAL;
+ }
+
+ disconnect_database(conn);
+
+ if (status == POSTMASTER_STILL_STARTING)
+ {
+ pg_log_error("server did not end recovery");
+ exit(1);
+ }
+
+ pg_log_info("postmaster reached the consistent state");
+}
+
+/*
+ * Create a publication that includes all tables in the database.
+ */
+static void
+create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ /* Check if the publication needs to be created. */
+ appendPQExpBuffer(str,
+ "SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
+ dbinfo->pubname);
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain publication information: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) == 1)
+ {
+ /*
+ * If publication name already exists and puballtables is true, let's
+ * use it. A previous run of pg_createsubscriber must have created
+ * this publication. Bail out.
+ */
+ if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
+ {
+ pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ return;
+ }
+ else
+ {
+ /*
+ * Unfortunately, if it reaches this code path, it will always
+ * fail (unless you decide to change the existing publication
+ * name). That's bad but it is very unlikely that the user will
+ * choose a name with pg_createsubscriber_ prefix followed by the
+ * exact database oid in which puballtables is false.
+ */
+ pg_log_error("publication \"%s\" does not replicate changes for all tables",
+ dbinfo->pubname);
+ pg_log_error_hint("Consider renaming this publication.");
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(str);
+
+ pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
+ dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_publication = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove publication if it couldn't finish all steps.
+ */
+static void
+drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Create a subscription with some predefined options.
+ *
+ * A replication slot was already created in a previous step. Let's use it. By
+ * default, the subscription name is used as replication slot name. It is
+ * not required to copy data. The subscription will be created but it will not
+ * be enabled now. That's because the replication progress must be set and the
+ * replication origin name (one of the function arguments) contains the
+ * subscription OID in its name. Once the subscription is created,
+ * set_replication_progress() can obtain the chosen origin name and set up its
+ * initial location.
+ */
+static void
+create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str,
+ "CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
+ "WITH (create_slot = false, copy_data = false, enabled = false)",
+ dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
+ dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+ }
+
+ /* for cleanup purposes */
+ dbinfo->made_subscription = true;
+
+ if (!dry_run)
+ PQclear(res);
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Remove subscription if it couldn't finish all steps.
+ */
+static void
+drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Sets the replication progress to the consistent LSN.
+ *
+ * The subscriber caught up to the consistent LSN provided by the temporary
+ * replication slot. The goal is to set up the initial location for the logical
+ * replication that is the exact LSN that the subscriber was promoted. Once the
+ * subscription is enabled it will start streaming from that location onwards.
+ * In dry run mode, the subscription OID and LSN are set to invalid values for
+ * printing purposes.
+ */
+static void
+set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+ Oid suboid;
+ char originname[NAMEDATALEN];
+ char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+
+ Assert(conn != NULL);
+
+ appendPQExpBuffer(str,
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not obtain subscription OID: %s",
+ PQresultErrorMessage(res));
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (PQntuples(res) != 1 && !dry_run)
+ {
+ pg_log_error("could not obtain subscription OID: got %d rows, expected %d rows",
+ PQntuples(res), 1);
+ PQclear(res);
+ PQfinish(conn);
+ exit(1);
+ }
+
+ if (dry_run)
+ {
+ suboid = InvalidOid;
+ snprintf(lsnstr, sizeof(lsnstr), "%X/%X", LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ suboid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ snprintf(lsnstr, sizeof(lsnstr), "%s", lsn);
+ }
+
+ /*
+ * The origin name is defined as pg_%u. %u is the subscription OID. See
+ * ApplyWorkerMain().
+ */
+ snprintf(originname, sizeof(originname), "pg_%u", suboid);
+
+ PQclear(res);
+
+ pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
+ originname, lsnstr, dbinfo->dbname);
+
+ resetPQExpBuffer(str);
+ appendPQExpBuffer(str,
+ "SELECT pg_catalog.pg_replication_origin_advance('%s', '%s')", originname, lsnstr);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not set replication progress for the subscription \"%s\": %s",
+ dbinfo->subname, PQresultErrorMessage(res));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+/*
+ * Enables the subscription.
+ *
+ * The subscription was created in a previous step but it was disabled. After
+ * adjusting the initial location, enabling the subscription is the last step
+ * of this setup.
+ */
+static void
+enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+{
+ PQExpBuffer str = createPQExpBuffer();
+ PGresult *res;
+
+ Assert(conn != NULL);
+
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+
+ pg_log_debug("command is: %s", str->data);
+
+ if (!dry_run)
+ {
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
+ }
+
+ PQclear(res);
+ }
+
+ destroyPQExpBuffer(str);
+}
+
+int
+main(int argc, char **argv)
+{
+ static struct option long_options[] =
+ {
+ {"help", no_argument, NULL, '?'},
+ {"version", no_argument, NULL, 'V'},
+ {"pgdata", required_argument, NULL, 'D'},
+ {"publisher-server", required_argument, NULL, 'P'},
+ {"subscriber-server", required_argument, NULL, 'S'},
+ {"database", required_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"recovery-timeout", required_argument, NULL, 't'},
+ {"retain", no_argument, NULL, 'r'},
+ {"verbose", no_argument, NULL, 'v'},
+ {NULL, 0, NULL, 0}
+ };
+
+ int c;
+ int option_index;
+
+ char *base_dir;
+ char *server_start_log;
+ int len;
+
+ char *pub_base_conninfo = NULL;
+ char *sub_base_conninfo = NULL;
+ char *dbname_conninfo = NULL;
+ char temp_replslot[NAMEDATALEN] = {0};
+
+ uint64 pub_sysid;
+ uint64 sub_sysid;
+ struct stat statbuf;
+
+ PGconn *conn;
+ char *consistent_lsn;
+
+ PQExpBuffer recoveryconfcontents = NULL;
+
+ char pidfile[MAXPGPATH];
+
+ pg_logging_init(argv[0]);
+ pg_logging_set_level(PG_LOG_WARNING);
+ progname = get_progname(argv[0]);
+ set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_createsubscriber"));
+
+ if (argc > 1)
+ {
+ if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+ {
+ usage();
+ exit(0);
+ }
+ else if (strcmp(argv[1], "-V") == 0
+ || strcmp(argv[1], "--version") == 0)
+ {
+ puts("pg_createsubscriber (PostgreSQL) " PG_VERSION);
+ exit(0);
+ }
+ }
+
+ /*
+ * Don't allow it to be run as root. It uses pg_ctl which does not allow
+ * it either.
+ */
+#ifndef WIN32
+ if (geteuid() == 0)
+ {
+ pg_log_error("cannot be executed by \"root\"");
+ pg_log_error_hint("You must run %s as the PostgreSQL superuser.",
+ progname);
+ exit(1);
+ }
+#endif
+
+ while ((c = getopt_long(argc, argv, "D:P:S:d:nrt:v",
+ long_options, &option_index)) != -1)
+ {
+ switch (c)
+ {
+ case 'D':
+ subscriber_dir = pg_strdup(optarg);
+ break;
+ case 'P':
+ pub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'S':
+ sub_conninfo_str = pg_strdup(optarg);
+ break;
+ case 'd':
+ /* Ignore duplicated database names. */
+ if (!simple_string_list_member(&database_names, optarg))
+ {
+ simple_string_list_append(&database_names, optarg);
+ num_dbs++;
+ }
+ break;
+ case 'n':
+ dry_run = true;
+ break;
+ case 'r':
+ retain = true;
+ break;
+ case 't':
+ recovery_timeout = atoi(optarg);
+ break;
+ case 'v':
+ pg_logging_increase_verbosity();
+ break;
+ default:
+ /* getopt_long already emitted a complaint */
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Any non-option arguments?
+ */
+ if (optind < argc)
+ {
+ pg_log_error("too many command-line arguments (first is \"%s\")",
+ argv[optind]);
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Required arguments
+ */
+ if (subscriber_dir == NULL)
+ {
+ pg_log_error("no subscriber data directory specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Parse connection string. Build a base connection string that might be
+ * reused by multiple databases.
+ */
+ if (pub_conninfo_str == NULL)
+ {
+ /*
+ * TODO use primary_conninfo (if available) from subscriber and
+ * extract publisher connection string. Assume that there are
+ * identical entries for physical and logical replication. If there is
+ * not, we would fail anyway.
+ */
+ pg_log_error("no publisher connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
+ "publisher");
+ if (pub_base_conninfo == NULL)
+ exit(1);
+
+ if (sub_conninfo_str == NULL)
+ {
+ pg_log_error("no subscriber connection string specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
+ if (sub_base_conninfo == NULL)
+ exit(1);
+
+ if (database_names.head == NULL)
+ {
+ pg_log_info("no database was specified");
+
+ /*
+ * If --database option is not provided, try to obtain the dbname from
+ * the publisher conninfo. If dbname parameter is not available, error
+ * out.
+ */
+ if (dbname_conninfo)
+ {
+ simple_string_list_append(&database_names, dbname_conninfo);
+ num_dbs++;
+
+ pg_log_info("database \"%s\" was extracted from the publisher connection string",
+ dbname_conninfo);
+ }
+ else
+ {
+ pg_log_error("no database name specified");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
+ */
+ if (!get_exec_path(argv[0]))
+ exit(1);
+
+ /* rudimentary check for a data directory. */
+ if (!check_data_directory(subscriber_dir))
+ exit(1);
+
+ /* Store database information for publisher and subscriber. */
+ dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+
+ /* Register a function to clean up objects in case of failure. */
+ atexit(cleanup_objects_atexit);
+
+ /*
+ * Check if the subscriber data directory has the same system identifier
+ * than the publisher data directory.
+ */
+ pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
+ sub_sysid = get_control_from_datadir(subscriber_dir);
+ if (pub_sysid != sub_sysid)
+ {
+ pg_log_error("subscriber data directory is not a copy of the source database cluster");
+ exit(1);
+ }
+
+ /*
+ * Create the output directory to store any data generated by this tool.
+ */
+ base_dir = (char *) pg_malloc0(MAXPGPATH);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ if (len >= MAXPGPATH)
+ {
+ pg_log_error("directory path for subscriber is too long");
+ exit(1);
+ }
+
+ if (mkdir(base_dir, pg_dir_create_mode) < 0 && errno != EEXIST)
+ {
+ pg_log_error("could not create directory \"%s\": %m", base_dir);
+ exit(1);
+ }
+
+ server_start_log = server_logfile_name(subscriber_dir);
+
+ /* subscriber PID file. */
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+
+ /*
+ * The standby server must be running. That's because some checks will be
+ * done (is it ready for a logical replication setup?). After that, stop
+ * the subscriber in preparation to modify some recovery parameters that
+ * require a restart.
+ */
+ if (stat(pidfile, &statbuf) == 0)
+ {
+ /*
+ * Check if the standby server is ready for logical replication.
+ */
+ if (!check_subscriber(dbinfo))
+ exit(1);
+
+ /*
+ * Check if the primary server is ready for logical replication. This
+ * routine checks if a replication slot is in use on primary so it
+ * relies on check_subscriber() to obtain the primary_slot_name.
+ * That's why it is called after it.
+ */
+ if (!check_publisher(dbinfo))
+ exit(1);
+
+ /*
+ * Create the required objects for each database on publisher. This
+ * step is here mainly because if we stop the standby we cannot verify
+ * if the primary slot is in use. We could use an extra connection for
+ * it but it doesn't seem worth.
+ */
+ if (!setup_publisher(dbinfo))
+ exit(1);
+
+ /* Stop the standby server. */
+ pg_log_info("standby is up and running");
+ pg_log_info("stopping the server to start the transformation steps");
+ stop_standby_server(pg_ctl_path, subscriber_dir);
+ }
+ else
+ {
+ pg_log_error("standby is not running");
+ pg_log_error_hint("Start the standby and try again.");
+ exit(1);
+ }
+
+ /*
+ * Create a temporary logical replication slot to get a consistent LSN.
+ *
+ * This consistent LSN will be used later to advanced the recently created
+ * replication slots. It is ok to use a temporary replication slot here
+ * because it will have a short lifetime and it is only used as a mark to
+ * start the logical replication.
+ *
+ * XXX we should probably use the last created replication slot to get a
+ * consistent LSN but it should be changed after adding pg_basebackup
+ * support.
+ */
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn == NULL)
+ exit(1);
+ consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
+ temp_replslot);
+
+ /*
+ * Write recovery parameters.
+ *
+ * Despite of the recovery parameters will be written to the subscriber,
+ * use a publisher connection for the follwing recovery functions. The
+ * connection is only used to check the current server version (physical
+ * replica, same server version). The subscriber is not running yet. In
+ * dry run mode, the recovery parameters *won't* be written. An invalid
+ * LSN is used for printing purposes.
+ */
+ recoveryconfcontents = GenerateRecoveryConfig(conn, NULL);
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_inclusive = true\n");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_action = promote\n");
+
+ if (dry_run)
+ {
+ appendPQExpBuffer(recoveryconfcontents, "# dry run mode");
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%X/%X'\n",
+ LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
+ }
+ else
+ {
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
+ consistent_lsn);
+ WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ }
+ disconnect_database(conn);
+
+ pg_log_debug("recovery parameters:\n%s", recoveryconfcontents->data);
+
+ /*
+ * Start subscriber and wait until accepting connections.
+ */
+ pg_log_info("starting the subscriber");
+ start_standby_server(pg_ctl_path, subscriber_dir, server_start_log);
+
+ /*
+ * Waiting the subscriber to be promoted.
+ */
+ wait_for_end_recovery(dbinfo[0].subconninfo);
+
+ /*
+ * Create the subscription for each database on subscriber. It does not
+ * enable it immediately because it needs to adjust the logical
+ * replication start point to the LSN reported by consistent_lsn (see
+ * set_replication_progress). It also cleans up publications created by
+ * this tool and replication to the standby.
+ */
+ if (!setup_subscriber(dbinfo, consistent_lsn))
+ exit(1);
+
+ /*
+ * If the primary_slot_name exists on primary, drop it.
+ *
+ * XXX we might not fail here. Instead, we provide a warning so the user
+ * eventually drops this replication slot later.
+ */
+ if (primary_slot_name != NULL)
+ {
+ conn = connect_database(dbinfo[0].pubconninfo);
+ if (conn != NULL)
+ {
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ }
+ else
+ {
+ pg_log_warning("could not drop replication slot \"%s\" on primary", primary_slot_name);
+ pg_log_warning_hint("Drop this replication slot soon to avoid retention of WAL files.");
+ }
+ disconnect_database(conn);
+ }
+
+ /*
+ * Stop the subscriber.
+ */
+ pg_log_info("stopping the subscriber");
+ stop_standby_server(pg_ctl_path, subscriber_dir);
+
+ /*
+ * Change system identifier.
+ */
+ modify_sysid(pg_resetwal_path, subscriber_dir);
+
+ /*
+ * The log file is kept if retain option is specified or this tool does
+ * not run successfully. Otherwise, log file is removed.
+ */
+ if (!retain)
+ unlink(server_start_log);
+
+ success = true;
+
+ pg_log_info("Done!");
+
+ return 0;
+}
diff --git a/src/bin/pg_basebackup/t/040_pg_createsubscriber.pl b/src/bin/pg_basebackup/t/040_pg_createsubscriber.pl
new file mode 100644
index 0000000000..0f02b1bfac
--- /dev/null
+++ b/src/bin/pg_basebackup/t/040_pg_createsubscriber.pl
@@ -0,0 +1,44 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test checking options of pg_createsubscriber.
+#
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+program_help_ok('pg_createsubscriber');
+program_version_ok('pg_createsubscriber');
+program_options_handling_ok('pg_createsubscriber');
+
+my $datadir = PostgreSQL::Test::Utils::tempdir;
+
+command_fails(['pg_createsubscriber'],
+ 'no subscriber data directory specified');
+command_fails(
+ [
+ 'pg_createsubscriber',
+ '--pgdata', $datadir
+ ],
+ 'no publisher connection string specified');
+command_fails(
+ [
+ 'pg_createsubscriber',
+ '--dry-run',
+ '--pgdata', $datadir,
+ '--publisher-server', 'dbname=postgres'
+ ],
+ 'no subscriber connection string specified');
+command_fails(
+ [
+ 'pg_createsubscriber',
+ '--verbose',
+ '--pgdata', $datadir,
+ '--publisher-server', 'dbname=postgres',
+ '--subscriber-server', 'dbname=postgres'
+ ],
+ 'no database name specified');
+
+done_testing();
diff --git a/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
new file mode 100644
index 0000000000..534bc53a76
--- /dev/null
+++ b/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
@@ -0,0 +1,139 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+#
+# Test using a standby server as the subscriber.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node_p;
+my $node_f;
+my $node_s;
+my $result;
+
+# Set up node P as primary
+$node_p = PostgreSQL::Test::Cluster->new('node_p');
+$node_p->init(allows_streaming => 'logical');
+$node_p->start;
+
+# Set up node F as about-to-fail node
+# The extra option forces it to initialize a new cluster instead of copying a
+# previously initdb's cluster.
+$node_f = PostgreSQL::Test::Cluster->new('node_f');
+$node_f->init(allows_streaming => 'logical', extra => [ '--no-instructions' ]);
+$node_f->start;
+
+# On node P
+# - create databases
+# - create test tables
+# - insert a row
+$node_p->safe_psql(
+ 'postgres', q(
+ CREATE DATABASE pg1;
+ CREATE DATABASE pg2;
+));
+$node_p->safe_psql('pg1', 'CREATE TABLE tbl1 (a text)');
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('first row')");
+$node_p->safe_psql('pg2', 'CREATE TABLE tbl2 (a text)');
+
+# Set up node S as standby linking to node P
+$node_p->backup('backup_1');
+$node_s = PostgreSQL::Test::Cluster->new('node_s');
+$node_s->init_from_backup($node_p, 'backup_1', has_streaming => 1);
+$node_s->append_conf('postgresql.conf', 'log_min_messages = debug2');
+$node_s->set_standby_mode();
+$node_s->start;
+
+# Insert another row on node P and wait node S to catch up
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('second row')");
+$node_p->wait_for_replay_catchup($node_s);
+
+# Run pg_createsubscriber on about-to-fail node F
+command_fails(
+ [
+ 'pg_createsubscriber', '--verbose',
+ '--pgdata', $node_f->data_dir,
+ '--publisher-server', $node_p->connstr('pg1'),
+ '--subscriber-server', $node_f->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'subscriber data directory is not a copy of the source database cluster');
+
+# dry run mode on node S
+command_ok(
+ [
+ 'pg_createsubscriber', '--verbose', '--dry-run',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-server', $node_p->connstr('pg1'),
+ '--subscriber-server', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_createsubscriber --dry-run on node S');
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+# Check if node S is still a standby
+is($node_s->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
+ 't', 'standby is in recovery');
+
+# Run pg_createsubscriber on node S
+command_ok(
+ [
+ 'pg_createsubscriber', '--verbose',
+ '--pgdata', $node_s->data_dir,
+ '--publisher-server', $node_p->connstr('pg1'),
+ '--subscriber-server', $node_s->connstr('pg1'),
+ '--database', 'pg1',
+ '--database', 'pg2'
+ ],
+ 'run pg_createsubscriber on node S');
+
+# Insert rows on P
+$node_p->safe_psql('pg1', "INSERT INTO tbl1 VALUES('third row')");
+$node_p->safe_psql('pg2', "INSERT INTO tbl2 VALUES('row 1')");
+
+# PID sets to undefined because subscriber was stopped behind the scenes.
+# Start subscriber
+$node_s->{_pid} = undef;
+$node_s->start;
+
+# Get subscription names
+$result = $node_s->safe_psql(
+ 'postgres', qq(
+ SELECT subname FROM pg_subscription WHERE subname ~ '^pg_createsubscriber_'
+));
+my @subnames = split("\n", $result);
+
+# Wait subscriber to catch up
+$node_s->wait_for_subscription_sync($node_p, $subnames[0]);
+$node_s->wait_for_subscription_sync($node_p, $subnames[1]);
+
+# Check result on database pg1
+$result = $node_s->safe_psql('pg1', 'SELECT * FROM tbl1');
+is( $result, qq(first row
+second row
+third row),
+ 'logical replication works on database pg1');
+
+# Check result on database pg2
+$result = $node_s->safe_psql('pg2', 'SELECT * FROM tbl2');
+is( $result, qq(row 1),
+ 'logical replication works on database pg2');
+
+# Different system identifier?
+my $sysid_p = $node_p->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+my $sysid_s = $node_s->safe_psql('postgres', 'SELECT system_identifier FROM pg_control_system()');
+ok($sysid_p != $sysid_s, 'system identifier was changed');
+
+# clean up
+$node_p->teardown_node;
+$node_s->teardown_node;
+
+done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 91433d439b..f51f1ff23f 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1505,6 +1505,7 @@ LogicalRepBeginData
LogicalRepCommitData
LogicalRepCommitPreparedTxnData
LogicalRepCtxStruct
+LogicalRepInfo
LogicalRepMsgType
LogicalRepPartMapEntry
LogicalRepPreparedTxnData
--
2.43.0
[application/octet-stream] v12-0002-Avoid-to-use-replication-connections.patch (4.1K, ../../TY3PR01MB9889320DDAF153FC8A532630F57C2@TY3PR01MB9889.jpnprd01.prod.outlook.com/3-v12-0002-Avoid-to-use-replication-connections.patch)
download | inline diff:
From 36e626128b40a03ab7c50702fa91cb8acdd90e2f Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <[email protected]>
Date: Wed, 31 Jan 2024 07:39:35 +0000
Subject: [PATCH v12 2/5] Avoid to use replication connections
---
doc/src/sgml/ref/pg_createsubscriber.sgml | 1 -
src/bin/pg_basebackup/pg_createsubscriber.c | 27 +++++++++------------
2 files changed, 11 insertions(+), 17 deletions(-)
diff --git a/doc/src/sgml/ref/pg_createsubscriber.sgml b/doc/src/sgml/ref/pg_createsubscriber.sgml
index 1c78ff92e0..53b42e6161 100644
--- a/doc/src/sgml/ref/pg_createsubscriber.sgml
+++ b/doc/src/sgml/ref/pg_createsubscriber.sgml
@@ -61,7 +61,6 @@ PostgreSQL documentation
The <application>pg_createsubscriber</application> should be run at the target
server. The source server (known as publisher server) should accept logical
replication connections from the target server (known as subscriber server).
- The target server should accept local logical replication connection.
</para>
</refsect1>
diff --git a/src/bin/pg_basebackup/pg_createsubscriber.c b/src/bin/pg_basebackup/pg_createsubscriber.c
index 478560b3e4..47970b10d6 100644
--- a/src/bin/pg_basebackup/pg_createsubscriber.c
+++ b/src/bin/pg_basebackup/pg_createsubscriber.c
@@ -397,12 +397,8 @@ connect_database(const char *conninfo)
{
PGconn *conn;
PGresult *res;
- const char *rconninfo;
- /* logical replication mode */
- rconninfo = psprintf("%s replication=database", conninfo);
-
- conn = PQconnectdb(rconninfo);
+ conn = PQconnectdb(conninfo);
if (PQstatus(conn) != CONNECTION_OK)
{
pg_log_error("connection to database failed: %s", PQerrorMessage(conn));
@@ -446,26 +442,26 @@ get_sysid_from_conn(const char *conninfo)
if (conn == NULL)
exit(1);
- res = PQexec(conn, "IDENTIFY_SYSTEM");
+ res = PQexec(conn, "SELECT * FROM pg_control_system();");
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
- pg_log_error("could not send replication command \"%s\": %s",
+ pg_log_error("could not send command \"%s\": %s",
"IDENTIFY_SYSTEM", PQresultErrorMessage(res));
PQclear(res);
disconnect_database(conn);
exit(1);
}
- if (PQntuples(res) != 1 || PQnfields(res) < 3)
+ if (PQntuples(res) != 1 || PQnfields(res) < 4)
{
pg_log_error("could not identify system: got %d rows and %d fields, expected %d rows and %d or more fields",
- PQntuples(res), PQnfields(res), 1, 3);
+ PQntuples(res), PQnfields(res), 1, 4);
PQclear(res);
disconnect_database(conn);
exit(1);
}
- sysid = strtou64(PQgetvalue(res, 0, 0), NULL, 10);
+ sysid = strtou64(PQgetvalue(res, 0, 2), NULL, 10);
pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
@@ -477,7 +473,7 @@ get_sysid_from_conn(const char *conninfo)
/*
* Obtain the system identifier from control file. It will be used to compare
* if a data directory is a clone of another one. This routine is used locally
- * and avoids a replication connection.
+ * and avoids a connection establishment.
*/
static uint64
get_control_from_datadir(const char *datadir)
@@ -905,10 +901,8 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
- appendPQExpBuffer(str, "CREATE_REPLICATION_SLOT \"%s\"", slot_name);
- if (transient_replslot)
- appendPQExpBufferStr(str, " TEMPORARY");
- appendPQExpBufferStr(str, " LOGICAL \"pgoutput\" NOEXPORT_SNAPSHOT");
+ appendPQExpBuffer(str, "SELECT * FROM pg_create_logical_replication_slot('%s', 'pgoutput', %s, false, false);",
+ slot_name, transient_replslot ? "true" : "false");
pg_log_debug("command is: %s", str->data);
@@ -948,7 +942,8 @@ drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_nam
pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
- appendPQExpBuffer(str, "DROP_REPLICATION_SLOT \"%s\"", slot_name);
+ appendPQExpBuffer(str, "SELECT * FROM pg_drop_replication_slot('%s');",
+ slot_name);
pg_log_debug("command is: %s", str->data);
--
2.43.0
[application/octet-stream] v12-0003-Remove-P-and-use-primary_conninfo-instead.patch (11.7K, ../../TY3PR01MB9889320DDAF153FC8A532630F57C2@TY3PR01MB9889.jpnprd01.prod.outlook.com/4-v12-0003-Remove-P-and-use-primary_conninfo-instead.patch)
download | inline diff:
From 1315038f73179760000137cd34575645dc8fe86a Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <[email protected]>
Date: Wed, 31 Jan 2024 09:20:54 +0000
Subject: [PATCH v12 3/5] Remove -P and use primary_conninfo instead
XXX: This may be a problematic when the OS user who started target instance is
not the current OS user and PGPASSWORD environment variable was used for
connecting to the primary server. In this case, the password would not be
written in the primary_conninfo and the PGPASSWORD variable might not be set.
This may lead an connection error. Is this a real issue? Note that using
PGPASSWORD is not recommended.
---
doc/src/sgml/ref/pg_createsubscriber.sgml | 17 +---
src/bin/pg_basebackup/pg_createsubscriber.c | 98 ++++++++++++-------
.../t/040_pg_createsubscriber.pl | 8 --
.../t/041_pg_createsubscriber_standby.pl | 5 +-
4 files changed, 65 insertions(+), 63 deletions(-)
diff --git a/doc/src/sgml/ref/pg_createsubscriber.sgml b/doc/src/sgml/ref/pg_createsubscriber.sgml
index 53b42e6161..0abe1f6f28 100644
--- a/doc/src/sgml/ref/pg_createsubscriber.sgml
+++ b/doc/src/sgml/ref/pg_createsubscriber.sgml
@@ -29,11 +29,6 @@ PostgreSQL documentation
<arg choice="plain"><option>--pgdata</option></arg>
</group>
<replaceable>datadir</replaceable>
- <group choice="req">
- <arg choice="plain"><option>-P</option></arg>
- <arg choice="plain"><option>--publisher-server</option></arg>
- </group>
- <replaceable>connstr</replaceable>
<group choice="req">
<arg choice="plain"><option>-S</option></arg>
<arg choice="plain"><option>--subscriber-server</option></arg>
@@ -83,16 +78,6 @@ PostgreSQL documentation
</listitem>
</varlistentry>
- <varlistentry>
- <term><option>-P <replaceable class="parameter">connstr</replaceable></option></term>
- <term><option>--publisher-server=<replaceable class="parameter">connstr</replaceable></option></term>
- <listitem>
- <para>
- The connection string to the publisher. For details see <xref linkend="libpq-connstring"/>.
- </para>
- </listitem>
- </varlistentry>
-
<varlistentry>
<term><option>-S <replaceable class="parameter">connstr</replaceable></option></term>
<term><option>--subscriber-server=<replaceable class="parameter">connstr</replaceable></option></term>
@@ -304,7 +289,7 @@ PostgreSQL documentation
To create a logical replica for databases <literal>hr</literal> and
<literal>finance</literal> from a physical replica at <literal>foo</literal>:
<screen>
-<prompt>$</prompt> <userinput>pg_createsubscriber -D /usr/local/pgsql/data -P "host=foo" -S "host=localhost" -d hr -d finance</userinput>
+<prompt>$</prompt> <userinput>pg_createsubscriber -D /usr/local/pgsql/data -S "host=localhost" -d hr -d finance</userinput>
</screen>
</para>
diff --git a/src/bin/pg_basebackup/pg_createsubscriber.c b/src/bin/pg_basebackup/pg_createsubscriber.c
index 47970b10d6..f0e9db7793 100644
--- a/src/bin/pg_basebackup/pg_createsubscriber.c
+++ b/src/bin/pg_basebackup/pg_createsubscriber.c
@@ -51,10 +51,10 @@ typedef struct LogicalRepInfo
static void cleanup_objects_atexit(void);
static void usage();
-static char *get_base_conninfo(char *conninfo, char *dbname,
- const char *noderole);
+static char *get_base_conninfo(char *conninfo, char *dbname);
static bool get_exec_path(const char *path);
static bool check_data_directory(const char *datadir);
+static char *get_primary_conninfo(const char *base_conninfo);
static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
static PGconn *connect_database(const char *conninfo);
@@ -88,7 +88,6 @@ static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
static const char *progname;
static char *subscriber_dir = NULL;
-static char *pub_conninfo_str = NULL;
static char *sub_conninfo_str = NULL;
static SimpleStringList database_names = {NULL, NULL};
static char *primary_slot_name = NULL;
@@ -167,7 +166,6 @@ usage(void)
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nOptions:\n"));
printf(_(" -D, --pgdata=DATADIR location for the subscriber data directory\n"));
- printf(_(" -P, --publisher-server=CONNSTR publisher connection string\n"));
printf(_(" -S, --subscriber-server=CONNSTR subscriber connection string\n"));
printf(_(" -d, --database=DBNAME database to create a subscription\n"));
printf(_(" -n, --dry-run stop before modifying anything\n"));
@@ -192,7 +190,7 @@ usage(void)
* dbname.
*/
static char *
-get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
+get_base_conninfo(char *conninfo, char *dbname)
{
PQExpBuffer buf = createPQExpBuffer();
PQconninfoOption *conn_opts = NULL;
@@ -201,7 +199,7 @@ get_base_conninfo(char *conninfo, char *dbname, const char *noderole)
char *ret;
int i;
- pg_log_info("validating connection string on %s", noderole);
+ pg_log_info("validating connection string on subscriber");
conn_opts = PQconninfoParse(conninfo, &errmsg);
if (conn_opts == NULL)
@@ -425,6 +423,58 @@ disconnect_database(PGconn *conn)
PQfinish(conn);
}
+/*
+ * Obtain primary_conninfo from the target server. The value would be used for
+ * connecting from the pg_createsubscriber itself and logical replication apply
+ * worker.
+ */
+static char *
+get_primary_conninfo(const char *base_conninfo)
+{
+ PGconn *conn;
+ PGresult *res;
+ char *conninfo;
+ char *primaryconninfo;
+
+ pg_log_info("getting primary_conninfo from standby");
+
+ /*
+ * Construct a connection string to the target instance. Since dbinfo has
+ * not stored infomation yet, we must directly get the first element of the
+ * database list.
+ */
+ conninfo = concat_conninfo_dbname(base_conninfo, database_names.head->val);
+
+ conn = connect_database(conninfo);
+ if (conn == NULL)
+ exit(1);
+
+ res = PQexec(conn, "SHOW primary_conninfo;");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("could not send command \"%s\": %s",
+ "SHOW primary_conninfo;", PQresultErrorMessage(res));
+ PQclear(res);
+ disconnect_database(conn);
+ exit(1);
+ }
+
+ primaryconninfo = pg_strdup(PQgetvalue(res, 0, 0));
+
+ if (strlen(primaryconninfo) == 0)
+ {
+ pg_log_error("primary_conninfo is empty");
+ pg_log_error_hint("Check whether the target server is really a physical standby.");
+ exit(1);
+ }
+
+ pg_free(conninfo);
+ PQclear(res);
+ disconnect_database(conn);
+
+ return primaryconninfo;
+}
+
/*
* Obtain the system identifier using the provided connection. It will be used
* to compare if a data directory is a clone of another one.
@@ -1452,7 +1502,6 @@ main(int argc, char **argv)
{"help", no_argument, NULL, '?'},
{"version", no_argument, NULL, 'V'},
{"pgdata", required_argument, NULL, 'D'},
- {"publisher-server", required_argument, NULL, 'P'},
{"subscriber-server", required_argument, NULL, 'S'},
{"database", required_argument, NULL, 'd'},
{"dry-run", no_argument, NULL, 'n'},
@@ -1519,7 +1568,7 @@ main(int argc, char **argv)
}
#endif
- while ((c = getopt_long(argc, argv, "D:P:S:d:nrt:v",
+ while ((c = getopt_long(argc, argv, "D:S:d:nrt:v",
long_options, &option_index)) != -1)
{
switch (c)
@@ -1527,9 +1576,6 @@ main(int argc, char **argv)
case 'D':
subscriber_dir = pg_strdup(optarg);
break;
- case 'P':
- pub_conninfo_str = pg_strdup(optarg);
- break;
case 'S':
sub_conninfo_str = pg_strdup(optarg);
break;
@@ -1581,34 +1627,13 @@ main(int argc, char **argv)
exit(1);
}
- /*
- * Parse connection string. Build a base connection string that might be
- * reused by multiple databases.
- */
- if (pub_conninfo_str == NULL)
- {
- /*
- * TODO use primary_conninfo (if available) from subscriber and
- * extract publisher connection string. Assume that there are
- * identical entries for physical and logical replication. If there is
- * not, we would fail anyway.
- */
- pg_log_error("no publisher connection string specified");
- pg_log_error_hint("Try \"%s --help\" for more information.", progname);
- exit(1);
- }
- pub_base_conninfo = get_base_conninfo(pub_conninfo_str, dbname_conninfo,
- "publisher");
- if (pub_base_conninfo == NULL)
- exit(1);
-
if (sub_conninfo_str == NULL)
{
pg_log_error("no subscriber connection string specified");
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
exit(1);
}
- sub_base_conninfo = get_base_conninfo(sub_conninfo_str, NULL, "subscriber");
+ sub_base_conninfo = get_base_conninfo(sub_conninfo_str, dbname_conninfo);
if (sub_base_conninfo == NULL)
exit(1);
@@ -1618,7 +1643,7 @@ main(int argc, char **argv)
/*
* If --database option is not provided, try to obtain the dbname from
- * the publisher conninfo. If dbname parameter is not available, error
+ * the subscriber conninfo. If dbname parameter is not available, error
* out.
*/
if (dbname_conninfo)
@@ -1626,7 +1651,7 @@ main(int argc, char **argv)
simple_string_list_append(&database_names, dbname_conninfo);
num_dbs++;
- pg_log_info("database \"%s\" was extracted from the publisher connection string",
+ pg_log_info("database \"%s\" was extracted from the subscriber connection string",
dbname_conninfo);
}
else
@@ -1637,6 +1662,9 @@ main(int argc, char **argv)
}
}
+ /* Obtain a connection string from the target */
+ pub_base_conninfo = get_primary_conninfo(sub_base_conninfo);
+
/*
* Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
*/
diff --git a/src/bin/pg_basebackup/t/040_pg_createsubscriber.pl b/src/bin/pg_basebackup/t/040_pg_createsubscriber.pl
index 0f02b1bfac..5c240a5417 100644
--- a/src/bin/pg_basebackup/t/040_pg_createsubscriber.pl
+++ b/src/bin/pg_basebackup/t/040_pg_createsubscriber.pl
@@ -17,18 +17,11 @@ my $datadir = PostgreSQL::Test::Utils::tempdir;
command_fails(['pg_createsubscriber'],
'no subscriber data directory specified');
-command_fails(
- [
- 'pg_createsubscriber',
- '--pgdata', $datadir
- ],
- 'no publisher connection string specified');
command_fails(
[
'pg_createsubscriber',
'--dry-run',
'--pgdata', $datadir,
- '--publisher-server', 'dbname=postgres'
],
'no subscriber connection string specified');
command_fails(
@@ -36,7 +29,6 @@ command_fails(
'pg_createsubscriber',
'--verbose',
'--pgdata', $datadir,
- '--publisher-server', 'dbname=postgres',
'--subscriber-server', 'dbname=postgres'
],
'no database name specified');
diff --git a/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
index 534bc53a76..a9d03acc87 100644
--- a/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
+++ b/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
@@ -56,19 +56,17 @@ command_fails(
[
'pg_createsubscriber', '--verbose',
'--pgdata', $node_f->data_dir,
- '--publisher-server', $node_p->connstr('pg1'),
'--subscriber-server', $node_f->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
],
- 'subscriber data directory is not a copy of the source database cluster');
+ 'target database is not a physical standby');
# dry run mode on node S
command_ok(
[
'pg_createsubscriber', '--verbose', '--dry-run',
'--pgdata', $node_s->data_dir,
- '--publisher-server', $node_p->connstr('pg1'),
'--subscriber-server', $node_s->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
@@ -88,7 +86,6 @@ command_ok(
[
'pg_createsubscriber', '--verbose',
'--pgdata', $node_s->data_dir,
- '--publisher-server', $node_p->connstr('pg1'),
'--subscriber-server', $node_s->connstr('pg1'),
'--database', 'pg1',
'--database', 'pg2'
--
2.43.0
[application/octet-stream] v12-0004-Exit-earlier-when-we-are-in-the-dry_run-mode.patch (11.0K, ../../TY3PR01MB9889320DDAF153FC8A532630F57C2@TY3PR01MB9889.jpnprd01.prod.outlook.com/5-v12-0004-Exit-earlier-when-we-are-in-the-dry_run-mode.patch)
download | inline diff:
From c1aa11c772925044318fc0288e5a16e6bc66f977 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <[email protected]>
Date: Wed, 31 Jan 2024 10:45:21 +0000
Subject: [PATCH v12 4/5] Exit earlier when we are in the dry_run mode
---
src/bin/pg_basebackup/pg_createsubscriber.c | 198 ++++++++------------
1 file changed, 75 insertions(+), 123 deletions(-)
diff --git a/src/bin/pg_basebackup/pg_createsubscriber.c b/src/bin/pg_basebackup/pg_createsubscriber.c
index f0e9db7793..6f832f7551 100644
--- a/src/bin/pg_basebackup/pg_createsubscriber.c
+++ b/src/bin/pg_basebackup/pg_createsubscriber.c
@@ -584,8 +584,7 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
cf->system_identifier |= ((uint64) tv.tv_usec) << 12;
cf->system_identifier |= getpid() & 0xFFF;
- if (!dry_run)
- update_controlfile(datadir, cf, true);
+ update_controlfile(datadir, cf, true);
pg_log_info("system identifier is %llu on subscriber", (unsigned long long) cf->system_identifier);
@@ -595,14 +594,12 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
pg_log_debug("command is: %s", cmd_str);
- if (!dry_run)
- {
- rc = system(cmd_str);
- if (rc == 0)
- pg_log_info("subscriber successfully changed the system identifier");
- else
- pg_log_error("subscriber failed to change system identifier: exit code: %d", rc);
- }
+ rc = system(cmd_str);
+
+ if (rc == 0)
+ pg_log_info("subscriber successfully changed the system identifier");
+ else
+ pg_log_error("subscriber failed to change system identifier: exit code: %d", rc);
pfree(cf);
}
@@ -676,7 +673,7 @@ setup_publisher(LogicalRepInfo *dbinfo)
dbinfo[i].subname = pg_strdup(replslotname);
/* Create replication slot on publisher. */
- if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL || dry_run)
+ if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL)
pg_log_info("create replication slot \"%s\" on publisher", replslotname);
else
return false;
@@ -956,26 +953,20 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
pg_log_debug("command is: %s", str->data);
- if (!dry_run)
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
- res = PQexec(conn, str->data);
- if (PQresultStatus(res) != PGRES_TUPLES_OK)
- {
- pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
- PQresultErrorMessage(res));
- return lsn;
- }
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQresultErrorMessage(res));
+ return lsn;
}
/* for cleanup purposes */
if (!transient_replslot)
dbinfo->made_replslot = true;
- if (!dry_run)
- {
- lsn = pg_strdup(PQgetvalue(res, 0, 1));
- PQclear(res);
- }
+ lsn = pg_strdup(PQgetvalue(res, 0, 1));
+ PQclear(res);
destroyPQExpBuffer(str);
@@ -997,15 +988,12 @@ drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_nam
pg_log_debug("command is: %s", str->data);
- if (!dry_run)
- {
- res = PQexec(conn, str->data);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
- PQerrorMessage(conn));
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
+ PQerrorMessage(conn));
- PQclear(res);
- }
+ PQclear(res);
destroyPQExpBuffer(str);
}
@@ -1138,11 +1126,8 @@ wait_for_end_recovery(const char *conninfo)
PQclear(res);
- /*
- * Does the recovery process finish? In dry run mode, there is no
- * recovery mode. Bail out as the recovery process has ended.
- */
- if (!in_recovery || dry_run)
+ /* Does the recovery process finish? */
+ if (!in_recovery)
{
status = POSTMASTER_READY;
break;
@@ -1239,24 +1224,19 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
pg_log_debug("command is: %s", str->data);
- if (!dry_run)
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
- res = PQexec(conn, str->data);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- {
- pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
- dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
- PQfinish(conn);
- exit(1);
- }
+ pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
+ dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
}
/* for cleanup purposes */
dbinfo->made_publication = true;
- if (!dry_run)
- PQclear(res);
-
+ PQclear(res);
destroyPQExpBuffer(str);
}
@@ -1277,14 +1257,11 @@ drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
pg_log_debug("command is: %s", str->data);
- if (!dry_run)
- {
- res = PQexec(conn, str->data);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
- PQclear(res);
- }
+ PQclear(res);
destroyPQExpBuffer(str);
}
@@ -1318,24 +1295,19 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
pg_log_debug("command is: %s", str->data);
- if (!dry_run)
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
- res = PQexec(conn, str->data);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- {
- pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
- dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
- PQfinish(conn);
- exit(1);
- }
+ pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
+ dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
}
/* for cleanup purposes */
dbinfo->made_subscription = true;
- if (!dry_run)
- PQclear(res);
-
+ PQclear(res);
destroyPQExpBuffer(str);
}
@@ -1356,14 +1328,11 @@ drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
pg_log_debug("command is: %s", str->data);
- if (!dry_run)
- {
- res = PQexec(conn, str->data);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
- PQclear(res);
- }
+ PQclear(res);
destroyPQExpBuffer(str);
}
@@ -1402,7 +1371,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
exit(1);
}
- if (PQntuples(res) != 1 && !dry_run)
+ if (PQntuples(res) != 1)
{
pg_log_error("could not obtain subscription OID: got %d rows, expected %d rows",
PQntuples(res), 1);
@@ -1411,16 +1380,8 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
exit(1);
}
- if (dry_run)
- {
- suboid = InvalidOid;
- snprintf(lsnstr, sizeof(lsnstr), "%X/%X", LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
- }
- else
- {
- suboid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
- snprintf(lsnstr, sizeof(lsnstr), "%s", lsn);
- }
+ suboid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ snprintf(lsnstr, sizeof(lsnstr), "%s", lsn);
/*
* The origin name is defined as pg_%u. %u is the subscription OID. See
@@ -1439,20 +1400,16 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
pg_log_debug("command is: %s", str->data);
- if (!dry_run)
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
- res = PQexec(conn, str->data);
- if (PQresultStatus(res) != PGRES_TUPLES_OK)
- {
- pg_log_error("could not set replication progress for the subscription \"%s\": %s",
- dbinfo->subname, PQresultErrorMessage(res));
- PQfinish(conn);
- exit(1);
- }
-
- PQclear(res);
+ pg_log_error("could not set replication progress for the subscription \"%s\": %s",
+ dbinfo->subname, PQresultErrorMessage(res));
+ PQfinish(conn);
+ exit(1);
}
+ PQclear(res);
destroyPQExpBuffer(str);
}
@@ -1477,20 +1434,16 @@ enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
pg_log_debug("command is: %s", str->data);
- if (!dry_run)
+ res = PQexec(conn, str->data);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
- res = PQexec(conn, str->data);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- {
- pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
- PQerrorMessage(conn));
- PQfinish(conn);
- exit(1);
- }
-
- PQclear(res);
+ pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
+ PQerrorMessage(conn));
+ PQfinish(conn);
+ exit(1);
}
+ PQclear(res);
destroyPQExpBuffer(str);
}
@@ -1759,6 +1712,14 @@ main(int argc, char **argv)
exit(1);
}
+ /*
+ * Exit earlier when we are in the dry_run mode.
+ *
+ * XXX: Should we keep turning on the standby server in case of dry_run?
+ */
+ if (dry_run)
+ goto cleanup;
+
/*
* Create a temporary logical replication slot to get a consistent LSN.
*
@@ -1783,26 +1744,16 @@ main(int argc, char **argv)
* Despite of the recovery parameters will be written to the subscriber,
* use a publisher connection for the follwing recovery functions. The
* connection is only used to check the current server version (physical
- * replica, same server version). The subscriber is not running yet. In
- * dry run mode, the recovery parameters *won't* be written. An invalid
- * LSN is used for printing purposes.
+ * replica, same server version). The subscriber is not running yet.
*/
recoveryconfcontents = GenerateRecoveryConfig(conn, NULL);
appendPQExpBuffer(recoveryconfcontents, "recovery_target_inclusive = true\n");
appendPQExpBuffer(recoveryconfcontents, "recovery_target_action = promote\n");
- if (dry_run)
- {
- appendPQExpBuffer(recoveryconfcontents, "# dry run mode");
- appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%X/%X'\n",
- LSN_FORMAT_ARGS((XLogRecPtr) InvalidXLogRecPtr));
- }
- else
- {
- appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
- consistent_lsn);
- WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
- }
+ appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
+ consistent_lsn);
+ WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+
disconnect_database(conn);
pg_log_debug("recovery parameters:\n%s", recoveryconfcontents->data);
@@ -1860,6 +1811,7 @@ main(int argc, char **argv)
*/
modify_sysid(pg_resetwal_path, subscriber_dir);
+cleanup:
/*
* The log file is kept if retain option is specified or this tool does
* not run successfully. Otherwise, log file is removed.
--
2.43.0
[application/octet-stream] v12-0005-Divide-LogicalReplInfo-into-some-strcutures.patch (49.5K, ../../TY3PR01MB9889320DDAF153FC8A532630F57C2@TY3PR01MB9889.jpnprd01.prod.outlook.com/6-v12-0005-Divide-LogicalReplInfo-into-some-strcutures.patch)
download | inline diff:
From dd7be7f34e1ac68c2b6fd193dec2aa6b2832e1b9 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <[email protected]>
Date: Mon, 29 Jan 2024 07:03:59 +0000
Subject: [PATCH v12 5/5] Divide LogicalReplInfo into some strcutures
---
src/bin/pg_basebackup/pg_createsubscriber.c | 660 ++++++++++--------
.../t/041_pg_createsubscriber_standby.pl | 2 +-
src/tools/pgindent/typedefs.list | 5 +-
3 files changed, 365 insertions(+), 302 deletions(-)
diff --git a/src/bin/pg_basebackup/pg_createsubscriber.c b/src/bin/pg_basebackup/pg_createsubscriber.c
index 6f832f7551..898fa3f114 100644
--- a/src/bin/pg_basebackup/pg_createsubscriber.c
+++ b/src/bin/pg_basebackup/pg_createsubscriber.c
@@ -32,54 +32,78 @@
#define PGS_OUTPUT_DIR "pg_createsubscriber_output.d"
-typedef struct LogicalRepInfo
+typedef struct LogicalRepPerdbInfo
{
- Oid oid; /* database OID */
- char *dbname; /* database name */
- char *pubconninfo; /* publication connection string for logical
- * replication */
- char *subconninfo; /* subscription connection string for logical
- * replication */
- char *pubname; /* publication name */
- char *subname; /* subscription name (also replication slot
- * name) */
-
- bool made_replslot; /* replication slot was created */
- bool made_publication; /* publication was created */
- bool made_subscription; /* subscription was created */
-} LogicalRepInfo;
+ Oid oid;
+ char *dbname;
+ bool made_replslot; /* replication slot was created */
+ bool made_publication; /* publication was created */
+ bool made_subscription; /* subscription was created */
+} LogicalRepPerdbInfo;
+
+typedef struct LogicalRepPerdbInfoArr
+{
+ LogicalRepPerdbInfo *perdb; /* array of db infos */
+ int ndbs; /* number of db infos */
+} LogicalRepPerdbInfoArr;
+
+typedef struct PrimaryInfo
+{
+ char *base_conninfo;
+ uint64 sysid;
+ bool made_transient_replslot;
+} PrimaryInfo;
+
+typedef struct StandbyInfo
+{
+ char *base_conninfo;
+ char *bindir;
+ char *pgdata;
+ char *primary_slot_name;
+ char *server_log;
+ uint64 sysid;
+} StandbyInfo;
static void cleanup_objects_atexit(void);
static void usage();
-static char *get_base_conninfo(char *conninfo, char *dbname);
-static bool get_exec_path(const char *path);
+static bool get_exec_base_path(const char *path);
static bool check_data_directory(const char *datadir);
-static char *get_primary_conninfo(const char *base_conninfo);
+static char *get_primary_conninfo(StandbyInfo *standby);
static char *concat_conninfo_dbname(const char *conninfo, const char *dbname);
-static LogicalRepInfo *store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo);
-static PGconn *connect_database(const char *conninfo);
+static void store_db_names(LogicalRepPerdbInfo **perdb, int ndbs);
+static PGconn *connect_database(const char *base_conninfo, const char*dbname);
static void disconnect_database(PGconn *conn);
-static uint64 get_sysid_from_conn(const char *conninfo);
-static uint64 get_control_from_datadir(const char *datadir);
-static void modify_sysid(const char *pg_resetwal_path, const char *datadir);
-static bool check_publisher(LogicalRepInfo *dbinfo);
-static bool setup_publisher(LogicalRepInfo *dbinfo);
-static bool check_subscriber(LogicalRepInfo *dbinfo);
-static bool setup_subscriber(LogicalRepInfo *dbinfo, const char *consistent_lsn);
-static char *create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
- char *slot_name);
-static void drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name);
+static void get_sysid_for_primary(PrimaryInfo *primary, char *dbname);
+static void get_sysid_for_standby(StandbyInfo *standby);
+static void modify_sysid(const char *bindir, const char *datadir);
+static bool check_publisher(PrimaryInfo *primary, LogicalRepPerdbInfoArr *dbarr);
+static bool setup_publisher(PrimaryInfo *primary, LogicalRepPerdbInfoArr *dbarr);
+static bool check_subscriber(StandbyInfo *standby, LogicalRepPerdbInfoArr *dbarr);
+static bool setup_subscriber(StandbyInfo *standby, PrimaryInfo *primary,
+ LogicalRepPerdbInfoArr *dbarr,
+ const char *consistent_lsn);
+static char *create_logical_replication_slot(PGconn *conn, bool temporary,
+ LogicalRepPerdbInfo *perdb);
+static void drop_replication_slot(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *slot_name);
static char *server_logfile_name(const char *datadir);
-static void start_standby_server(const char *pg_ctl_path, const char *datadir, const char *logfile);
-static void stop_standby_server(const char *pg_ctl_path, const char *datadir);
+static void start_standby_server(StandbyInfo *standby);
+static void stop_standby_server(StandbyInfo *standby);
static void pg_ctl_status(const char *pg_ctl_cmd, int rc, int action);
-static void wait_for_end_recovery(const char *conninfo);
-static void create_publication(PGconn *conn, LogicalRepInfo *dbinfo);
-static void drop_publication(PGconn *conn, LogicalRepInfo *dbinfo);
-static void create_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-static void drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
-static void set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn);
-static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
+
+
+static void wait_for_end_recovery(StandbyInfo *standby,
+ const char *dbname);
+static void create_publication(PGconn *conn, PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb);
+static void drop_publication(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void create_subscription(PGconn *conn, StandbyInfo *standby,
+ PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb);
+static void drop_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb);
+static void set_replication_progress(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *lsn);
+static void enable_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb);
#define USEC_PER_SEC 1000000
#define WAIT_INTERVAL 1 /* 1 second */
@@ -87,21 +111,17 @@ static void enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo);
/* Options */
static const char *progname;
-static char *subscriber_dir = NULL;
static char *sub_conninfo_str = NULL;
static SimpleStringList database_names = {NULL, NULL};
-static char *primary_slot_name = NULL;
static bool dry_run = false;
static bool retain = false;
static int recovery_timeout = 0;
static bool success = false;
-static char *pg_ctl_path = NULL;
-static char *pg_resetwal_path = NULL;
-
-static LogicalRepInfo *dbinfo;
-static int num_dbs = 0;
+static LogicalRepPerdbInfoArr dbarr;
+static PrimaryInfo primary;
+static StandbyInfo standby;
enum WaitPMResult
{
@@ -111,6 +131,30 @@ enum WaitPMResult
POSTMASTER_FAILED
};
+/*
+ * Build the replication slot name. The name must not exceed
+ * NAMEDATALEN - 1. This current schema uses a maximum of 42
+ * characters (20 + 10 + 1 + 10 + '\0'). PID is included to reduce the
+ * probability of collision. By default, subscription name is used as
+ * replication slot name.
+ */
+static inline void
+get_subscription_name(Oid oid, int pid, char *subname, Size szsub)
+{
+ snprintf(subname, szsub, "pg_createsubscriber_%u_%d", oid, pid);
+}
+
+/*
+ * Build the publication name. The name must not exceed NAMEDATALEN -
+ * 1. This current schema uses a maximum of 31 characters (20 + 10 +
+ * '\0').
+ */
+static inline void
+get_publication_name(Oid oid, char *pubname, Size szpub)
+{
+ snprintf(pubname, szpub, "pg_createsubscriber_%u", oid);
+}
+
/*
* Cleanup objects that were created by pg_createsubscriber if there is an error.
@@ -128,33 +172,54 @@ cleanup_objects_atexit(void)
if (success)
return;
- for (i = 0; i < num_dbs; i++)
+ for (i = 0; i < dbarr.ndbs; i++)
{
- if (dbinfo[i].made_subscription)
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[i];
+
+ if (perdb->made_subscription)
{
- conn = connect_database(dbinfo[i].subconninfo);
+ conn = connect_database(standby.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- drop_subscription(conn, &dbinfo[i]);
- if (dbinfo[i].made_publication)
- drop_publication(conn, &dbinfo[i]);
+ drop_subscription(conn, perdb);
disconnect_database(conn);
}
}
- if (dbinfo[i].made_publication || dbinfo[i].made_replslot)
+ if (perdb->made_publication || perdb->made_replslot)
{
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- if (dbinfo[i].made_publication)
- drop_publication(conn, &dbinfo[i]);
- if (dbinfo[i].made_replslot)
- drop_replication_slot(conn, &dbinfo[i], NULL);
- disconnect_database(conn);
+ if (perdb->made_publication)
+ drop_publication(conn, perdb);
+ if (perdb->made_replslot)
+ {
+ char replslotname[NAMEDATALEN];
+
+ get_subscription_name(perdb->oid, (int) getpid(),
+ replslotname, NAMEDATALEN);
+ drop_replication_slot(conn, perdb, replslotname);
+ }
}
}
}
+
+ if (primary.made_transient_replslot)
+ {
+ char transient_replslot[NAMEDATALEN];
+
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
+
+ if (conn != NULL)
+ {
+ snprintf(transient_replslot, NAMEDATALEN, "pg_subscriber_%d_startpoint",
+ (int) getpid());
+
+ drop_replication_slot(conn, &dbarr.perdb[0], transient_replslot);
+ disconnect_database(conn);
+ }
+ }
}
static void
@@ -236,15 +301,16 @@ get_base_conninfo(char *conninfo, char *dbname)
}
/*
- * Get the absolute path from other PostgreSQL binaries (pg_ctl and
- * pg_resetwal) that is used by it.
+ * Get the absolute binary path from another PostgreSQL binary (pg_ctl) and set
+ * to StandbyInfo.
*/
static bool
-get_exec_path(const char *path)
+get_exec_base_path(const char *path)
{
- int rc;
+ int rc;
+ char pg_ctl_path[MAXPGPATH];
+ char *p;
- pg_ctl_path = pg_malloc(MAXPGPATH);
rc = find_other_exec(path, "pg_ctl",
"pg_ctl (PostgreSQL) " PG_VERSION "\n",
pg_ctl_path);
@@ -269,30 +335,12 @@ get_exec_path(const char *path)
pg_log_debug("pg_ctl path is: %s", pg_ctl_path);
- pg_resetwal_path = pg_malloc(MAXPGPATH);
- rc = find_other_exec(path, "pg_resetwal",
- "pg_resetwal (PostgreSQL) " PG_VERSION "\n",
- pg_resetwal_path);
- if (rc < 0)
- {
- char full_path[MAXPGPATH];
-
- if (find_my_exec(path, full_path) < 0)
- strlcpy(full_path, progname, sizeof(full_path));
- if (rc == -1)
- pg_log_error("The program \"%s\" is needed by %s but was not found in the\n"
- "same directory as \"%s\".\n"
- "Check your installation.",
- "pg_resetwal", progname, full_path);
- else
- pg_log_error("The program \"%s\" was found by \"%s\"\n"
- "but was not the same version as %s.\n"
- "Check your installation.",
- "pg_resetwal", full_path, progname);
- return false;
- }
+ /* Extract the directory part from the path */
+ p = strrchr(pg_ctl_path, 'p');
+ Assert(p);
- pg_log_debug("pg_resetwal path is: %s", pg_resetwal_path);
+ *p = '\0';
+ standby.bindir = pg_strdup(pg_ctl_path);
return true;
}
@@ -356,45 +404,31 @@ concat_conninfo_dbname(const char *conninfo, const char *dbname)
}
/*
- * Store publication and subscription information.
+ * Initialize per-db structure and store the name of databases.
*/
-static LogicalRepInfo *
-store_pub_sub_info(const char *pub_base_conninfo, const char *sub_base_conninfo)
+static void
+store_db_names(LogicalRepPerdbInfo **perdb, int ndbs)
{
- LogicalRepInfo *dbinfo;
- SimpleStringListCell *cell;
- int i = 0;
+ SimpleStringListCell *cell;
+ int i = 0;
- dbinfo = (LogicalRepInfo *) pg_malloc(num_dbs * sizeof(LogicalRepInfo));
+ dbarr.perdb = (LogicalRepPerdbInfo *) pg_malloc0(ndbs *
+ sizeof(LogicalRepPerdbInfo));
for (cell = database_names.head; cell; cell = cell->next)
{
- char *conninfo;
-
- /* Publisher. */
- conninfo = concat_conninfo_dbname(pub_base_conninfo, cell->val);
- dbinfo[i].pubconninfo = conninfo;
- dbinfo[i].dbname = cell->val;
- dbinfo[i].made_replslot = false;
- dbinfo[i].made_publication = false;
- dbinfo[i].made_subscription = false;
- /* other struct fields will be filled later. */
-
- /* Subscriber. */
- conninfo = concat_conninfo_dbname(sub_base_conninfo, cell->val);
- dbinfo[i].subconninfo = conninfo;
-
+ (*perdb)[i].dbname = pg_strdup(cell->val);
i++;
}
-
- return dbinfo;
}
static PGconn *
-connect_database(const char *conninfo)
+connect_database(const char *base_conninfo, const char*dbname)
{
PGconn *conn;
PGresult *res;
+ char *conninfo = concat_conninfo_dbname(base_conninfo,
+ dbname);
conn = PQconnectdb(conninfo);
if (PQstatus(conn) != CONNECTION_OK)
@@ -412,6 +446,7 @@ connect_database(const char *conninfo)
}
PQclear(res);
+ pfree(conninfo);
return conn;
}
@@ -429,11 +464,10 @@ disconnect_database(PGconn *conn)
* worker.
*/
static char *
-get_primary_conninfo(const char *base_conninfo)
+get_primary_conninfo(StandbyInfo *standby)
{
PGconn *conn;
PGresult *res;
- char *conninfo;
char *primaryconninfo;
pg_log_info("getting primary_conninfo from standby");
@@ -443,9 +477,7 @@ get_primary_conninfo(const char *base_conninfo)
* not stored infomation yet, we must directly get the first element of the
* database list.
*/
- conninfo = concat_conninfo_dbname(base_conninfo, database_names.head->val);
-
- conn = connect_database(conninfo);
+ conn = connect_database(standby->base_conninfo, database_names.head->val);
if (conn == NULL)
exit(1);
@@ -468,7 +500,6 @@ get_primary_conninfo(const char *base_conninfo)
exit(1);
}
- pg_free(conninfo);
PQclear(res);
disconnect_database(conn);
@@ -476,19 +507,18 @@ get_primary_conninfo(const char *base_conninfo)
}
/*
- * Obtain the system identifier using the provided connection. It will be used
- * to compare if a data directory is a clone of another one.
+ * Obtain the system identifier from the primary server. It will be used to
+ * compare if a data directory is a clone of another one.
*/
-static uint64
-get_sysid_from_conn(const char *conninfo)
+static void
+get_sysid_for_primary(PrimaryInfo *primary, char *dbname)
{
PGconn *conn;
PGresult *res;
- uint64 sysid;
pg_log_info("getting system identifier from publisher");
- conn = connect_database(conninfo);
+ conn = connect_database(primary->base_conninfo, dbname);
if (conn == NULL)
exit(1);
@@ -511,13 +541,12 @@ get_sysid_from_conn(const char *conninfo)
exit(1);
}
- sysid = strtou64(PQgetvalue(res, 0, 2), NULL, 10);
+ primary->sysid = strtou64(PQgetvalue(res, 0, 2), NULL, 10);
- pg_log_info("system identifier is %llu on publisher", (unsigned long long) sysid);
+ pg_log_info("system identifier is %llu on publisher",
+ (unsigned long long) primary->sysid);
disconnect_database(conn);
-
- return sysid;
}
/*
@@ -525,29 +554,26 @@ get_sysid_from_conn(const char *conninfo)
* if a data directory is a clone of another one. This routine is used locally
* and avoids a connection establishment.
*/
-static uint64
-get_control_from_datadir(const char *datadir)
+static void
+get_sysid_for_standby(StandbyInfo *standby)
{
ControlFileData *cf;
bool crc_ok;
- uint64 sysid;
pg_log_info("getting system identifier from subscriber");
- cf = get_controlfile(datadir, &crc_ok);
+ cf = get_controlfile(standby->pgdata, &crc_ok);
if (!crc_ok)
{
pg_log_error("control file appears to be corrupt");
exit(1);
}
- sysid = cf->system_identifier;
+ standby->sysid = cf->system_identifier;
- pg_log_info("system identifier is %llu on subscriber", (unsigned long long) sysid);
+ pg_log_info("system identifier is %llu on subscriber", (unsigned long long) standby->sysid);
pfree(cf);
-
- return sysid;
}
/*
@@ -556,7 +582,7 @@ get_control_from_datadir(const char *datadir)
* files from one of the systems might be used in the other one.
*/
static void
-modify_sysid(const char *pg_resetwal_path, const char *datadir)
+modify_sysid(const char *bindir, const char *datadir)
{
ControlFileData *cf;
bool crc_ok;
@@ -590,7 +616,7 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
pg_log_info("running pg_resetwal on the subscriber");
- cmd_str = psprintf("\"%s\" -D \"%s\"", pg_resetwal_path, datadir);
+ cmd_str = psprintf("\"%s/pg_resetwal\" -D \"%s\"", bindir, datadir);
pg_log_debug("command is: %s", cmd_str);
@@ -609,17 +635,16 @@ modify_sysid(const char *pg_resetwal_path, const char *datadir)
* replication.
*/
static bool
-setup_publisher(LogicalRepInfo *dbinfo)
+setup_publisher(PrimaryInfo *primary, LogicalRepPerdbInfoArr *dbarr)
{
PGconn *conn;
PGresult *res;
- for (int i = 0; i < num_dbs; i++)
+ for (int i = 0; i < dbarr->ndbs; i++)
{
- char pubname[NAMEDATALEN];
- char replslotname[NAMEDATALEN];
+ LogicalRepPerdbInfo *perdb = &dbarr->perdb[i];
- conn = connect_database(dbinfo[i].pubconninfo);
+ conn = connect_database(primary->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -639,43 +664,20 @@ setup_publisher(LogicalRepInfo *dbinfo)
}
/* Remember database OID. */
- dbinfo[i].oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
+ perdb->oid = strtoul(PQgetvalue(res, 0, 0), NULL, 10);
PQclear(res);
- /*
- * Build the publication name. The name must not exceed NAMEDATALEN -
- * 1. This current schema uses a maximum of 31 characters (20 + 10 +
- * '\0').
- */
- snprintf(pubname, sizeof(pubname), "pg_createsubscriber_%u", dbinfo[i].oid);
- dbinfo[i].pubname = pg_strdup(pubname);
-
/*
* Create publication on publisher. This step should be executed
* *before* promoting the subscriber to avoid any transactions between
* consistent LSN and the new publication rows (such transactions
* wouldn't see the new publication rows resulting in an error).
*/
- create_publication(conn, &dbinfo[i]);
-
- /*
- * Build the replication slot name. The name must not exceed
- * NAMEDATALEN - 1. This current schema uses a maximum of 42
- * characters (20 + 10 + 1 + 10 + '\0'). PID is included to reduce the
- * probability of collision. By default, subscription name is used as
- * replication slot name.
- */
- snprintf(replslotname, sizeof(replslotname),
- "pg_createsubscriber_%u_%d",
- dbinfo[i].oid,
- (int) getpid());
- dbinfo[i].subname = pg_strdup(replslotname);
+ create_publication(conn, primary, perdb);
/* Create replication slot on publisher. */
- if (create_logical_replication_slot(conn, &dbinfo[i], replslotname) != NULL)
- pg_log_info("create replication slot \"%s\" on publisher", replslotname);
- else
+ if (create_logical_replication_slot(conn, false, perdb) == NULL)
return false;
disconnect_database(conn);
@@ -688,7 +690,7 @@ setup_publisher(LogicalRepInfo *dbinfo)
* Is the primary server ready for logical replication?
*/
static bool
-check_publisher(LogicalRepInfo *dbinfo)
+check_publisher(PrimaryInfo *primary, LogicalRepPerdbInfoArr *dbarr)
{
PGconn *conn;
PGresult *res;
@@ -711,7 +713,7 @@ check_publisher(LogicalRepInfo *dbinfo)
* max_replication_slots >= current + number of dbs to be converted
* max_wal_senders >= current + number of dbs to be converted
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary->base_conninfo, dbarr->perdb[0].dbname);
if (conn == NULL)
exit(1);
@@ -749,10 +751,11 @@ check_publisher(LogicalRepInfo *dbinfo)
* use after the transformation, hence, it will be removed at the end of
* this process.
*/
- if (primary_slot_name)
+ if (standby.primary_slot_name)
{
appendPQExpBuffer(str,
- "SELECT 1 FROM pg_replication_slots WHERE active AND slot_name = '%s'", primary_slot_name);
+ "SELECT 1 FROM pg_replication_slots WHERE active AND slot_name = '%s'",
+ standby.primary_slot_name);
pg_log_debug("command is: %s", str->data);
@@ -767,13 +770,14 @@ check_publisher(LogicalRepInfo *dbinfo)
{
pg_log_error("could not obtain replication slot information: got %d rows, expected %d row",
PQntuples(res), 1);
- pg_free(primary_slot_name); /* it is not being used. */
- primary_slot_name = NULL;
+ pg_free(standby.primary_slot_name); /* it is not being used. */
+ standby.primary_slot_name = NULL;
return false;
}
else
{
- pg_log_info("primary has replication slot \"%s\"", primary_slot_name);
+ pg_log_info("primary has replication slot \"%s\"",
+ standby.primary_slot_name);
}
PQclear(res);
@@ -787,17 +791,21 @@ check_publisher(LogicalRepInfo *dbinfo)
return false;
}
- if (max_repslots - cur_repslots < num_dbs)
+ if (max_repslots - cur_repslots < dbarr->ndbs)
{
- pg_log_error("publisher requires %d replication slots, but only %d remain", num_dbs, max_repslots - cur_repslots);
- pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", cur_repslots + num_dbs);
+ pg_log_error("publisher requires %d replication slots, but only %d remain",
+ dbarr->ndbs, max_repslots - cur_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.",
+ cur_repslots + dbarr->ndbs);
return false;
}
- if (max_walsenders - cur_walsenders < num_dbs)
+ if (max_walsenders - cur_walsenders < dbarr->ndbs)
{
- pg_log_error("publisher requires %d wal sender processes, but only %d remain", num_dbs, max_walsenders - cur_walsenders);
- pg_log_error_hint("Consider increasing max_wal_senders to at least %d.", cur_walsenders + num_dbs);
+ pg_log_error("publisher requires %d wal sender processes, but only %d remain",
+ dbarr->ndbs, max_walsenders - cur_walsenders);
+ pg_log_error_hint("Consider increasing max_wal_senders to at least %d.",
+ cur_walsenders + dbarr->ndbs);
return false;
}
@@ -808,7 +816,7 @@ check_publisher(LogicalRepInfo *dbinfo)
* Is the standby server ready for logical replication?
*/
static bool
-check_subscriber(LogicalRepInfo *dbinfo)
+check_subscriber(StandbyInfo *standby, LogicalRepPerdbInfoArr *dbarr)
{
PGconn *conn;
PGresult *res;
@@ -828,7 +836,7 @@ check_subscriber(LogicalRepInfo *dbinfo)
* max_logical_replication_workers >= number of dbs to be converted
* max_worker_processes >= 1 + number of dbs to be converted
*/
- conn = connect_database(dbinfo[0].subconninfo);
+ conn = connect_database(standby->base_conninfo, dbarr->perdb[0].dbname);
if (conn == NULL)
exit(1);
@@ -845,35 +853,41 @@ check_subscriber(LogicalRepInfo *dbinfo)
max_repslots = atoi(PQgetvalue(res, 1, 0));
max_wprocs = atoi(PQgetvalue(res, 2, 0));
if (strcmp(PQgetvalue(res, 3, 0), "") != 0)
- primary_slot_name = pg_strdup(PQgetvalue(res, 3, 0));
+ standby->primary_slot_name = pg_strdup(PQgetvalue(res, 3, 0));
pg_log_debug("subscriber: max_logical_replication_workers: %d", max_lrworkers);
pg_log_debug("subscriber: max_replication_slots: %d", max_repslots);
pg_log_debug("subscriber: max_worker_processes: %d", max_wprocs);
- pg_log_debug("subscriber: primary_slot_name: %s", primary_slot_name);
+ pg_log_debug("subscriber: primary_slot_name: %s", standby->primary_slot_name);
PQclear(res);
disconnect_database(conn);
- if (max_repslots < num_dbs)
+ if (max_repslots < dbarr->ndbs)
{
- pg_log_error("subscriber requires %d replication slots, but only %d remain", num_dbs, max_repslots);
- pg_log_error_hint("Consider increasing max_replication_slots to at least %d.", num_dbs);
+ pg_log_error("subscriber requires %d replication slots, but only %d remain",
+ dbarr->ndbs, max_repslots);
+ pg_log_error_hint("Consider increasing max_replication_slots to at least %d.",
+ dbarr->ndbs);
return false;
}
- if (max_lrworkers < num_dbs)
+ if (max_lrworkers < dbarr->ndbs)
{
- pg_log_error("subscriber requires %d logical replication workers, but only %d remain", num_dbs, max_lrworkers);
- pg_log_error_hint("Consider increasing max_logical_replication_workers to at least %d.", num_dbs);
+ pg_log_error("subscriber requires %d logical replication workers, but only %d remain",
+ dbarr->ndbs, max_lrworkers);
+ pg_log_error_hint("Consider increasing max_logical_replication_workers to at least %d.",
+ dbarr->ndbs);
return false;
}
- if (max_wprocs < num_dbs + 1)
+ if (max_wprocs < dbarr->ndbs + 1)
{
- pg_log_error("subscriber requires %d worker processes, but only %d remain", num_dbs + 1, max_wprocs);
- pg_log_error_hint("Consider increasing max_worker_processes to at least %d.", num_dbs + 1);
+ pg_log_error("subscriber requires %d worker processes, but only %d remain",
+ dbarr->ndbs + 1, max_wprocs);
+ pg_log_error_hint("Consider increasing max_worker_processes to at least %d.",
+ dbarr->ndbs + 1);
return false;
}
@@ -885,14 +899,17 @@ check_subscriber(LogicalRepInfo *dbinfo)
* enable the subscriptions. That's the last step for logical repliation setup.
*/
static bool
-setup_subscriber(LogicalRepInfo *dbinfo, const char *consistent_lsn)
+setup_subscriber(StandbyInfo *standby, PrimaryInfo *primary,
+ LogicalRepPerdbInfoArr *dbarr, const char *consistent_lsn)
{
PGconn *conn;
- for (int i = 0; i < num_dbs; i++)
+ for (int i = 0; i < dbarr->ndbs; i++)
{
+ LogicalRepPerdbInfo *perdb = &dbarr->perdb[i];
+
/* Connect to subscriber. */
- conn = connect_database(dbinfo[i].subconninfo);
+ conn = connect_database(standby->base_conninfo, perdb->dbname);
if (conn == NULL)
exit(1);
@@ -901,15 +918,15 @@ setup_subscriber(LogicalRepInfo *dbinfo, const char *consistent_lsn)
* available on the subscriber when the physical replica is promoted.
* Remove publications from the subscriber because it has no use.
*/
- drop_publication(conn, &dbinfo[i]);
+ drop_publication(conn, perdb);
- create_subscription(conn, &dbinfo[i]);
+ create_subscription(conn, standby, primary, perdb);
/* Set the replication progress to the correct LSN. */
- set_replication_progress(conn, &dbinfo[i], consistent_lsn);
+ set_replication_progress(conn, perdb, consistent_lsn);
/* Enable subscription. */
- enable_subscription(conn, &dbinfo[i]);
+ enable_subscription(conn, perdb);
disconnect_database(conn);
}
@@ -925,45 +942,55 @@ setup_subscriber(LogicalRepInfo *dbinfo, const char *consistent_lsn)
* result set that contains the consistent LSN.
*/
static char *
-create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
- char *slot_name)
+create_logical_replication_slot(PGconn *conn, bool temporary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res = NULL;
char *lsn = NULL;
- bool transient_replslot = false;
+ char slot_name[NAMEDATALEN];
Assert(conn != NULL);
/*
- * If no slot name is informed, it is a transient replication slot used
- * only for catch up purposes.
+ * Construct a name of logical replication slot. The formatting is
+ * different depends on its persistency.
+ *
+ * For persistent slots: the name must be same as the subscription.
+ * For temporary slots: OID is not needed, but another string is added.
*/
- if (slot_name[0] == '\0')
- {
- snprintf(slot_name, NAMEDATALEN, "pg_createsubscriber_%d_startpoint",
+ if (temporary)
+ snprintf(slot_name, NAMEDATALEN, "pg_subscriber_%d_startpoint",
(int) getpid());
- transient_replslot = true;
- }
+ else
+ get_subscription_name(perdb->oid, (int) getpid(), slot_name,
+ NAMEDATALEN);
- pg_log_info("creating the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+ pg_log_info("creating the replication slot \"%s\" on database \"%s\"",
+ slot_name, perdb->dbname);
appendPQExpBuffer(str, "SELECT * FROM pg_create_logical_replication_slot('%s', 'pgoutput', %s, false, false);",
- slot_name, transient_replslot ? "true" : "false");
+ slot_name, temporary ? "true" : "false");
pg_log_debug("command is: %s", str->data);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
- pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
- PQresultErrorMessage(res));
+ pg_log_error("could not create replication slot \"%s\" on database \"%s\": %s",
+ slot_name, perdb->dbname, PQresultErrorMessage(res));
+
return lsn;
}
+ pg_log_info("create replication slot \"%s\" on publisher", slot_name);
+
/* for cleanup purposes */
- if (!transient_replslot)
- dbinfo->made_replslot = true;
+ if (temporary)
+ primary.made_transient_replslot = true;
+ else
+ perdb->made_replslot = true;
+
lsn = pg_strdup(PQgetvalue(res, 0, 1));
PQclear(res);
@@ -974,14 +1001,16 @@ create_logical_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo,
}
static void
-drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_name)
+drop_replication_slot(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *slot_name)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
Assert(conn != NULL);
- pg_log_info("dropping the replication slot \"%s\" on database \"%s\"", slot_name, dbinfo->dbname);
+ pg_log_info("dropping the replication slot \"%s\" on database \"%s\"",
+ slot_name, perdb->dbname);
appendPQExpBuffer(str, "SELECT * FROM pg_drop_replication_slot('%s');",
slot_name);
@@ -990,8 +1019,9 @@ drop_replication_slot(PGconn *conn, LogicalRepInfo *dbinfo, const char *slot_nam
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s", slot_name, dbinfo->dbname,
- PQerrorMessage(conn));
+ pg_log_error("could not drop replication slot \"%s\" on database \"%s\": %s",
+ slot_name, perdb->dbname,
+ PQerrorMessage(conn));
PQclear(res);
@@ -1026,23 +1056,25 @@ server_logfile_name(const char *datadir)
}
static void
-start_standby_server(const char *pg_ctl_path, const char *datadir, const char *logfile)
+start_standby_server(StandbyInfo *standby)
{
char *pg_ctl_cmd;
int rc;
- pg_ctl_cmd = psprintf("\"%s\" start -D \"%s\" -s -l \"%s\"", pg_ctl_path, datadir, logfile);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" start -D \"%s\" -s -l \"%s\"",
+ standby->bindir, standby->pgdata, standby->server_log);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 1);
}
static void
-stop_standby_server(const char *pg_ctl_path, const char *datadir)
+stop_standby_server(StandbyInfo *standby)
{
char *pg_ctl_cmd;
int rc;
- pg_ctl_cmd = psprintf("\"%s\" stop -D \"%s\" -s", pg_ctl_path, datadir);
+ pg_ctl_cmd = psprintf("\"%s/pg_ctl\" stop -D \"%s\" -s", standby->bindir,
+ standby->pgdata);
rc = system(pg_ctl_cmd);
pg_ctl_status(pg_ctl_cmd, rc, 0);
}
@@ -1091,7 +1123,7 @@ pg_ctl_status(const char *pg_ctl_cmd, int rc, int action)
* the recovery process. By default, it waits forever.
*/
static void
-wait_for_end_recovery(const char *conninfo)
+wait_for_end_recovery(StandbyInfo *standby, const char *dbname)
{
PGconn *conn;
PGresult *res;
@@ -1100,7 +1132,7 @@ wait_for_end_recovery(const char *conninfo)
pg_log_info("waiting the postmaster to reach the consistent state");
- conn = connect_database(conninfo);
+ conn = connect_database(standby->base_conninfo, dbname);
if (conn == NULL)
exit(1);
@@ -1139,7 +1171,7 @@ wait_for_end_recovery(const char *conninfo)
if (recovery_timeout > 0 && timer >= recovery_timeout)
{
pg_log_error("recovery timed out");
- stop_standby_server(pg_ctl_path, subscriber_dir);
+ stop_standby_server(standby);
exit(1);
}
@@ -1164,17 +1196,21 @@ wait_for_end_recovery(const char *conninfo)
* Create a publication that includes all tables in the database.
*/
static void
-create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+create_publication(PGconn *conn, PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
/* Check if the publication needs to be created. */
appendPQExpBuffer(str,
"SELECT puballtables FROM pg_catalog.pg_publication WHERE pubname = '%s'",
- dbinfo->pubname);
+ pubname);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
@@ -1194,7 +1230,7 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
*/
if (strcmp(PQgetvalue(res, 0, 0), "t") == 0)
{
- pg_log_info("publication \"%s\" already exists", dbinfo->pubname);
+ pg_log_info("publication \"%s\" already exists", pubname);
return;
}
else
@@ -1207,7 +1243,7 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* exact database oid in which puballtables is false.
*/
pg_log_error("publication \"%s\" does not replicate changes for all tables",
- dbinfo->pubname);
+ pubname);
pg_log_error_hint("Consider renaming this publication.");
PQclear(res);
PQfinish(conn);
@@ -1218,9 +1254,9 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
PQclear(res);
resetPQExpBuffer(str);
- pg_log_info("creating publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+ pg_log_info("creating publication \"%s\" on database \"%s\"", pubname, perdb->dbname);
- appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", dbinfo->pubname);
+ appendPQExpBuffer(str, "CREATE PUBLICATION %s FOR ALL TABLES", pubname);
pg_log_debug("command is: %s", str->data);
@@ -1228,13 +1264,13 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
pg_log_error("could not create publication \"%s\" on database \"%s\": %s",
- dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ pubname, perdb->dbname, PQerrorMessage(conn));;
PQfinish(conn);
exit(1);
}
/* for cleanup purposes */
- dbinfo->made_publication = true;
+ perdb->made_publication = true;
PQclear(res);
destroyPQExpBuffer(str);
@@ -1244,25 +1280,29 @@ create_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* Remove publication if it couldn't finish all steps.
*/
static void
-drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
+drop_publication(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("dropping publication \"%s\" on database \"%s\"", dbinfo->pubname, dbinfo->dbname);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
+ pg_log_info("dropping publication \"%s\" on database \"%s\"",
+ pubname, perdb->dbname);
- appendPQExpBuffer(str, "DROP PUBLICATION %s", dbinfo->pubname);
+ appendPQExpBuffer(str, "DROP PUBLICATION %s", pubname);
pg_log_debug("command is: %s", str->data);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop publication \"%s\" on database \"%s\": %s", dbinfo->pubname, dbinfo->dbname, PQerrorMessage(conn));
+ pg_log_error("could not drop publication \"%s\" on database \"%s\": %s",
+ pubname, perdb->dbname, PQerrorMessage(conn));
PQclear(res);
-
destroyPQExpBuffer(str);
}
@@ -1279,19 +1319,30 @@ drop_publication(PGconn *conn, LogicalRepInfo *dbinfo)
* initial location.
*/
static void
-create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+create_subscription(PGconn *conn, StandbyInfo *standby,
+ PrimaryInfo *primary,
+ LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
+ char pubname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("creating subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+ get_publication_name(perdb->oid, pubname, NAMEDATALEN);
+
+ pg_log_info("creating subscription \"%s\" on database \"%s\"", subname,
+ perdb->dbname);
appendPQExpBuffer(str,
"CREATE SUBSCRIPTION %s CONNECTION '%s' PUBLICATION %s "
"WITH (create_slot = false, copy_data = false, enabled = false)",
- dbinfo->subname, dbinfo->pubconninfo, dbinfo->pubname);
+ subname,
+ concat_conninfo_dbname(primary->base_conninfo,
+ perdb->dbname),
+ pubname);
pg_log_debug("command is: %s", str->data);
@@ -1299,13 +1350,13 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
pg_log_error("could not create subscription \"%s\" on database \"%s\": %s",
- dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ subname, perdb->dbname, PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
/* for cleanup purposes */
- dbinfo->made_subscription = true;
+ perdb->made_subscription = true;
PQclear(res);
destroyPQExpBuffer(str);
@@ -1315,22 +1366,27 @@ create_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* Remove subscription if it couldn't finish all steps.
*/
static void
-drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+drop_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("dropping subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
+ pg_log_info("dropping subscription \"%s\" on database \"%s\"",
+ subname, perdb->dbname);
- appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", dbinfo->subname);
+ appendPQExpBuffer(str, "DROP SUBSCRIPTION %s", subname);
pg_log_debug("command is: %s", str->data);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s", dbinfo->subname, dbinfo->dbname, PQerrorMessage(conn));
+ pg_log_error("could not drop subscription \"%s\" on database \"%s\": %s",
+ subname, perdb->dbname, PQerrorMessage(conn));
PQclear(res);
@@ -1348,18 +1404,23 @@ drop_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
* printing purposes.
*/
static void
-set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
+set_replication_progress(PGconn *conn, LogicalRepPerdbInfo *perdb,
+ const char *lsn)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
Oid suboid;
char originname[NAMEDATALEN];
char lsnstr[17 + 1]; /* MAXPG_LSNLEN = 17 */
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+
appendPQExpBuffer(str,
- "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'", dbinfo->subname);
+ "SELECT oid FROM pg_catalog.pg_subscription WHERE subname = '%s'",
+ subname);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
@@ -1392,7 +1453,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
PQclear(res);
pg_log_info("setting the replication progress (node name \"%s\" ; LSN %s) on database \"%s\"",
- originname, lsnstr, dbinfo->dbname);
+ originname, lsnstr, perdb->dbname);
resetPQExpBuffer(str);
appendPQExpBuffer(str,
@@ -1404,7 +1465,7 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
pg_log_error("could not set replication progress for the subscription \"%s\": %s",
- dbinfo->subname, PQresultErrorMessage(res));
+ subname, PQresultErrorMessage(res));
PQfinish(conn);
exit(1);
}
@@ -1421,24 +1482,27 @@ set_replication_progress(PGconn *conn, LogicalRepInfo *dbinfo, const char *lsn)
* of this setup.
*/
static void
-enable_subscription(PGconn *conn, LogicalRepInfo *dbinfo)
+enable_subscription(PGconn *conn, LogicalRepPerdbInfo *perdb)
{
PQExpBuffer str = createPQExpBuffer();
PGresult *res;
+ char subname[NAMEDATALEN];
Assert(conn != NULL);
- pg_log_info("enabling subscription \"%s\" on database \"%s\"", dbinfo->subname, dbinfo->dbname);
+ get_subscription_name(perdb->oid, (int) getpid(), subname, NAMEDATALEN);
+ pg_log_info("enabling subscription \"%s\" on database \"%s\"", subname,
+ perdb->dbname);
- appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", dbinfo->subname);
+ appendPQExpBuffer(str, "ALTER SUBSCRIPTION %s ENABLE", subname);
pg_log_debug("command is: %s", str->data);
res = PQexec(conn, str->data);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
- pg_log_error("could not enable subscription \"%s\": %s", dbinfo->subname,
- PQerrorMessage(conn));
+ pg_log_error("could not enable subscription \"%s\": %s", subname,
+ PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
@@ -1468,16 +1532,10 @@ main(int argc, char **argv)
int option_index;
char *base_dir;
- char *server_start_log;
int len;
- char *pub_base_conninfo = NULL;
- char *sub_base_conninfo = NULL;
char *dbname_conninfo = NULL;
- char temp_replslot[NAMEDATALEN] = {0};
- uint64 pub_sysid;
- uint64 sub_sysid;
struct stat statbuf;
PGconn *conn;
@@ -1527,7 +1585,7 @@ main(int argc, char **argv)
switch (c)
{
case 'D':
- subscriber_dir = pg_strdup(optarg);
+ standby.pgdata = pg_strdup(optarg);
break;
case 'S':
sub_conninfo_str = pg_strdup(optarg);
@@ -1537,7 +1595,7 @@ main(int argc, char **argv)
if (!simple_string_list_member(&database_names, optarg))
{
simple_string_list_append(&database_names, optarg);
- num_dbs++;
+ dbarr.ndbs++;
}
break;
case 'n':
@@ -1573,7 +1631,7 @@ main(int argc, char **argv)
/*
* Required arguments
*/
- if (subscriber_dir == NULL)
+ if (standby.pgdata == NULL)
{
pg_log_error("no subscriber data directory specified");
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1586,8 +1644,8 @@ main(int argc, char **argv)
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
exit(1);
}
- sub_base_conninfo = get_base_conninfo(sub_conninfo_str, dbname_conninfo);
- if (sub_base_conninfo == NULL)
+ standby.base_conninfo = get_base_conninfo(sub_conninfo_str, dbname_conninfo);
+ if (standby.base_conninfo == NULL)
exit(1);
if (database_names.head == NULL)
@@ -1602,7 +1660,7 @@ main(int argc, char **argv)
if (dbname_conninfo)
{
simple_string_list_append(&database_names, dbname_conninfo);
- num_dbs++;
+ dbarr.ndbs++;
pg_log_info("database \"%s\" was extracted from the subscriber connection string",
dbname_conninfo);
@@ -1616,20 +1674,20 @@ main(int argc, char **argv)
}
/* Obtain a connection string from the target */
- pub_base_conninfo = get_primary_conninfo(sub_base_conninfo);
+ primary.base_conninfo = get_primary_conninfo(&standby);
/*
* Get the absolute path of pg_ctl and pg_resetwal on the subscriber.
*/
- if (!get_exec_path(argv[0]))
+ if (!get_exec_base_path(argv[0]))
exit(1);
/* rudimentary check for a data directory. */
- if (!check_data_directory(subscriber_dir))
+ if (!check_data_directory(standby.pgdata))
exit(1);
- /* Store database information for publisher and subscriber. */
- dbinfo = store_pub_sub_info(pub_base_conninfo, sub_base_conninfo);
+ /* Store database information to dbarr */
+ store_db_names(&dbarr.perdb, dbarr.ndbs);
/* Register a function to clean up objects in case of failure. */
atexit(cleanup_objects_atexit);
@@ -1638,9 +1696,9 @@ main(int argc, char **argv)
* Check if the subscriber data directory has the same system identifier
* than the publisher data directory.
*/
- pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
- sub_sysid = get_control_from_datadir(subscriber_dir);
- if (pub_sysid != sub_sysid)
+ get_sysid_for_primary(&primary, dbarr.perdb[0].dbname);
+ get_sysid_for_standby(&standby);
+ if (primary.sysid != standby.sysid)
{
pg_log_error("subscriber data directory is not a copy of the source database cluster");
exit(1);
@@ -1650,7 +1708,7 @@ main(int argc, char **argv)
* Create the output directory to store any data generated by this tool.
*/
base_dir = (char *) pg_malloc0(MAXPGPATH);
- len = snprintf(base_dir, MAXPGPATH, "%s/%s", subscriber_dir, PGS_OUTPUT_DIR);
+ len = snprintf(base_dir, MAXPGPATH, "%s/%s", standby.pgdata, PGS_OUTPUT_DIR);
if (len >= MAXPGPATH)
{
pg_log_error("directory path for subscriber is too long");
@@ -1663,10 +1721,10 @@ main(int argc, char **argv)
exit(1);
}
- server_start_log = server_logfile_name(subscriber_dir);
+ standby.server_log = server_logfile_name(standby.pgdata);
/* subscriber PID file. */
- snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", subscriber_dir);
+ snprintf(pidfile, MAXPGPATH, "%s/postmaster.pid", standby.pgdata);
/*
* The standby server must be running. That's because some checks will be
@@ -1679,7 +1737,7 @@ main(int argc, char **argv)
/*
* Check if the standby server is ready for logical replication.
*/
- if (!check_subscriber(dbinfo))
+ if (!check_subscriber(&standby, &dbarr))
exit(1);
/*
@@ -1688,7 +1746,7 @@ main(int argc, char **argv)
* relies on check_subscriber() to obtain the primary_slot_name.
* That's why it is called after it.
*/
- if (!check_publisher(dbinfo))
+ if (!check_publisher(&primary, &dbarr))
exit(1);
/*
@@ -1697,13 +1755,13 @@ main(int argc, char **argv)
* if the primary slot is in use. We could use an extra connection for
* it but it doesn't seem worth.
*/
- if (!setup_publisher(dbinfo))
+ if (!setup_publisher(&primary, &dbarr))
exit(1);
/* Stop the standby server. */
pg_log_info("standby is up and running");
pg_log_info("stopping the server to start the transformation steps");
- stop_standby_server(pg_ctl_path, subscriber_dir);
+ stop_standby_server(&standby);
}
else
{
@@ -1732,11 +1790,10 @@ main(int argc, char **argv)
* consistent LSN but it should be changed after adding pg_basebackup
* support.
*/
- conn = connect_database(dbinfo[0].pubconninfo);
+ conn = connect_database(primary.base_conninfo, dbarr.perdb[0].dbname);
if (conn == NULL)
exit(1);
- consistent_lsn = create_logical_replication_slot(conn, &dbinfo[0],
- temp_replslot);
+ consistent_lsn = create_logical_replication_slot(conn, true, &dbarr.perdb[0]);
/*
* Write recovery parameters.
@@ -1752,7 +1809,7 @@ main(int argc, char **argv)
appendPQExpBuffer(recoveryconfcontents, "recovery_target_lsn = '%s'\n",
consistent_lsn);
- WriteRecoveryConfig(conn, subscriber_dir, recoveryconfcontents);
+ WriteRecoveryConfig(conn, standby.pgdata, recoveryconfcontents);
disconnect_database(conn);
@@ -1762,12 +1819,12 @@ main(int argc, char **argv)
* Start subscriber and wait until accepting connections.
*/
pg_log_info("starting the subscriber");
- start_standby_server(pg_ctl_path, subscriber_dir, server_start_log);
+ start_standby_server(&standby);
/*
* Waiting the subscriber to be promoted.
*/
- wait_for_end_recovery(dbinfo[0].subconninfo);
+ wait_for_end_recovery(&standby, dbarr.perdb[0].dbname);
/*
* Create the subscription for each database on subscriber. It does not
@@ -1776,7 +1833,7 @@ main(int argc, char **argv)
* set_replication_progress). It also cleans up publications created by
* this tool and replication to the standby.
*/
- if (!setup_subscriber(dbinfo, consistent_lsn))
+ if (!setup_subscriber(&standby, &primary, &dbarr, consistent_lsn))
exit(1);
/*
@@ -1785,12 +1842,15 @@ main(int argc, char **argv)
* XXX we might not fail here. Instead, we provide a warning so the user
* eventually drops this replication slot later.
*/
- if (primary_slot_name != NULL)
+ if (standby.primary_slot_name != NULL)
{
- conn = connect_database(dbinfo[0].pubconninfo);
+ char *primary_slot_name = standby.primary_slot_name;
+ LogicalRepPerdbInfo *perdb = &dbarr.perdb[0];
+
+ conn = connect_database(primary.base_conninfo, perdb->dbname);
if (conn != NULL)
{
- drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ drop_replication_slot(conn, perdb, primary_slot_name);
}
else
{
@@ -1804,12 +1864,12 @@ main(int argc, char **argv)
* Stop the subscriber.
*/
pg_log_info("stopping the subscriber");
- stop_standby_server(pg_ctl_path, subscriber_dir);
+ stop_standby_server(&standby);
/*
* Change system identifier.
*/
- modify_sysid(pg_resetwal_path, subscriber_dir);
+ modify_sysid(standby.bindir, standby.pgdata);
cleanup:
/*
@@ -1817,7 +1877,7 @@ cleanup:
* not run successfully. Otherwise, log file is removed.
*/
if (!retain)
- unlink(server_start_log);
+ unlink(standby.server_log);
success = true;
diff --git a/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl b/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
index a9d03acc87..856bf0de3c 100644
--- a/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
+++ b/src/bin/pg_basebackup/t/041_pg_createsubscriber_standby.pl
@@ -88,7 +88,7 @@ command_ok(
'--pgdata', $node_s->data_dir,
'--subscriber-server', $node_s->connstr('pg1'),
'--database', 'pg1',
- '--database', 'pg2'
+ '--database', 'pg2', '-r'
],
'run pg_createsubscriber on node S');
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index f51f1ff23f..3b1ec3fce1 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1505,9 +1505,10 @@ LogicalRepBeginData
LogicalRepCommitData
LogicalRepCommitPreparedTxnData
LogicalRepCtxStruct
-LogicalRepInfo
LogicalRepMsgType
LogicalRepPartMapEntry
+LogicalRepPerdbInfo
+LogicalRepPerdbInfoArr
LogicalRepPreparedTxnData
LogicalRepRelId
LogicalRepRelMapEntry
@@ -1886,6 +1887,7 @@ PREDICATELOCK
PREDICATELOCKTAG
PREDICATELOCKTARGET
PREDICATELOCKTARGETTAG
+PrimaryInfo
PROCESS_INFORMATION
PROCLOCK
PROCLOCKTAG
@@ -2461,6 +2463,7 @@ SQLValueFunctionOp
SSL
SSLExtensionInfoContext
SSL_CTX
+StandbyInfo
STARTUPINFO
STRLEN
SV
--
2.43.0
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 02:01 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 09:26 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 12:52 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
@ 2024-01-31 14:25 ` Fabrízio de Royes Mello <[email protected]>
2024-01-31 14:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Fabrízio de Royes Mello @ 2024-01-31 14:25 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: Euler Taveira <[email protected]>; [email protected] <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Shlok Kyal <[email protected]>
On Wed, Jan 31, 2024 at 9:52 AM Hayato Kuroda (Fujitsu) <
[email protected]> wrote:
>
> Dear Euler,
>
> I extracted some review comments which may require many efforts. I hope
this makes them
> easy to review.
>
> 0001: not changed from yours.
> 0002: avoid to use replication connections. Source: comment #3[1]
> 0003: Remove -P option and use primary_conninfo instead. Source: [2]
> 0004: Exit earlier when dry_run is specified. Source: [3]
> 0005: Refactor data structures. Source: [4]
>
> [1]:
https://www.postgresql.org/message-id/TY3PR01MB9889593399165B9A04106741F5662%40TY3PR01MB9889.jpnprd0...
> [2]:
https://www.postgresql.org/message-id/TY3PR01MB98897C85700C6DF942D2D0A3F5792%40TY3PR01MB9889.jpnprd0...
> [3]:
https://www.postgresql.org/message-id/TY3PR01MB98897C85700C6DF942D2D0A3F5792%40TY3PR01MB9889.jpnprd0...
> [4]:
https://www.postgresql.org/message-id/TY3PR01MB9889C362FF76102C88FA1C29F56F2%40TY3PR01MB9889.jpnprd0...
>
Hey folks,
Jumping into this a bit late here... I'm trying a simple
pg_createsubscriber but getting an error:
~/pgsql took 19s
✦ ➜ pg_createsubscriber -d fabrizio -r -D /tmp/replica5434 -S 'host=/tmp
port=5434'
pg_createsubscriber: error: could not create subscription
"pg_createsubscriber_16384_695617" on database "fabrizio": ERROR: syntax
error at or near "/"
LINE 1: ..._16384_695617 CONNECTION 'user=fabrizio passfile='/home/fabr...
^
pg_createsubscriber: error: could not drop replication slot
"pg_createsubscriber_16384_695617" on database "fabrizio":
pg_createsubscriber: error: could not drop replication slot
"pg_subscriber_695617_startpoint" on database "fabrizio": ERROR:
replication slot "pg_subscriber_695617_startpoint" does not exist
And the LOG contains the following:
~/pgsql took 12s
✦ ➜ cat
/tmp/replica5434/pg_createsubscriber_output.d/server_start_20240131T110318.730.log
2024-01-31 11:03:19.138 -03 [695632] LOG: starting PostgreSQL 17devel on
x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0,
64-bit
2024-01-31 11:03:19.138 -03 [695632] LOG: listening on IPv6 address "::1",
port 5434
2024-01-31 11:03:19.138 -03 [695632] LOG: listening on IPv4 address
"127.0.0.1", port 5434
2024-01-31 11:03:19.158 -03 [695632] LOG: listening on Unix socket
"/tmp/.s.PGSQL.5434"
2024-01-31 11:03:19.179 -03 [695645] LOG: database system was shut down in
recovery at 2024-01-31 11:03:18 -03
2024-01-31 11:03:19.180 -03 [695645] LOG: entering standby mode
2024-01-31 11:03:19.192 -03 [695645] LOG: redo starts at 0/4000028
2024-01-31 11:03:19.198 -03 [695645] LOG: consistent recovery state
reached at 0/504DB08
2024-01-31 11:03:19.198 -03 [695645] LOG: invalid record length at
0/504DB08: expected at least 24, got 0
2024-01-31 11:03:19.198 -03 [695632] LOG: database system is ready to
accept read-only connections
2024-01-31 11:03:19.215 -03 [695646] LOG: started streaming WAL from
primary at 0/5000000 on timeline 1
2024-01-31 11:03:29.587 -03 [695645] LOG: recovery stopping after WAL
location (LSN) "0/504F260"
2024-01-31 11:03:29.587 -03 [695645] LOG: redo done at 0/504F260 system
usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 10.39 s
2024-01-31 11:03:29.587 -03 [695645] LOG: last completed transaction was
at log time 2024-01-31 11:03:18.761544-03
2024-01-31 11:03:29.587 -03 [695646] FATAL: terminating walreceiver
process due to administrator command
2024-01-31 11:03:29.598 -03 [695645] LOG: selected new timeline ID: 2
2024-01-31 11:03:29.680 -03 [695645] LOG: archive recovery complete
2024-01-31 11:03:29.690 -03 [695643] LOG: checkpoint starting:
end-of-recovery immediate wait
2024-01-31 11:03:29.795 -03 [695643] LOG: checkpoint complete: wrote 51
buffers (0.3%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.021 s,
sync=0.034 s, total=0.115 s; sync files=17, longest=0.011 s, average=0.002
s; distance=16700 kB, estimate=16700 kB; lsn=0/504F298, redo lsn=0/504F298
2024-01-31 11:03:29.805 -03 [695632] LOG: database system is ready to
accept connections
2024-01-31 11:03:30.332 -03 [695658] ERROR: syntax error at or near "/" at
character 90
2024-01-31 11:03:30.332 -03 [695658] STATEMENT: CREATE SUBSCRIPTION
pg_createsubscriber_16384_695617 CONNECTION 'user=fabrizio
passfile='/home/fabrizio/.pgpass' channel_binding=prefer host=localhost
port=5432 sslmode=prefer sslcompression=0 sslcertmode=allow sslsni=1
ssl_min_protocol_version=TLSv1.2 gssencmode=disable krbsrvname=postgres
gssdelegation=0 target_session_attrs=any load_balance_hosts=disable
dbname=fabrizio' PUBLICATION pg_createsubscriber_16384 WITH (create_slot =
false, copy_data = false, enabled = false)
Seems we need to escape connection params similar we do in dblink [1]
Regards,
[1]
https://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=contrib/dblink/dblink.c;h=19a362526d21d...
--
Fabrízio de Royes Mello
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 02:01 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 09:26 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 12:52 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 14:25 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
@ 2024-01-31 14:34 ` Euler Taveira <[email protected]>
2024-01-31 14:55 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Euler Taveira @ 2024-01-31 14:34 UTC (permalink / raw)
To: Fabrízio de Royes Mello <[email protected]>; [email protected] <[email protected]>; +Cc: [email protected] <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Shlok Kyal <[email protected]>
On Wed, Jan 31, 2024, at 11:25 AM, Fabrízio de Royes Mello wrote:
> Jumping into this a bit late here... I'm trying a simple pg_createsubscriber but getting an error:
Try v11. It seems v12-0002 is not correct.
> Seems we need to escape connection params similar we do in dblink [1]
I think it is a consequence of v12-0003. I didn't review v12 yet but although I
have added a comment saying it might be possible to use primary_conninfo, I'm
not 100% convinced that's the right direction.
/*
* TODO use primary_conninfo (if available) from subscriber and
* extract publisher connection string. Assume that there are
* identical entries for physical and logical replication. If there is
* not, we would fail anyway.
*/
--
Euler Taveira
EDB https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 02:01 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 09:26 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 12:52 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 14:25 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
2024-01-31 14:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
@ 2024-01-31 14:55 ` Fabrízio de Royes Mello <[email protected]>
2024-01-31 15:37 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Fabrízio de Royes Mello @ 2024-01-31 14:55 UTC (permalink / raw)
To: Euler Taveira <[email protected]>; +Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Shlok Kyal <[email protected]>
On Wed, Jan 31, 2024 at 11:35 AM Euler Taveira <[email protected]> wrote:
>
> On Wed, Jan 31, 2024, at 11:25 AM, Fabrízio de Royes Mello wrote:
>
> Jumping into this a bit late here... I'm trying a simple
pg_createsubscriber but getting an error:
>
>
> Try v11. It seems v12-0002 is not correct.
Using v11 I'm getting this error:
~/pgsql took 22s
✦ ➜ pg_createsubscriber -d fabrizio -r -D /tmp/replica5434 -S 'host=/tmp
port=5434' -P 'host=/tmp port=5432'
NOTICE: changed the failover state of replication slot
"pg_createsubscriber_16384_706609" on publisher to false
pg_createsubscriber: error: could not drop replication slot
"pg_createsubscriber_706609_startpoint" on database "fabrizio": ERROR:
replication slot "pg_createsubscriber_706609_startpoint" does not exist
Write-ahead log reset
Attached the output log.
Regards,
--
Fabrízio de Royes Mello
Attachments:
[text/x-log] server_start_20240131T115324.735.log (3.2K, ../../CAFcNs+rSG9DcEewsoA=85DXhSRh+nyKrrcr64FEDytcZf6QaEQ@mail.gmail.com/3-server_start_20240131T115324.735.log)
download | inline:
2024-01-31 11:53:31.882 -03 [706626] LOG: starting PostgreSQL 17devel on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
2024-01-31 11:53:31.882 -03 [706626] LOG: listening on IPv6 address "::1", port 5434
2024-01-31 11:53:31.882 -03 [706626] LOG: listening on IPv4 address "127.0.0.1", port 5434
2024-01-31 11:53:31.990 -03 [706626] LOG: listening on Unix socket "/tmp/.s.PGSQL.5434"
2024-01-31 11:53:32.082 -03 [706639] LOG: database system was shut down in recovery at 2024-01-31 11:53:30 -03
2024-01-31 11:53:32.084 -03 [706639] LOG: entering standby mode
2024-01-31 11:53:32.100 -03 [706639] LOG: redo starts at 0/4000028
2024-01-31 11:53:32.189 -03 [706639] LOG: consistent recovery state reached at 0/69EFF00
2024-01-31 11:53:32.189 -03 [706639] LOG: invalid record length at 0/69F01D0: expected at least 24, got 0
2024-01-31 11:53:32.189 -03 [706626] LOG: database system is ready to accept read-only connections
2024-01-31 11:53:32.200 -03 [706640] LOG: started streaming WAL from primary at 0/6000000 on timeline 1
2024-01-31 11:53:32.218 -03 [706639] LOG: recovery stopping after WAL location (LSN) "0/6A1F4C0"
2024-01-31 11:53:32.218 -03 [706639] LOG: redo done at 0/6A1F4C0 system usage: CPU: user: 0.07 s, system: 0.01 s, elapsed: 0.11 s
2024-01-31 11:53:32.219 -03 [706639] LOG: last completed transaction was at log time 2024-01-31 11:53:31.745182-03
2024-01-31 11:53:32.220 -03 [706640] FATAL: terminating walreceiver process due to administrator command
2024-01-31 11:53:32.266 -03 [706639] LOG: selected new timeline ID: 2
2024-01-31 11:53:32.390 -03 [706639] LOG: archive recovery complete
2024-01-31 11:53:32.403 -03 [706637] LOG: checkpoint starting: end-of-recovery immediate wait
2024-01-31 11:53:32.809 -03 [706637] LOG: checkpoint complete: wrote 2141 buffers (13.1%); 0 WAL file(s) added, 0 removed, 2 recycled; write=0.067 s, sync=0.198 s, total=0.419 s; sync files=51, longest=0.018 s, average=0.004 s; distance=43133 kB, estimate=43133 kB; lsn=0/6A1F4F8, redo lsn=0/6A1F4F8
2024-01-31 11:53:32.827 -03 [706626] LOG: database system is ready to accept connections
2024-01-31 11:53:33.398 -03 [706626] LOG: received fast shutdown request
2024-01-31 11:53:33.409 -03 [706650] LOG: logical replication apply worker for subscription "pg_createsubscriber_16384_706609" has started
2024-01-31 11:53:33.414 -03 [706626] LOG: aborting any active transactions
2024-01-31 11:53:33.414 -03 [706650] FATAL: terminating logical replication worker due to administrator command
2024-01-31 11:53:33.415 -03 [706626] LOG: background worker "logical replication launcher" (PID 706645) exited with exit code 1
2024-01-31 11:53:33.415 -03 [706626] LOG: background worker "logical replication apply worker" (PID 706650) exited with exit code 1
2024-01-31 11:53:33.415 -03 [706637] LOG: shutting down
2024-01-31 11:53:33.438 -03 [706637] LOG: checkpoint starting: shutdown immediate
2024-01-31 11:53:33.642 -03 [706637] LOG: checkpoint complete: wrote 23 buffers (0.1%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.052 s, sync=0.047 s, total=0.228 s; sync files=14, longest=0.011 s, average=0.004 s; distance=3 kB, estimate=38820 kB; lsn=0/6A20270, redo lsn=0/6A20270
2024-01-31 11:53:33.648 -03 [706626] LOG: database system is shut down
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 02:01 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 09:26 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 12:52 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 14:25 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
2024-01-31 14:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-31 14:55 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
@ 2024-01-31 15:37 ` Euler Taveira <[email protected]>
2024-01-31 17:39 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Euler Taveira @ 2024-01-31 15:37 UTC (permalink / raw)
To: Fabrízio de Royes Mello <[email protected]>; +Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Shlok Kyal <[email protected]>
On Wed, Jan 31, 2024, at 11:55 AM, Fabrízio de Royes Mello wrote:
>
> On Wed, Jan 31, 2024 at 11:35 AM Euler Taveira <[email protected]> wrote:
> >
> > On Wed, Jan 31, 2024, at 11:25 AM, Fabrízio de Royes Mello wrote:
> >
> > Jumping into this a bit late here... I'm trying a simple pg_createsubscriber but getting an error:
> >
> >
> > Try v11. It seems v12-0002 is not correct.
>
> Using v11 I'm getting this error:
>
> ~/pgsql took 22s
> ✦ ➜ pg_createsubscriber -d fabrizio -r -D /tmp/replica5434 -S 'host=/tmp port=5434' -P 'host=/tmp port=5432'
> NOTICE: changed the failover state of replication slot "pg_createsubscriber_16384_706609" on publisher to false
> pg_createsubscriber: error: could not drop replication slot "pg_createsubscriber_706609_startpoint" on database "fabrizio": ERROR: replication slot "pg_createsubscriber_706609_startpoint" does not exist
> Write-ahead log reset
Hmm. I didn't try it with the failover patch that was recently applied. Did you
have any special configuration on primary?
--
Euler Taveira
EDB https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 02:01 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 09:26 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 12:52 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 14:25 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
2024-01-31 14:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-31 14:55 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
2024-01-31 15:37 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
@ 2024-01-31 17:39 ` Fabrízio de Royes Mello <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Fabrízio de Royes Mello @ 2024-01-31 17:39 UTC (permalink / raw)
To: Euler Taveira <[email protected]>; +Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>; Shlok Kyal <[email protected]>
On Wed, Jan 31, 2024 at 12:38 PM Euler Taveira <[email protected]> wrote:
>
>
> Hmm. I didn't try it with the failover patch that was recently applied.
Did you
> have any special configuration on primary?
>
Nothing special, here the configurations I've changed after bootstrap:
port = '5432'
wal_level = 'logical'
max_wal_senders = '8'
max_replication_slots = '6'
hot_standby_feedback = 'on'
max_prepared_transactions = '10'
max_locks_per_transaction = '512'
Regards,
--
Fabrízio de Royes Mello
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: speed up a logical replica setup
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
@ 2024-01-23 23:58 ` Euler Taveira <[email protected]>
1 sibling, 0 replies; 77+ messages in thread
From: Euler Taveira @ 2024-01-23 23:58 UTC (permalink / raw)
To: [email protected] <[email protected]>; '[email protected]' <[email protected]>; +Cc: Shlok Kyal <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Amit Kapila <[email protected]>
On Mon, Jan 22, 2024, at 4:06 AM, Hayato Kuroda (Fujitsu) wrote:
> I analyzed and found a reason. This is because publications are invisible for some transactions.
>
> As the first place, below operations were executed in this case.
> Tuples were inserted after getting consistent_lsn, but before starting the standby.
> After doing the workload, I confirmed again that the publication was created.
>
> 1. on primary, logical replication slots were created.
> 2. on primary, another replication slot was created.
> 3. ===on primary, some tuples were inserted. ===
> 4. on standby, a server process was started
> 5. on standby, the process waited until all changes have come.
> 6. on primary, publications were created.
> 7. on standby, subscriptions were created.
> 8. on standby, a replication progress for each subscriptions was set to given LSN (got at step2).
> =====pg_subscriber finished here=====
> 9. on standby, a server process was started again
> 10. on standby, subscriptions were enabled. They referred slots created at step1.
> 11. on primary, decoding was started but ERROR was raised.
Good catch! It is a design flaw.
> In this case, tuples were inserted *before creating publication*.
> So I thought that the decoded transaction could not see the publication because
> it was committed after insertions.
>
> One solution is to create a publication before creating a consistent slot.
> Changes which came before creating the slot were surely replicated to the standby,
> so upcoming transactions can see the object. We are planning to patch set to fix
> the issue in this approach.
I'll include a similar code in the next patch and also explain why we should
create the publication earlier. (I'm renaming
create_all_logical_replication_slots to setup_publisher and calling
create_publication from there and also adding the proposed GUC checks in it.)
--
Euler Taveira
EDB https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: trying again to get incremental backup
@ 2023-10-24 16:08 Robert Haas <[email protected]>
2023-10-30 14:45 ` Re: trying again to get incremental backup Robert Haas <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Robert Haas @ 2023-10-24 16:08 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: Dilip Kumar <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
On Tue, Oct 24, 2023 at 10:53 AM Peter Eisentraut <[email protected]> wrote:
> The easiest answer is to have it off by default. Let people figure out
> what works for them. There are various factors like storage, network,
> server performance, RTO that will determine what combination of full
> backup, incremental backup, and WAL replay will satisfy someone's
> requirements. I suppose tests could be set up to determine this to some
> degree. But we could also start slow and let people figure it out
> themselves. When pg_basebackup was added, it was also disabled by default.
>
> If we think that 7d is a good setting, then I would suggest to consider,
> like 10d. Otherwise, if you do a weekly incremental backup and you have
> a time change or a hiccup of some kind one day, you lose your backup
> sequence.
>
> Another possible answer is, like, 400 days? Because why not? What is a
> reasonable upper limit for this?
In concept, I don't think this should even be time-based. What you
should do is remove WAL summaries once you know that you've taken as
many incremental backups that might use them as you're ever going to
do. But PostgreSQL itself doesn't have any way of knowing what your
intended backup patterns are. If your incremental backup fails on
Monday night and you run it manually on Tuesday morning, you might
still rerun it as an incremental backup. If it fails every night for a
month and you finally realize and decide to intervene manually, maybe
you want a new full backup at that point. It's been a month. But on
the other hand maybe you don't. There's no time-based answer to this
question that is really correct, and I think it's quite possible that
your backup software might want to shut off time-based deletion
altogether and make its own decisions about when to nuke summaries.
However, I also don't think that's a great default setting. It could
easily lead to people wasting a bunch of disk space for no reason.
As far as the 7d value, I figured that nighty incremental backups
would be fairly common. If we think weekly incremental backups would
be common, then pushing this out to 10d would make sense. While
there's no reason you couldn't take an annual incremental backup, and
thus want a 400d value, it seems like a pretty niche use case.
Note that whether to remove summaries is a separate question from
whether to generate them in the first place. Right now, I have
wal_summarize_mb controlling whether they get generated in the first
place, but as I noted in another recent email, that isn't an entirely
satisfying solution.
> It looks like each file entry in the manifest takes about 150 bytes, so
> 1 GB would allow for 1024**3/150 = 7158278 files. That seems fine for now?
I suspect a few people have more files than that. They'll just have to
wait to use this feature until we get incremental JSON parsing (or
undo the decision to use JSON for the manifest).
> The current user experience of pg_basebackup is that it waits possibly a
> long time for a checkpoint, and there is an option to make it go faster,
> but there is no timeout AFAICT. Is this substantially different? Could
> we just let it wait forever?
We could. I installed the timeout because the first versions of the
feature were buggy, and I didn't like having my tests hang forever
with no indication of what had gone wrong. At least in my experience
so far, the time spent waiting for WAL summarization is typically
quite short, because only the WAL that needs to be summarized is
whatever was emitted since the last time it woke up up through the
start LSN of the backup. That's probably not much, and the next time
the summarizer wakes up, the file should appear within moments. So,
it's a little different from the checkpoint case, where long waits are
expected.
> Also, does waiting for checkpoint and WAL summarization happen in
> parallel? If so, what if it starts a checkpoint that might take 15 min
> to complete, and then after 60 seconds it kicks you off because the WAL
> summarization isn't ready. That might be wasteful.
It is not parallel. The trouble is, we don't really have any way to
know whether WAL summarization is going to fail for whatever reason.
We don't expect that to happen, but if somebody changes the
permissions on the WAL summary directory or attaches gdb to the WAL
summarizer process or something of that sort, it might.
We could check at the outset whether we seem to be really far behind
and emit a warning. For instance, if we're 1TB behind on WAL
summarization when the checkpoint is requested, chances are something
is busted and we're probably not going to catch up any time soon. We
could warn the user about that and let them make their own decision
about whether to cancel. But, that idea won't help in unattended
operation, and the threshold for "really far behind" is not very
clear. It might be better to wait until we get more experience with
how things actually fail before doing too much engineering here, but
I'm also open to suggestions.
> Is the WAL summary file format documented anywhere in your patch set
> yet? My only thought was, maybe the file format could be human-readable
> (more like backup_label) to avoid this. But maybe not.
The comment in blkreftable.c just above "#define BLOCKS_PER_CHUNK"
gives an overview of the format. I think that we probably don't want
to convert to a text format, because this format is extremely
space-efficient and very convenient to transfer between disk and
memory. We don't want to run out of memory when summarizing large
ranges of WAL, or when taking an incremental backup that requires
combining many individual summaries into a combined summary that tells
us what needs to be included in the backup.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: trying again to get incremental backup
2023-10-24 16:08 Re: trying again to get incremental backup Robert Haas <[email protected]>
@ 2023-10-30 14:45 ` Robert Haas <[email protected]>
2023-10-30 18:46 ` Re: trying again to get incremental backup Andres Freund <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Robert Haas @ 2023-10-30 14:45 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: Dilip Kumar <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
On Tue, Oct 24, 2023 at 12:08 PM Robert Haas <[email protected]> wrote:
> Note that whether to remove summaries is a separate question from
> whether to generate them in the first place. Right now, I have
> wal_summarize_mb controlling whether they get generated in the first
> place, but as I noted in another recent email, that isn't an entirely
> satisfying solution.
I did some more research on this. My conclusion is that I should
remove wal_summarize_mb and just have a GUC summarize_wal = on|off
that controls whether the summarizer runs at all. There will be one
summary file per checkpoint, no matter how far apart checkpoints are
or how large the summary gets. Below I'll explain the reasoning; let
me know if you disagree.
What I describe above would be a bad plan if it were realistically
possible for a summary file to get so large that it might run the
machine out of memory either when producing it or when trying to make
use of it for an incremental backup. This seems to be a somewhat
difficult scenario to create. So far, I haven't been able to generate
WAL summary files more than a few tens of megabytes in size, even when
summarizing 50+ GB of WAL per summary file. One reason why it's hard
to produce large summary files is because, for a single relation fork,
the WAL summary size converges to 1 bit per modified block when the
number of modified blocks is large. This means that, even if you have
a terabyte sized relation, you're looking at no more than perhaps 20MB
of summary data no matter how much of it gets modified. Now, somebody
could have a 30TB relation and then if they modify the whole thing
they could have the better part of a gigabyte of summary data for that
relation, but if you've got a 30TB table you probably have enough
memory that that's no big deal.
But, what if you have multiple relations? I initialized pgbench with a
scale factor of 30000 and also with 30000 partitions and did a 1-hour
run. I got 4 checkpoints during that time and each one produced an
approximately 16MB summary file. The efficiency here drops
considerably. For example, one of the files is 16495398 bytes and
records information on 7498403 modified blocks, which works out to
about 2.2 bytes per modified block. That's more than an order of
magnitude worse than what I got in the single-relation case, where the
summary file didn't even use two *bits* per modified block. But here
again, the file just isn't that big in absolute terms. To get a 1GB+
WAL summary file, you'd need to modify millions of relation forks,
maybe tens of millions, and most installations aren't even going to
have that many relation forks, let alone be modifying them all
frequently.
My conclusion here is that it's pretty hard to have a database where
WAL summarization is going to use too much memory. I wouldn't be
terribly surprised if there are some extreme cases where it happens,
but those databases probably aren't great candidates for incremental
backup anyway. They're probably databases with millions of relations
and frequent, widely-scattered modifications to those relations. And
if you have that kind of high turnover rate then incremental backup
isn't going to as helpful anyway, so there's probably no reason to
enable WAL summarization in the first place. Maybe if you have that
plus in the same database cluster you have a 100TB of completely
static data that is never modified, and if you also do all of this on
a pretty small machine, then you can find a case where incremental
backup would have worked well but for the memory consumed by WAL
summarization.
But I think that's sufficiently niche that the current patch shouldn't
concern itself with such cases. If we find that they're common enough
to worry about, we might eventually want to do something to mitigate
them, but whether that thing looks anything like wal_summarize_mb
seems pretty unclear. So I conclude that it's a mistake to include
that GUC as currently designed and propose to replace it with a
Boolean as described above.
Comments?
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: trying again to get incremental backup
2023-10-24 16:08 Re: trying again to get incremental backup Robert Haas <[email protected]>
2023-10-30 14:45 ` Re: trying again to get incremental backup Robert Haas <[email protected]>
@ 2023-10-30 18:46 ` Andres Freund <[email protected]>
2023-11-06 20:36 ` Re: trying again to get incremental backup Robert Haas <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Andres Freund @ 2023-10-30 18:46 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Dilip Kumar <[email protected]>; pgsql-hackers
Hi,
On 2023-10-30 10:45:03 -0400, Robert Haas wrote:
> On Tue, Oct 24, 2023 at 12:08 PM Robert Haas <[email protected]> wrote:
> > Note that whether to remove summaries is a separate question from
> > whether to generate them in the first place. Right now, I have
> > wal_summarize_mb controlling whether they get generated in the first
> > place, but as I noted in another recent email, that isn't an entirely
> > satisfying solution.
>
> I did some more research on this. My conclusion is that I should
> remove wal_summarize_mb and just have a GUC summarize_wal = on|off
> that controls whether the summarizer runs at all. There will be one
> summary file per checkpoint, no matter how far apart checkpoints are
> or how large the summary gets. Below I'll explain the reasoning; let
> me know if you disagree.
> What I describe above would be a bad plan if it were realistically
> possible for a summary file to get so large that it might run the
> machine out of memory either when producing it or when trying to make
> use of it for an incremental backup. This seems to be a somewhat
> difficult scenario to create. So far, I haven't been able to generate
> WAL summary files more than a few tens of megabytes in size, even when
> summarizing 50+ GB of WAL per summary file. One reason why it's hard
> to produce large summary files is because, for a single relation fork,
> the WAL summary size converges to 1 bit per modified block when the
> number of modified blocks is large. This means that, even if you have
> a terabyte sized relation, you're looking at no more than perhaps 20MB
> of summary data no matter how much of it gets modified. Now, somebody
> could have a 30TB relation and then if they modify the whole thing
> they could have the better part of a gigabyte of summary data for that
> relation, but if you've got a 30TB table you probably have enough
> memory that that's no big deal.
I'm not particularly worried about the rewriting-30TB-table case - that'd also
generate >= 30TB of WAL most of the time. Which realistically is going to
trigger a few checkpoints, even on very big instances.
> But, what if you have multiple relations? I initialized pgbench with a
> scale factor of 30000 and also with 30000 partitions and did a 1-hour
> run. I got 4 checkpoints during that time and each one produced an
> approximately 16MB summary file.
Hm, I assume the pgbench run will be fairly massively bottlenecked on IO, due
to having to read data from disk, lots of full page write and having to write
out lots of data? I.e. we won't do all that many transactions during the 1h?
> To get a 1GB+ WAL summary file, you'd need to modify millions of relation
> forks, maybe tens of millions, and most installations aren't even going to
> have that many relation forks, let alone be modifying them all frequently.
I tried to find bad cases for a bit - and I am not worried. I wrote a pgbench
script to create 10k single-row relations in each script, ran that with 96
clients, checkpointed, and ran a pgbench script that updated the single row in
each table.
After creation of the relation WAL summarizer uses
LOG: level: 1; Wal Summarizer: 378433680 total in 43 blocks; 5628936 free (66 chunks); 372804744 used
and creates a 26MB summary file.
After checkpoint & updates WAL summarizer uses:
LOG: level: 1; Wal Summarizer: 369205392 total in 43 blocks; 5864536 free (26 chunks); 363340856 used
and creates a 26MB summary file.
Sure, 350MB ain't nothing, but simply just executing \dt in the database
created by this makes the backend use 260MB after. Which isn't going away,
whereas WAL summarizer drops its memory usage soon after.
> But I think that's sufficiently niche that the current patch shouldn't
> concern itself with such cases. If we find that they're common enough
> to worry about, we might eventually want to do something to mitigate
> them, but whether that thing looks anything like wal_summarize_mb
> seems pretty unclear. So I conclude that it's a mistake to include
> that GUC as currently designed and propose to replace it with a
> Boolean as described above.
After playing with this for a while, I don't see a reason for wal_summarize_mb
from a memory usage POV at least.
I wonder if there are use cases that might like to consume WAL summaries
before the next checkpoint? For those wal_summarize_mb likely wouldn't be a
good control, but they might want to request a summary file to be created at
some point?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: trying again to get incremental backup
2023-10-24 16:08 Re: trying again to get incremental backup Robert Haas <[email protected]>
2023-10-30 14:45 ` Re: trying again to get incremental backup Robert Haas <[email protected]>
2023-10-30 18:46 ` Re: trying again to get incremental backup Andres Freund <[email protected]>
@ 2023-11-06 20:36 ` Robert Haas <[email protected]>
2023-11-10 11:27 ` Re: trying again to get incremental backup Dilip Kumar <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Robert Haas @ 2023-11-06 20:36 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Dilip Kumar <[email protected]>; pgsql-hackers
On Mon, Oct 30, 2023 at 2:46 PM Andres Freund <[email protected]> wrote:
> After playing with this for a while, I don't see a reason for wal_summarize_mb
> from a memory usage POV at least.
Here's v8. Changes:
- Replace wal_summarize_mb GUC with summarize_wal = on | off.
- Document the summarize_wal and wal_summary_keep_time GUCs.
- Refuse to start with summarize_wal = on and wal_level = minimal.
- Increase default wal_summary_keep_time to 10d from 7d, per (what I
think was) a suggestion from Peter E.
- Fix fencepost errors when deciding which WAL summaries are needed
for a backup.
- Fix indentation damage.
- Standardize on ereport(DEBUG1, ...) in walsummarizer.c vs. various
more and less chatty things I had before.
- Include the timeline in some error messages because not having it
proved confusing.
- Be more consistent about ignoring the FSM fork.
- Fix a bug that could cause WAL summarization to error out when
switching timelines.
- Fix the division between the wal summarizer and incremental backup
patches so that the former passes tests without the latter.
- Fix some things that an older compiler didn't like, including adding
pg_attribute_printf in some places.
- Die with an error instead of crashing if someone feeds us a manifest
with no WAL ranges.
- Sort the block numbers that need to be read from a relation file
before reading them, so that we're certain to read them in ascending
order.
- Be more careful about computing the truncation_block_length of an
incremental file; don't do math on a block number that might be
InvalidBlockNumber.
- Fix pg_combinebackup so it doesn't fail when zero-filled blocks are
added to a relation between the prior backup and the incremental
backup.
- Improve the pg_combinebackup -d output so that it explains in detail
how it's carrying out reconstruction, to improve debuggability.
- Disable WAL summarization by default, but add a test patch to the
series to enable it, because running the whole test suite with it
turned on is good for bug-hunting.
- In pg_walsummary, zero a struct before using instead of starting
with arbitrary junk values.
To do list:
- Figure out whether to do something other than uploading the whole
summary, per discussion with Jakub Wartak.
- Decide what to do about the 60-second waiting-for-WAL-summarization timeout.
- Make incremental backup fail quickly if WAL summarization is not even enabled.
- Have pg_basebackup error out nicely if an incremental backup is
requested from an older server that can't do that.
- Add some kind of tests for pg_walsummary.
--
Robert Haas
EDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v8-0005-Add-new-pg_walsummary-tool.patch (17.0K, ../../CA+TgmoYiPn7GE_GFvGf+150XOt-Pkdm8h4rzy42jYCRrceUWhg@mail.gmail.com/2-v8-0005-Add-new-pg_walsummary-tool.patch)
download | inline diff:
From dfa9822958d39ef997c1e071239a2bdfac347bd1 Mon Sep 17 00:00:00 2001
From: Robert Haas <[email protected]>
Date: Wed, 25 Oct 2023 13:01:06 -0400
Subject: [PATCH v8 5/6] Add new pg_walsummary tool.
This can dump the contents of WAL summary files, either those in
pg_wal/summaries, or the INCREMENTAL_BACKUP files that are part of
an incremental backup proper.
XXX. Needs tests.
---
doc/src/sgml/ref/allfiles.sgml | 1 +
doc/src/sgml/ref/pg_walsummary.sgml | 122 +++++++++++
doc/src/sgml/reference.sgml | 1 +
src/bin/Makefile | 1 +
src/bin/meson.build | 1 +
src/bin/pg_walsummary/.gitignore | 1 +
src/bin/pg_walsummary/Makefile | 42 ++++
src/bin/pg_walsummary/meson.build | 24 +++
src/bin/pg_walsummary/pg_walsummary.c | 280 ++++++++++++++++++++++++++
src/tools/pgindent/typedefs.list | 2 +
10 files changed, 475 insertions(+)
create mode 100644 doc/src/sgml/ref/pg_walsummary.sgml
create mode 100644 src/bin/pg_walsummary/.gitignore
create mode 100644 src/bin/pg_walsummary/Makefile
create mode 100644 src/bin/pg_walsummary/meson.build
create mode 100644 src/bin/pg_walsummary/pg_walsummary.c
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index fda4690eab..4a42999b18 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -219,6 +219,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY pgtesttiming SYSTEM "pgtesttiming.sgml">
<!ENTITY pgupgrade SYSTEM "pgupgrade.sgml">
<!ENTITY pgwaldump SYSTEM "pg_waldump.sgml">
+<!ENTITY pgwalsummary SYSTEM "pg_walsummary.sgml">
<!ENTITY postgres SYSTEM "postgres-ref.sgml">
<!ENTITY psqlRef SYSTEM "psql-ref.sgml">
<!ENTITY reindexdb SYSTEM "reindexdb.sgml">
diff --git a/doc/src/sgml/ref/pg_walsummary.sgml b/doc/src/sgml/ref/pg_walsummary.sgml
new file mode 100644
index 0000000000..3a2122b067
--- /dev/null
+++ b/doc/src/sgml/ref/pg_walsummary.sgml
@@ -0,0 +1,122 @@
+<!--
+doc/src/sgml/ref/pg_walsummary.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="app-pgwalsummary">
+ <indexterm zone="app-pgwalsummary">
+ <primary>pg_walsummary</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle><application>pg_walsummary</application></refentrytitle>
+ <manvolnum>1</manvolnum>
+ <refmiscinfo>Application</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>pg_walsummary</refname>
+ <refpurpose>print contents of WAL summary files</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_walsummary</command>
+ <arg rep="repeat" choice="opt"><replaceable>option</replaceable></arg>
+ <arg rep="repeat"><replaceable>file</replaceable></arg>
+ </cmdsynopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+ <para>
+ <application>pg_walsummary</application> is used to print the contents of
+ WAL summary files. These binary files are found with the
+ <literal>pg_wal/summaries</literal> subdirectory of the data directory,
+ and can be converted to text using this tool. This is not ordinarily
+ necessary, since WAL summary files primary exist to support
+ <link linkend="backup-incremental-backup">incremental backup</link>,
+ but it may be useful for debugging purposes.
+ </para>
+
+ <para>
+ A WAL summary file is indexed by tablespace OID, relation OID, and relation
+ fork. For each relation fork, it stores the list of blocks that were
+ modified by WAL within the range summarized in the file. It can also
+ store a "limit block," which is 0 if the relation fork was created or
+ truncated within the relevant WAL range, and otherwise the shortest length
+ to which the relation fork was truncated. If the relation fork was not
+ created, deleted, or truncated within the relevant WAL range, the limit
+ block is undefined or infinite and will not be printed by this tool.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Options</title>
+
+ <para>
+ <variablelist>
+ <varlistentry>
+ <term><option>-i</option></term>
+ <term><option>--indivudual</option></term>
+ <listitem>
+ <para>
+ By default, <literal>pg_walsummary</literal> prints one line of output
+ for each range of one or more consecutive modified blocks. This can
+ make the output a lot briefer, since a relation where all blocks from
+ 0 through 999 were modified will produce only one line of output rather
+ than 1000 separate lines. This option requests a separate line of
+ output for every modified block.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-q</option></term>
+ <term><option>--quiet</option></term>
+ <listitem>
+ <para>
+ Do not print any output, except for errors. This can be useful
+ when you want to know whether a WAL summary file can be successfully
+ parsed but don't care about the contents.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-?</option></term>
+ <term><option>--help</option></term>
+ <listitem>
+ <para>
+ Shows help about <application>pg_walsummary</application> command line
+ arguments, and exits.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Environment</title>
+
+ <para>
+ The environment variable <envar>PG_COLOR</envar> specifies whether to use
+ color in diagnostic messages. Possible values are
+ <literal>always</literal>, <literal>auto</literal> and
+ <literal>never</literal>.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="app-pgbasebackup"/></member>
+ <member><xref linkend="app-pgcombinebackup"/></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index a07d2b5e01..aa94f6adf6 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -289,6 +289,7 @@
&pgtesttiming;
&pgupgrade;
&pgwaldump;
+ &pgwalsummary;
&postgres;
</reference>
diff --git a/src/bin/Makefile b/src/bin/Makefile
index aa2210925e..f98f58d39e 100644
--- a/src/bin/Makefile
+++ b/src/bin/Makefile
@@ -31,6 +31,7 @@ SUBDIRS = \
pg_upgrade \
pg_verifybackup \
pg_waldump \
+ pg_walsummary \
pgbench \
psql \
scripts
diff --git a/src/bin/meson.build b/src/bin/meson.build
index 4cb6fd59bb..d1e9ef4409 100644
--- a/src/bin/meson.build
+++ b/src/bin/meson.build
@@ -17,6 +17,7 @@ subdir('pg_test_timing')
subdir('pg_upgrade')
subdir('pg_verifybackup')
subdir('pg_waldump')
+subdir('pg_walsummary')
subdir('pgbench')
subdir('pgevent')
subdir('psql')
diff --git a/src/bin/pg_walsummary/.gitignore b/src/bin/pg_walsummary/.gitignore
new file mode 100644
index 0000000000..d71ec192fa
--- /dev/null
+++ b/src/bin/pg_walsummary/.gitignore
@@ -0,0 +1 @@
+pg_walsummary
diff --git a/src/bin/pg_walsummary/Makefile b/src/bin/pg_walsummary/Makefile
new file mode 100644
index 0000000000..852f7208f6
--- /dev/null
+++ b/src/bin/pg_walsummary/Makefile
@@ -0,0 +1,42 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/bin/pg_walsummary
+#
+# Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/bin/pg_walsummary/Makefile
+#
+#-------------------------------------------------------------------------
+
+PGFILEDESC = "pg_walsummary - print contents of WAL summary files"
+PGAPPICON=win32
+
+subdir = src/bin/pg_walsummary
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
+LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils
+
+OBJS = \
+ $(WIN32RES) \
+ pg_walsummary.o
+
+all: pg_walsummary
+
+pg_walsummary: $(OBJS) | submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
+
+install: all installdirs
+ $(INSTALL_PROGRAM) pg_walsummary$(X) '$(DESTDIR)$(bindir)/pg_walsummary$(X)'
+
+installdirs:
+ $(MKDIR_P) '$(DESTDIR)$(bindir)'
+
+uninstall:
+ rm -f '$(DESTDIR)$(bindir)/pg_walsummary$(X)'
+
+clean distclean maintainer-clean:
+ rm -f pg_walsummary$(X) $(OBJS)
diff --git a/src/bin/pg_walsummary/meson.build b/src/bin/pg_walsummary/meson.build
new file mode 100644
index 0000000000..c2092960c6
--- /dev/null
+++ b/src/bin/pg_walsummary/meson.build
@@ -0,0 +1,24 @@
+# Copyright (c) 2022-2023, PostgreSQL Global Development Group
+
+pg_walsummary_sources = files(
+ 'pg_walsummary.c',
+)
+
+if host_system == 'windows'
+ pg_walsummary_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'pg_walsummary',
+ '--FILEDESC', 'pg_walsummary - print contents of WAL summary files',])
+endif
+
+pg_walsummary = executable('pg_walsummary',
+ pg_walsummary_sources,
+ dependencies: [frontend_code],
+ kwargs: default_bin_args,
+)
+bin_targets += pg_walsummary
+
+tests += {
+ 'name': 'pg_walsummary',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir()
+}
diff --git a/src/bin/pg_walsummary/pg_walsummary.c b/src/bin/pg_walsummary/pg_walsummary.c
new file mode 100644
index 0000000000..0c0225eeb8
--- /dev/null
+++ b/src/bin/pg_walsummary/pg_walsummary.c
@@ -0,0 +1,280 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_walsummary.c
+ * Prints the contents of WAL summary files.
+ *
+ * Copyright (c) 2017-2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_walsummary/pg_walsummary.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include <fcntl.h>
+#include <limits.h>
+
+#include "common/blkreftable.h"
+#include "common/logging.h"
+#include "fe_utils/option_utils.h"
+#include "lib/stringinfo.h"
+#include "getopt_long.h"
+
+typedef struct ws_options
+{
+ bool individual;
+ bool quiet;
+} ws_options;
+
+typedef struct ws_file_info
+{
+ int fd;
+ char *filename;
+} ws_file_info;
+
+static BlockNumber *block_buffer = NULL;
+static unsigned block_buffer_size = 512; /* Initial size. */
+
+static void dump_one_relation(ws_options *opt, RelFileLocator *rlocator,
+ ForkNumber forknum, BlockNumber limit_block,
+ BlockRefTableReader *reader);
+static void help(const char *progname);
+static int compare_block_numbers(const void *a, const void *b);
+static int walsummary_read_callback(void *callback_arg, void *data,
+ int length);
+static void walsummary_error_callback(void *callback_arg, char *fmt,...) pg_attribute_printf(2, 3);
+
+/*
+ * Main program.
+ */
+int
+main(int argc, char *argv[])
+{
+ static struct option long_options[] = {
+ {"individual", no_argument, NULL, 'i'},
+ {"quiet", no_argument, NULL, 'q'},
+ {NULL, 0, NULL, 0}
+ };
+
+ const char *progname;
+ int optindex;
+ int c;
+ ws_options opt;
+
+ memset(&opt, 0, sizeof(ws_options));
+
+ pg_logging_init(argv[0]);
+ progname = get_progname(argv[0]);
+ handle_help_version_opts(argc, argv, progname, help);
+
+ /* process command-line options */
+ while ((c = getopt_long(argc, argv, "f:iqw:",
+ long_options, &optindex)) != -1)
+ {
+ switch (c)
+ {
+ case 'i':
+ opt.individual = true;
+ break;
+ case 'q':
+ opt.quiet = true;
+ break;
+ default:
+ /* getopt_long already emitted a complaint */
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ if (optind >= argc)
+ {
+ pg_log_error("%s: no input files specified", progname);
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ while (optind < argc)
+ {
+ ws_file_info ws;
+ BlockRefTableReader *reader;
+ RelFileLocator rlocator;
+ ForkNumber forknum;
+ BlockNumber limit_block;
+
+ ws.filename = argv[optind++];
+ if ((ws.fd = open(ws.filename, O_RDONLY | PG_BINARY, 0)) < 0)
+ pg_fatal("could not open file \"%s\": %m", ws.filename);
+
+ reader = CreateBlockRefTableReader(walsummary_read_callback, &ws,
+ ws.filename,
+ walsummary_error_callback, NULL);
+ while (BlockRefTableReaderNextRelation(reader, &rlocator, &forknum,
+ &limit_block))
+ dump_one_relation(&opt, &rlocator, forknum, limit_block, reader);
+
+ DestroyBlockRefTableReader(reader);
+ close(ws.fd);
+ }
+
+ exit(0);
+}
+
+/*
+ * Dump details for one relation.
+ */
+static void
+dump_one_relation(ws_options *opt, RelFileLocator *rlocator,
+ ForkNumber forknum, BlockNumber limit_block,
+ BlockRefTableReader *reader)
+{
+ unsigned i = 0;
+ unsigned nblocks;
+ BlockNumber startblock = InvalidBlockNumber;
+ BlockNumber endblock = InvalidBlockNumber;
+
+ /* Dump limit block, if any. */
+ if (limit_block != InvalidBlockNumber)
+ printf("TS %u, DB %u, REL %u, FORK %s: limit %u\n",
+ rlocator->spcOid, rlocator->dbOid, rlocator->relNumber,
+ forkNames[forknum], limit_block);
+
+ /* If we haven't allocated a block buffer yet, do that now. */
+ if (block_buffer == NULL)
+ block_buffer = palloc_array(BlockNumber, block_buffer_size);
+
+ /* Try to fill the block buffer. */
+ nblocks = BlockRefTableReaderGetBlocks(reader,
+ block_buffer,
+ block_buffer_size);
+
+ /* If we filled the block buffer completely, we must enlarge it. */
+ while (nblocks >= block_buffer_size)
+ {
+ unsigned new_size;
+
+ /* Double the size, being careful about overflow. */
+ new_size = block_buffer_size * 2;
+ if (new_size < block_buffer_size)
+ new_size = PG_UINT32_MAX;
+ block_buffer = repalloc_array(block_buffer, BlockNumber, new_size);
+
+ /* Try to fill the newly-allocated space. */
+ nblocks +=
+ BlockRefTableReaderGetBlocks(reader,
+ block_buffer + block_buffer_size,
+ new_size - block_buffer_size);
+
+ /* Save the new size for later calls. */
+ block_buffer_size = new_size;
+ }
+
+ /* If we don't need to produce any output, skip the rest of this. */
+ if (opt->quiet)
+ return;
+
+ /*
+ * Sort the returned block numbers. If the block reference table was using
+ * the bitmap representation for a given chunk, the block numbers in that
+ * chunk will already be sorted, but when the array-of-offsets
+ * representation is used, we can receive block numbers here out of order.
+ */
+ qsort(block_buffer, nblocks, sizeof(BlockNumber), compare_block_numbers);
+
+ /* Dump block references. */
+ while (i < nblocks)
+ {
+ /*
+ * Find the next range of blocks to print, but if --individual was
+ * specified, then consider each block a separate range.
+ */
+ startblock = endblock = block_buffer[i++];
+ if (!opt->individual)
+ {
+ while (i < nblocks && block_buffer[i] == endblock + 1)
+ {
+ endblock++;
+ i++;
+ }
+ }
+
+ /* Format this range of block numbers as a string. */
+ if (startblock == endblock)
+ printf("TS %u, DB %u, REL %u, FORK %s: block %u\n",
+ rlocator->spcOid, rlocator->dbOid, rlocator->relNumber,
+ forkNames[forknum], startblock);
+ else
+ printf("TS %u, DB %u, REL %u, FORK %s: blocks %u..%u\n",
+ rlocator->spcOid, rlocator->dbOid, rlocator->relNumber,
+ forkNames[forknum], startblock, endblock);
+ }
+}
+
+/*
+ * Quicksort comparator for block numbers.
+ */
+static int
+compare_block_numbers(const void *a, const void *b)
+{
+ BlockNumber aa = *(BlockNumber *) a;
+ BlockNumber bb = *(BlockNumber *) b;
+
+ if (aa > bb)
+ return 1;
+ else if (aa == bb)
+ return 0;
+ else
+ return -1;
+}
+
+/*
+ * Error callback.
+ */
+void
+walsummary_error_callback(void *callback_arg, char *fmt,...)
+{
+ va_list ap;
+
+ va_start(ap, fmt);
+ pg_log_generic_v(PG_LOG_ERROR, PG_LOG_PRIMARY, fmt, ap);
+ va_end(ap);
+
+ exit(1);
+}
+
+/*
+ * Read callback.
+ */
+int
+walsummary_read_callback(void *callback_arg, void *data, int length)
+{
+ ws_file_info *ws = callback_arg;
+ int rc;
+
+ if ((rc = read(ws->fd, data, length)) < 0)
+ pg_fatal("could not read file \"%s\": %m", ws->filename);
+
+ return rc;
+}
+
+/*
+ * help
+ *
+ * Prints help page for the program
+ *
+ * progname: the name of the executed program, such as "pg_walsummary"
+ */
+static void
+help(const char *progname)
+{
+ printf(_("%s prints the contents of a WAL summary file.\n\n"), progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION]... FILE...\n"), progname);
+ printf(_("\nOptions:\n"));
+ printf(_(" -i, --individual list block numbers individually, not as ranges\n"));
+ printf(_(" -q, --quiet don't print anything, just parse the files\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
+
+ printf(_("\nReport bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+ printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 95ae399cae..f4c141f6fb 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -4021,3 +4021,5 @@ cb_tablespace_mapping
manifest_data
manifest_writer
rfile
+ws_options
+ws_file_info
--
2.37.1 (Apple Git-137.1)
[application/octet-stream] v8-0001-Change-how-a-base-backup-decides-which-files-have.patch (11.4K, ../../CA+TgmoYiPn7GE_GFvGf+150XOt-Pkdm8h4rzy42jYCRrceUWhg@mail.gmail.com/3-v8-0001-Change-how-a-base-backup-decides-which-files-have.patch)
download | inline diff:
From f27134fe549da57e0839cc2ab4e5168fa879ad80 Mon Sep 17 00:00:00 2001
From: Robert Haas <[email protected]>
Date: Tue, 3 Oct 2023 13:32:28 -0400
Subject: [PATCH v8 1/6] Change how a base backup decides which files have
checksums.
Previously, it thought that any plain file located under global, base,
or a tablespace directory had checksums unless it was in a short list
of excluded files. Now, it thinks that files in those directories have
checksums if parse_filename_for_nontemp_relation says that they are
relation files. (Temporary relation files don't matter because they're
excluded from the backup anyway.)
This changes the behavior if you have stray files not managed by
PostgreSQL in the relevant directories. Previously, you'd get some
kind of checksum-related complaint if such files existed, assuming
that the cluster had checksums enabled and that the base backup
wasn't run with NOVERIFY_CHECKSUMS. Now, you won't get those
complaints any more. That seems like an improvement to me, because
those files were presumably not created by PostgreSQL and so there
is no reason to think that they would be checksummed like a
PostgreSQL relation file. (If we want to complain about such files,
we should complain about them existing at all, not just about their
checksums.)
The point of this change is to make the code more consistent.
sendDir() was already calling parse_filename_for_nontemp_relation()
as part of an effort to determine which files to include in the
backup. So, it already had the information about whether a certain
file was a relation file. sendFile() then used a separate method,
embodied in is_checksummed_file(), to make what is essentially
the same determination. It's better not to make the same decision
using two different methods, especially in closely-related code.
---
src/backend/backup/basebackup.c | 172 ++++++++++----------------------
1 file changed, 55 insertions(+), 117 deletions(-)
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index b537f46219..4ba63ad8a6 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -82,7 +82,8 @@ static int64 sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeo
backup_manifest_info *manifest, Oid spcoid);
static bool sendFile(bbsink *sink, const char *readfilename, const char *tarfilename,
struct stat *statbuf, bool missing_ok,
- Oid dboid, Oid spcoid,
+ Oid dboid, Oid spcoid, RelFileNumber relfilenumber,
+ unsigned segno,
backup_manifest_info *manifest);
static off_t read_file_data_into_buffer(bbsink *sink,
const char *readfilename, int fd,
@@ -104,7 +105,6 @@ static void convert_link_to_directory(const char *pathbuf, struct stat *statbuf)
static void perform_base_backup(basebackup_options *opt, bbsink *sink);
static void parse_basebackup_options(List *options, basebackup_options *opt);
static int compareWalFileNames(const ListCell *a, const ListCell *b);
-static bool is_checksummed_file(const char *fullpath, const char *filename);
static int basebackup_read_file(int fd, char *buf, size_t nbytes, off_t offset,
const char *filename, bool partial_read_ok);
@@ -213,23 +213,6 @@ static const struct exclude_list_item excludeFiles[] =
{NULL, false}
};
-/*
- * List of files excluded from checksum validation.
- *
- * Note: this list should be kept in sync with what pg_checksums.c
- * includes.
- */
-static const struct exclude_list_item noChecksumFiles[] = {
- {"pg_control", false},
- {"pg_filenode.map", false},
- {"pg_internal.init", true},
- {"PG_VERSION", false},
-#ifdef EXEC_BACKEND
- {"config_exec_params", true},
-#endif
- {NULL, false}
-};
-
/*
* Actually do a base backup for the specified tablespaces.
*
@@ -356,7 +339,8 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
errmsg("could not stat file \"%s\": %m",
XLOG_CONTROL_FILE)));
sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf,
- false, InvalidOid, InvalidOid, &manifest);
+ false, InvalidOid, InvalidOid,
+ InvalidRelFileNumber, 0, &manifest);
}
else
{
@@ -625,7 +609,8 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
errmsg("could not stat file \"%s\": %m", pathbuf)));
sendFile(sink, pathbuf, pathbuf, &statbuf, false,
- InvalidOid, InvalidOid, &manifest);
+ InvalidOid, InvalidOid, InvalidRelFileNumber, 0,
+ &manifest);
/* unconditionally mark file as archived */
StatusFilePath(pathbuf, fname, ".done");
@@ -1163,7 +1148,8 @@ sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
struct stat statbuf;
int64 size = 0;
const char *lastDir; /* Split last dir from parent path. */
- bool isDbDir = false; /* Does this directory contain relations? */
+ bool isRelationDir = false; /* Does directory contain relations? */
+ Oid dboid = InvalidOid;
/*
* Determine if the current path is a database directory that can contain
@@ -1190,17 +1176,23 @@ sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
strncmp(lastDir - (sizeof(TABLESPACE_VERSION_DIRECTORY) - 1),
TABLESPACE_VERSION_DIRECTORY,
sizeof(TABLESPACE_VERSION_DIRECTORY) - 1) == 0))
- isDbDir = true;
+ {
+ isRelationDir = true;
+ dboid = atooid(lastDir + 1);
+ }
}
+ else if (strcmp(path, "./global") == 0)
+ isRelationDir = true;
dir = AllocateDir(path);
while ((de = ReadDir(dir, path)) != NULL)
{
int excludeIdx;
bool excludeFound;
- RelFileNumber relNumber;
- ForkNumber relForkNum;
- unsigned segno;
+ RelFileNumber relfilenumber = InvalidRelFileNumber;
+ ForkNumber relForkNum = InvalidForkNumber;
+ unsigned segno = 0;
+ bool isRelationFile = false;
/* Skip special stuff */
if (strcmp(de->d_name, ".") == 0 || strcmp(de->d_name, "..") == 0)
@@ -1248,37 +1240,40 @@ sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
if (excludeFound)
continue;
+ /*
+ * If there could be non-temporary relation files in this directory,
+ * try to parse the filename.
+ */
+ if (isRelationDir)
+ isRelationFile =
+ parse_filename_for_nontemp_relation(de->d_name,
+ &relfilenumber,
+ &relForkNum, &segno);
+
/* Exclude all forks for unlogged tables except the init fork */
- if (isDbDir &&
- parse_filename_for_nontemp_relation(de->d_name, &relNumber,
- &relForkNum, &segno))
+ if (isRelationFile && relForkNum != INIT_FORKNUM)
{
- /* Never exclude init forks */
- if (relForkNum != INIT_FORKNUM)
- {
- char initForkFile[MAXPGPATH];
+ char initForkFile[MAXPGPATH];
- /*
- * If any other type of fork, check if there is an init fork
- * with the same RelFileNumber. If so, the file can be
- * excluded.
- */
- snprintf(initForkFile, sizeof(initForkFile), "%s/%u_init",
- path, relNumber);
+ /*
+ * If any other type of fork, check if there is an init fork with
+ * the same RelFileNumber. If so, the file can be excluded.
+ */
+ snprintf(initForkFile, sizeof(initForkFile), "%s/%u_init",
+ path, relfilenumber);
- if (lstat(initForkFile, &statbuf) == 0)
- {
- elog(DEBUG2,
- "unlogged relation file \"%s\" excluded from backup",
- de->d_name);
+ if (lstat(initForkFile, &statbuf) == 0)
+ {
+ elog(DEBUG2,
+ "unlogged relation file \"%s\" excluded from backup",
+ de->d_name);
- continue;
- }
+ continue;
}
}
/* Exclude temporary relations */
- if (isDbDir && looks_like_temp_rel_name(de->d_name))
+ if (OidIsValid(dboid) && looks_like_temp_rel_name(de->d_name))
{
elog(DEBUG2,
"temporary relation file \"%s\" excluded from backup",
@@ -1417,8 +1412,8 @@ sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
if (!sizeonly)
sent = sendFile(sink, pathbuf, pathbuf + basepathlen + 1, &statbuf,
- true, isDbDir ? atooid(lastDir + 1) : InvalidOid, spcoid,
- manifest);
+ true, dboid, spcoid,
+ relfilenumber, segno, manifest);
if (sent || sizeonly)
{
@@ -1440,40 +1435,6 @@ sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
return size;
}
-/*
- * Check if a file should have its checksum validated.
- * We validate checksums on files in regular tablespaces
- * (including global and default) only, and in those there
- * are some files that are explicitly excluded.
- */
-static bool
-is_checksummed_file(const char *fullpath, const char *filename)
-{
- /* Check that the file is in a tablespace */
- if (strncmp(fullpath, "./global/", 9) == 0 ||
- strncmp(fullpath, "./base/", 7) == 0 ||
- strncmp(fullpath, "/", 1) == 0)
- {
- int excludeIdx;
-
- /* Compare file against noChecksumFiles skip list */
- for (excludeIdx = 0; noChecksumFiles[excludeIdx].name != NULL; excludeIdx++)
- {
- int cmplen = strlen(noChecksumFiles[excludeIdx].name);
-
- if (!noChecksumFiles[excludeIdx].match_prefix)
- cmplen++;
- if (strncmp(filename, noChecksumFiles[excludeIdx].name,
- cmplen) == 0)
- return false;
- }
-
- return true;
- }
- else
- return false;
-}
-
/*
* Given the member, write the TAR header & send the file.
*
@@ -1488,6 +1449,7 @@ is_checksummed_file(const char *fullpath, const char *filename)
static bool
sendFile(bbsink *sink, const char *readfilename, const char *tarfilename,
struct stat *statbuf, bool missing_ok, Oid dboid, Oid spcoid,
+ RelFileNumber relfilenumber, unsigned segno,
backup_manifest_info *manifest)
{
int fd;
@@ -1495,8 +1457,6 @@ sendFile(bbsink *sink, const char *readfilename, const char *tarfilename,
int checksum_failures = 0;
off_t cnt;
pgoff_t bytes_done = 0;
- int segmentno = 0;
- char *segmentpath;
bool verify_checksum = false;
pg_checksum_context checksum_ctx;
@@ -1522,36 +1482,14 @@ sendFile(bbsink *sink, const char *readfilename, const char *tarfilename,
*/
Assert((sink->bbs_buffer_length % BLCKSZ) == 0);
- if (!noverify_checksums && DataChecksumsEnabled())
- {
- char *filename;
-
- /*
- * Get the filename (excluding path). As last_dir_separator()
- * includes the last directory separator, we chop that off by
- * incrementing the pointer.
- */
- filename = last_dir_separator(readfilename) + 1;
-
- if (is_checksummed_file(readfilename, filename))
- {
- verify_checksum = true;
-
- /*
- * Cut off at the segment boundary (".") to get the segment number
- * in order to mix it into the checksum.
- */
- segmentpath = strstr(filename, ".");
- if (segmentpath != NULL)
- {
- segmentno = atoi(segmentpath + 1);
- if (segmentno == 0)
- ereport(ERROR,
- (errmsg("invalid segment number %d in file \"%s\"",
- segmentno, filename)));
- }
- }
- }
+ /*
+ * If we weren't told not to verify checksums, and if checksums are
+ * enabled for this cluster, and if this is a relation file, then verify
+ * the checksum.
+ */
+ if (!noverify_checksums && DataChecksumsEnabled() &&
+ RelFileNumberIsValid(relfilenumber))
+ verify_checksum = true;
/*
* Loop until we read the amount of data the caller told us to expect. The
@@ -1566,7 +1504,7 @@ sendFile(bbsink *sink, const char *readfilename, const char *tarfilename,
/* Try to read some more data. */
cnt = read_file_data_into_buffer(sink, readfilename, fd, bytes_done,
remaining,
- blkno + segmentno * RELSEG_SIZE,
+ blkno + segno * RELSEG_SIZE,
verify_checksum,
&checksum_failures);
--
2.37.1 (Apple Git-137.1)
[application/octet-stream] v8-0006-Test-patch-Enable-summarize_wal-by-default.patch (4.7K, ../../CA+TgmoYiPn7GE_GFvGf+150XOt-Pkdm8h4rzy42jYCRrceUWhg@mail.gmail.com/4-v8-0006-Test-patch-Enable-summarize_wal-by-default.patch)
download | inline diff:
From aa779568f816259e9f225a8c6f0a2e7cdb0252fa Mon Sep 17 00:00:00 2001
From: Robert Haas <[email protected]>
Date: Mon, 6 Nov 2023 13:53:19 -0500
Subject: [PATCH v8 6/6] Test patch: Enable summarize_wal by default.
To avoid test failures, must remove the prohibition against running
summarize_wal=off with wal_level=minimal, because a bunch of tests
run with wal_level=minimal.
Not for commit.
---
src/backend/postmaster/postmaster.c | 3 ---
src/backend/postmaster/walsummarizer.c | 2 +-
src/backend/utils/misc/guc_tables.c | 2 +-
src/test/recovery/t/001_stream_rep.pl | 2 ++
src/test/recovery/t/019_replslot_limit.pl | 3 +++
src/test/recovery/t/020_archive_status.pl | 1 +
src/test/recovery/t/035_standby_logical_decoding.pl | 1 +
7 files changed, 9 insertions(+), 5 deletions(-)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7952fd5c4b..a804d07ce5 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -935,9 +935,6 @@ PostmasterMain(int argc, char *argv[])
if (max_wal_senders > 0 && wal_level == WAL_LEVEL_MINIMAL)
ereport(ERROR,
(errmsg("WAL streaming (max_wal_senders > 0) requires wal_level \"replica\" or \"logical\"")));
- if (summarize_wal && wal_level == WAL_LEVEL_MINIMAL)
- ereport(ERROR,
- (errmsg("WAL cannot be summarized when wal_level is \"minimal\"")));
/*
* Other one-time internal sanity checks can go here, if they are fast.
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 505351f663..1bd570b370 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -140,7 +140,7 @@ static XLogRecPtr redo_pointer_at_last_summary_removal = InvalidXLogRecPtr;
/*
* GUC parameters
*/
-bool summarize_wal = false;
+bool summarize_wal = true;
int wal_summarize_keep_time = 10 * 24 * 60;
static XLogRecPtr GetLatestLSN(TimeLineID *tli);
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index cf39f8a651..e631994da2 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -1796,7 +1796,7 @@ struct config_bool ConfigureNamesBool[] =
NULL
},
&summarize_wal,
- false,
+ true,
NULL, NULL, NULL
},
diff --git a/src/test/recovery/t/001_stream_rep.pl b/src/test/recovery/t/001_stream_rep.pl
index 95f9b0d772..0d0e63b8dc 100644
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -15,6 +15,8 @@ my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(
allows_streaming => 1,
auth_extra => [ '--create-role', 'repl_role' ]);
+# WAL summarization can postpone WAL recycling, leading to test failures
+$node_primary->append_conf('postgresql.conf', "summarize_wal = off");
$node_primary->start;
my $backup_name = 'my_backup';
diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl
index 7d94f15778..a8b342bb98 100644
--- a/src/test/recovery/t/019_replslot_limit.pl
+++ b/src/test/recovery/t/019_replslot_limit.pl
@@ -22,6 +22,7 @@ $node_primary->append_conf(
min_wal_size = 2MB
max_wal_size = 4MB
log_checkpoints = yes
+summarize_wal = off
));
$node_primary->start;
$node_primary->safe_psql('postgres',
@@ -256,6 +257,7 @@ $node_primary2->append_conf(
min_wal_size = 32MB
max_wal_size = 32MB
log_checkpoints = yes
+summarize_wal = off
));
$node_primary2->start;
$node_primary2->safe_psql('postgres',
@@ -310,6 +312,7 @@ $node_primary3->append_conf(
max_wal_size = 2MB
log_checkpoints = yes
max_slot_wal_keep_size = 1MB
+ summarize_wal = off
));
$node_primary3->start;
$node_primary3->safe_psql('postgres',
diff --git a/src/test/recovery/t/020_archive_status.pl b/src/test/recovery/t/020_archive_status.pl
index fa24153d4b..d0d6221368 100644
--- a/src/test/recovery/t/020_archive_status.pl
+++ b/src/test/recovery/t/020_archive_status.pl
@@ -15,6 +15,7 @@ $primary->init(
has_archiving => 1,
allows_streaming => 1);
$primary->append_conf('postgresql.conf', 'autovacuum = off');
+$primary->append_conf('postgresql.conf', 'summarize_wal = off');
$primary->start;
my $primary_data = $primary->data_dir;
diff --git a/src/test/recovery/t/035_standby_logical_decoding.pl b/src/test/recovery/t/035_standby_logical_decoding.pl
index 9c34c0d36c..482edc57a8 100644
--- a/src/test/recovery/t/035_standby_logical_decoding.pl
+++ b/src/test/recovery/t/035_standby_logical_decoding.pl
@@ -250,6 +250,7 @@ $node_primary->append_conf(
wal_level = 'logical'
max_replication_slots = 4
max_wal_senders = 4
+summarize_wal = off
});
$node_primary->dump_info;
$node_primary->start;
--
2.37.1 (Apple Git-137.1)
[application/octet-stream] v8-0003-Add-a-new-WAL-summarizer-process.patch (134.4K, ../../CA+TgmoYiPn7GE_GFvGf+150XOt-Pkdm8h4rzy42jYCRrceUWhg@mail.gmail.com/5-v8-0003-Add-a-new-WAL-summarizer-process.patch)
download | inline diff:
From b1631879f17c92eca327a4ae6e9b93901f95e582 Mon Sep 17 00:00:00 2001
From: Robert Haas <[email protected]>
Date: Wed, 25 Oct 2023 12:57:22 -0400
Subject: [PATCH v8 3/6] Add a new WAL summarizer process.
When active, this process writes WAL summary files to
$PGDATA/pg_wal/summaries. Each summary file contains information for a
certain range of LSNs on a certain TLI. For each relation, it stores a
"limit block" which is 0 if a relation is created or destroyed within
a certain range of WAL records, or otherwise the shortest length to
which the relation was truncated during that range of WAL records, or
otherwise InvalidBlockNumber. In addition, it stores a list of blocks
which have been modified during that range of WAL records, but
excluding blocks which were removed by truncation after they were
modified and never subsequently modified again. In other words, it
tells us which blocks need to copied in case of an incremental backup
covering that range of WAL records.
A new parameter summarize_wal enables or disables this new background
process. The background process also automatically deletes summary
files that are older than wal_summarize_keep_time, if that parameter
has a non-zero value and the summarizer is configured to run.
---
doc/src/sgml/config.sgml | 61 +
src/backend/access/transam/xlog.c | 101 +-
src/backend/backup/Makefile | 4 +-
src/backend/backup/meson.build | 2 +
src/backend/backup/walsummary.c | 356 +++++
src/backend/backup/walsummaryfuncs.c | 169 ++
src/backend/postmaster/Makefile | 1 +
src/backend/postmaster/auxprocess.c | 8 +
src/backend/postmaster/meson.build | 1 +
src/backend/postmaster/postmaster.c | 56 +
src/backend/postmaster/walsummarizer.c | 1374 +++++++++++++++++
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/utils/activity/pgstat_io.c | 4 +-
.../utils/activity/wait_event_names.txt | 5 +
src/backend/utils/init/miscinit.c | 3 +
src/backend/utils/misc/guc_tables.c | 26 +
src/backend/utils/misc/postgresql.conf.sample | 5 +
src/bin/initdb/initdb.c | 1 +
src/common/Makefile | 1 +
src/common/blkreftable.c | 1309 ++++++++++++++++
src/common/meson.build | 1 +
src/include/access/xlog.h | 1 +
src/include/backup/walsummary.h | 49 +
src/include/catalog/pg_proc.dat | 19 +
src/include/common/blkreftable.h | 120 ++
src/include/miscadmin.h | 3 +
src/include/postmaster/walsummarizer.h | 31 +
src/include/storage/proc.h | 9 +-
src/include/utils/guc_tables.h | 1 +
src/tools/pgindent/typedefs.list | 11 +
30 files changed, 3722 insertions(+), 11 deletions(-)
create mode 100644 src/backend/backup/walsummary.c
create mode 100644 src/backend/backup/walsummaryfuncs.c
create mode 100644 src/backend/postmaster/walsummarizer.c
create mode 100644 src/common/blkreftable.c
create mode 100644 src/include/backup/walsummary.h
create mode 100644 src/include/common/blkreftable.h
create mode 100644 src/include/postmaster/walsummarizer.h
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index bd70ff2e4b..862a143f17 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4134,6 +4134,67 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</variablelist>
</sect2>
+ <sect2 id="runtime-config-wal-summarization">
+ <title>WAL Summarization</title>
+
+ <!--
+ <para>
+ These settings control WAL summarization, a feature which must be
+ enabled in order to perform an
+ <link linkend="backup-incremental-backup">incremental backup</link>.
+ </para>
+ -->
+
+ <variablelist>
+ <varlistentry id="guc-summarize-wal" xreflabel="summarize_wal">
+ <term><varname>summarize_wal</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>summarize_wal</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Enables the WAL summarizer process. Note that WAL summarization can
+ be enabled either on a primary or on a standby. WAL summarization can
+ cannot be enabled when <varname>wal_level</varname> is set to
+ <literal>minimal</literal>. This parameter can only be set in the
+ <filename>postgresql.conf</filename> file or on the server command line.
+ The default is <literal>off</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="guc-wal-summarize-keep-time" xreflabel="wal_summarize_keep_time">
+ <term><varname>wal_summarize_keep_time</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>wal_summarize_keep_time</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Configures the amount of time after which the WAL summarizer
+ automatically removes old WAL summaries. The file timestamp is used to
+ determine which files are old enough to remove. Typically, you should set
+ this comfortably higher than the time that could pass between a backup
+ and a later incremental backup that depends on it. WAL summaries must
+ be available for the entire range of WAL records between the preceding
+ backup and the new one being taken; if not, the incremental backup will
+ fail. If this parameter is set to zero, WAL summaries will not be
+ automatically deleted, but it is safe to manually remove files that you
+ know will not be required for future incremental backups.
+ This parameter can only be set in the
+ <filename>postgresql.conf</filename> file or on the server command line.
+ The default is 10 days. If <literal>summarize_wal = off</literal>,
+ existing WAL summaries will not be removed regardless of the value of
+ this parameter, because the WAL summarizer will not run.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </sect2>
+
</sect1>
<sect1 id="runtime-config-replication">
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index b541be8eec..d69f4ac5a7 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -77,6 +77,7 @@
#include "port/pg_iovec.h"
#include "postmaster/bgwriter.h"
#include "postmaster/startup.h"
+#include "postmaster/walsummarizer.h"
#include "postmaster/walwriter.h"
#include "replication/logical.h"
#include "replication/origin.h"
@@ -3555,6 +3556,43 @@ XLogGetLastRemovedSegno(void)
return lastRemovedSegNo;
}
+/*
+ * Return the oldest WAL segment on the given TLI that still exists in
+ * XLOGDIR, or 0 if none.
+ */
+XLogSegNo
+XLogGetOldestSegno(TimeLineID tli)
+{
+ DIR *xldir;
+ struct dirent *xlde;
+ XLogSegNo oldest_segno = 0;
+
+ xldir = AllocateDir(XLOGDIR);
+ while ((xlde = ReadDir(xldir, XLOGDIR)) != NULL)
+ {
+ TimeLineID file_tli;
+ XLogSegNo file_segno;
+
+ /* Ignore files that are not XLOG segments */
+ if (!IsXLogFileName(xlde->d_name))
+ continue;
+
+ /* Parse filename to get TLI and segno. */
+ XLogFromFileName(xlde->d_name, &file_tli, &file_segno,
+ wal_segment_size);
+
+ /* Ignore anything that's not from the TLI of interest. */
+ if (tli != file_tli)
+ continue;
+
+ /* If it's the oldest so far, update oldest_segno. */
+ if (oldest_segno == 0 || file_segno < oldest_segno)
+ oldest_segno = file_segno;
+ }
+
+ FreeDir(xldir);
+ return oldest_segno;
+}
/*
* Update the last removed segno pointer in shared memory, to reflect that the
@@ -3834,8 +3872,8 @@ RemoveXlogFile(const struct dirent *segment_de,
}
/*
- * Verify whether pg_wal and pg_wal/archive_status exist.
- * If the latter does not exist, recreate it.
+ * Verify whether pg_wal, pg_wal/archive_status, and pg_wal/summaries exist.
+ * If the latter do not exist, recreate them.
*
* It is not the goal of this function to verify the contents of these
* directories, but to help in cases where someone has performed a cluster
@@ -3878,6 +3916,26 @@ ValidateXLOGDirectoryStructure(void)
(errmsg("could not create missing directory \"%s\": %m",
path)));
}
+
+ /* Check for summaries */
+ snprintf(path, MAXPGPATH, XLOGDIR "/summaries");
+ if (stat(path, &stat_buf) == 0)
+ {
+ /* Check for weird cases where it exists but isn't a directory */
+ if (!S_ISDIR(stat_buf.st_mode))
+ ereport(FATAL,
+ (errmsg("required WAL directory \"%s\" does not exist",
+ path)));
+ }
+ else
+ {
+ ereport(LOG,
+ (errmsg("creating missing WAL directory \"%s\"", path)));
+ if (MakePGDirectory(path) < 0)
+ ereport(FATAL,
+ (errmsg("could not create missing directory \"%s\": %m",
+ path)));
+ }
}
/*
@@ -5202,9 +5260,9 @@ StartupXLOG(void)
#endif
/*
- * Verify that pg_wal and pg_wal/archive_status exist. In cases where
- * someone has performed a copy for PITR, these directories may have been
- * excluded and need to be re-created.
+ * Verify that pg_wal, pg_wal/archive_status, and pg_wal/summaries exist.
+ * In cases where someone has performed a copy for PITR, these directories
+ * may have been excluded and need to be re-created.
*/
ValidateXLOGDirectoryStructure();
@@ -6921,6 +6979,25 @@ CreateCheckPoint(int flags)
*/
END_CRIT_SECTION();
+ /*
+ * WAL summaries end when the next XLOG_CHECKPOINT_REDO or
+ * XLOG_CHECKPOINT_SHUTDOWN record is reached. This is the first point
+ * where (a) we're not inside of a critical section and (b) we can be
+ * certain that the relevant record has been flushed to disk, which must
+ * happen before it can be summarized.
+ *
+ * If this is a shutdown checkpoint, then this happens reasonably
+ * promptly: we've only just inserted and flushed the
+ * XLOG_CHECKPOINT_SHUTDOWN record. If this is not a shutdown checkpoint,
+ * then this might not be very prompt at all: the XLOG_CHECKPOINT_REDO
+ * record was written before we began flushing data to disk, and that
+ * could be many minutes ago at this point. However, we don't XLogFlush()
+ * after inserting that record, so we're not guaranteed that it's on disk
+ * until after the above call that flushes the XLOG_CHECKPOINT_ONLINE
+ * record.
+ */
+ SetWalSummarizerLatch();
+
/*
* Let smgr do post-checkpoint cleanup (eg, deleting old files).
*/
@@ -7595,6 +7672,20 @@ KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo)
}
}
+ /*
+ * If WAL summarization is in use, don't remove WAL that has yet to be
+ * summarized.
+ */
+ keep = GetOldestUnsummarizedLSN(NULL, NULL);
+ if (keep != InvalidXLogRecPtr)
+ {
+ XLogSegNo unsummarized_segno;
+
+ XLByteToSeg(keep, unsummarized_segno, wal_segment_size);
+ if (unsummarized_segno < segno)
+ segno = unsummarized_segno;
+ }
+
/* but, keep at least wal_keep_size if that's set */
if (wal_keep_size_mb > 0)
{
diff --git a/src/backend/backup/Makefile b/src/backend/backup/Makefile
index b21bd8ff43..a67b3c58d4 100644
--- a/src/backend/backup/Makefile
+++ b/src/backend/backup/Makefile
@@ -25,6 +25,8 @@ OBJS = \
basebackup_server.o \
basebackup_sink.o \
basebackup_target.o \
- basebackup_throttle.o
+ basebackup_throttle.o \
+ walsummary.o \
+ walsummaryfuncs.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/backup/meson.build b/src/backend/backup/meson.build
index 11a79bbf80..0e2de91e9f 100644
--- a/src/backend/backup/meson.build
+++ b/src/backend/backup/meson.build
@@ -12,4 +12,6 @@ backend_sources += files(
'basebackup_target.c',
'basebackup_throttle.c',
'basebackup_zstd.c',
+ 'walsummary.c',
+ 'walsummaryfuncs.c'
)
diff --git a/src/backend/backup/walsummary.c b/src/backend/backup/walsummary.c
new file mode 100644
index 0000000000..ca9d750483
--- /dev/null
+++ b/src/backend/backup/walsummary.c
@@ -0,0 +1,356 @@
+/*-------------------------------------------------------------------------
+ *
+ * walsummary.c
+ * Functions for accessing and managing WAL summary data.
+ *
+ * Portions Copyright (c) 2010-2022, PostgreSQL Global Development Group
+ *
+ * src/backend/backup/walsummary.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include "access/xlog_internal.h"
+#include "backup/walsummary.h"
+#include "utils/wait_event.h"
+
+static bool IsWalSummaryFilename(char *filename);
+static int ListComparatorForWalSummaryFiles(const ListCell *a,
+ const ListCell *b);
+
+/*
+ * Get a list of WAL summaries.
+ *
+ * If tli != 0, only WAL summaries with the indicated TLI will be included.
+ *
+ * If start_lsn != InvalidXLogRecPtr, only summaries that end before the
+ * indicated LSN will be included.
+ *
+ * If end_lsn != InvalidXLogRecPtr, only summaries that start before the
+ * indicated LSN will be included.
+ *
+ * The intent is that you can call GetWalSummaries(tli, start_lsn, end_lsn)
+ * to get all WAL summaries on the indicated timeline that overlap the
+ * specified LSN range.
+ */
+List *
+GetWalSummaries(TimeLineID tli, XLogRecPtr start_lsn, XLogRecPtr end_lsn)
+{
+ DIR *sdir;
+ struct dirent *dent;
+ List *result = NIL;
+
+ sdir = AllocateDir(XLOGDIR "/summaries");
+ while ((dent = ReadDir(sdir, XLOGDIR "/summaries")) != NULL)
+ {
+ WalSummaryFile *ws;
+ uint32 tmp[5];
+ TimeLineID file_tli;
+ XLogRecPtr file_start_lsn;
+ XLogRecPtr file_end_lsn;
+
+ /* Decode filename, or skip if it's not in the expected format. */
+ if (!IsWalSummaryFilename(dent->d_name))
+ continue;
+ sscanf(dent->d_name, "%08X%08X%08X%08X%08X",
+ &tmp[0], &tmp[1], &tmp[2], &tmp[3], &tmp[4]);
+ file_tli = tmp[0];
+ file_start_lsn = ((uint64) tmp[1]) << 32 | tmp[2];
+ file_end_lsn = ((uint64) tmp[3]) << 32 | tmp[4];
+
+ /* Skip if it doesn't match the filter criteria. */
+ if (tli != 0 && tli != file_tli)
+ continue;
+ if (!XLogRecPtrIsInvalid(start_lsn) && start_lsn >= file_end_lsn)
+ continue;
+ if (!XLogRecPtrIsInvalid(end_lsn) && end_lsn <= file_start_lsn)
+ continue;
+
+ /* Add it to the list. */
+ ws = palloc(sizeof(WalSummaryFile));
+ ws->tli = file_tli;
+ ws->start_lsn = file_start_lsn;
+ ws->end_lsn = file_end_lsn;
+ result = lappend(result, ws);
+ }
+ FreeDir(sdir);
+
+ return result;
+}
+
+/*
+ * Build a new list of WAL summaries based on an existing list, but filtering
+ * out summaries that don't match the search parameters.
+ *
+ * If tli != 0, only WAL summaries with the indicated TLI will be included.
+ *
+ * If start_lsn != InvalidXLogRecPtr, only summaries that end before the
+ * indicated LSN will be included.
+ *
+ * If end_lsn != InvalidXLogRecPtr, only summaries that start before the
+ * indicated LSN will be included.
+ */
+List *
+FilterWalSummaries(List *wslist, TimeLineID tli,
+ XLogRecPtr start_lsn, XLogRecPtr end_lsn)
+{
+ List *result = NIL;
+ ListCell *lc;
+
+ /* Loop over input. */
+ foreach(lc, wslist)
+ {
+ WalSummaryFile *ws = lfirst(lc);
+
+ /* Skip if it doesn't match the filter criteria. */
+ if (tli != 0 && tli != ws->tli)
+ continue;
+ if (!XLogRecPtrIsInvalid(start_lsn) && start_lsn > ws->end_lsn)
+ continue;
+ if (!XLogRecPtrIsInvalid(end_lsn) && end_lsn < ws->start_lsn)
+ continue;
+
+ /* Add it to the result list. */
+ result = lappend(result, ws);
+ }
+
+ return result;
+}
+
+/*
+ * Check whether the supplied list of WalSummaryFile objects covers the
+ * whole range of LSNs from start_lsn to end_lsn. This function ignores
+ * timelines, so the caller should probably filter using the appropriate
+ * timeline before calling this.
+ *
+ * If the whole range of LSNs is covered, returns true, otherwise false.
+ * If false is returned, *missing_lsn is set either to InvalidXLogRecPtr
+ * if there are no WAL summary files in the input list, or to the first LSN
+ * in the range that is not covered by a WAL summary file in the input list.
+ */
+bool
+WalSummariesAreComplete(List *wslist, XLogRecPtr start_lsn,
+ XLogRecPtr end_lsn, XLogRecPtr *missing_lsn)
+{
+ XLogRecPtr current_lsn = start_lsn;
+ ListCell *lc;
+
+ /* Special case for empty list. */
+ if (wslist == NIL)
+ {
+ *missing_lsn = InvalidXLogRecPtr;
+ return false;
+ }
+
+ /* Make a private copy of the list and sort it by start LSN. */
+ wslist = list_copy(wslist);
+ list_sort(wslist, ListComparatorForWalSummaryFiles);
+
+ /*
+ * Consider summary files in order of increasing start_lsn, advancing the
+ * known-summarized range from start_lsn toward end_lsn.
+ *
+ * Normally, the summary files should cover non-overlapping WAL ranges,
+ * but this algorithm is intended to be correct even in case of overlap.
+ */
+ foreach(lc, wslist)
+ {
+ WalSummaryFile *ws = lfirst(lc);
+
+ if (ws->start_lsn > current_lsn)
+ {
+ /* We found a gap. */
+ break;
+ }
+ if (ws->end_lsn > current_lsn)
+ {
+ /*
+ * Next summary extends beyond end of previous summary, so extend
+ * the end of the range known to be summarized.
+ */
+ current_lsn = ws->end_lsn;
+
+ /*
+ * If the range we know to be summarized has reached the required
+ * end LSN, we have proved completeness.
+ */
+ if (current_lsn >= end_lsn)
+ return true;
+ }
+ }
+
+ /*
+ * We either ran out of summary files without reaching the end LSN, or we
+ * hit a gap in the sequence that resulted in us bailing out of the loop
+ * above.
+ */
+ *missing_lsn = current_lsn;
+ return false;
+}
+
+/*
+ * Open a WAL summary file.
+ *
+ * This will throw an error in case of trouble. As an exception, if
+ * missing_ok = true and the trouble is specifically that the file does
+ * not exist, it will not throw an error and will return a value less than 0.
+ */
+File
+OpenWalSummaryFile(WalSummaryFile *ws, bool missing_ok)
+{
+ char path[MAXPGPATH];
+ File file;
+
+ snprintf(path, MAXPGPATH,
+ XLOGDIR "/summaries/%08X%08X%08X%08X%08X.summary",
+ ws->tli,
+ LSN_FORMAT_ARGS(ws->start_lsn),
+ LSN_FORMAT_ARGS(ws->end_lsn));
+
+ file = PathNameOpenFile(path, O_RDONLY);
+ if (file < 0 && (errno != EEXIST || !missing_ok))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m", path)));
+
+ return file;
+}
+
+/*
+ * Remove a WAL summary file if the last modification time precedes the
+ * cutoff time.
+ */
+void
+RemoveWalSummaryIfOlderThan(WalSummaryFile *ws, time_t cutoff_time)
+{
+ char path[MAXPGPATH];
+ struct stat statbuf;
+
+ snprintf(path, MAXPGPATH,
+ XLOGDIR "/summaries/%08X%08X%08X%08X%08X.summary",
+ ws->tli,
+ LSN_FORMAT_ARGS(ws->start_lsn),
+ LSN_FORMAT_ARGS(ws->end_lsn));
+
+ if (lstat(path, &statbuf) != 0)
+ {
+ if (errno == ENOENT)
+ return;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not stat file \"%s\": %m", path)));
+ }
+ if (statbuf.st_mtime >= cutoff_time)
+ return;
+ if (unlink(path) != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not stat file \"%s\": %m", path)));
+ ereport(DEBUG2,
+ (errmsg_internal("removing file \"%s\"", path)));
+}
+
+/*
+ * Test whether a filename looks like a WAL summary file.
+ */
+static bool
+IsWalSummaryFilename(char *filename)
+{
+ return strspn(filename, "0123456789ABCDEF") == 40 &&
+ strcmp(filename + 40, ".summary") == 0;
+}
+
+/*
+ * Data read callback for use with CreateBlockRefTableReader.
+ */
+int
+ReadWalSummary(void *wal_summary_io, void *data, int length)
+{
+ WalSummaryIO *io = wal_summary_io;
+ int nbytes;
+
+ nbytes = FileRead(io->file, data, length, io->filepos,
+ WAIT_EVENT_WAL_SUMMARY_READ);
+ if (nbytes < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write file \"%s\": %m",
+ FilePathName(io->file))));
+
+ io->filepos += nbytes;
+ return nbytes;
+}
+
+/*
+ * Data write callback for use with WriteBlockRefTable.
+ */
+int
+WriteWalSummary(void *wal_summary_io, void *data, int length)
+{
+ WalSummaryIO *io = wal_summary_io;
+ int nbytes;
+
+ nbytes = FileWrite(io->file, data, length, io->filepos,
+ WAIT_EVENT_WAL_SUMMARY_WRITE);
+ if (nbytes < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write file \"%s\": %m",
+ FilePathName(io->file))));
+ if (nbytes != length)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write file \"%s\": wrote only %d of %d bytes at offset %u",
+ FilePathName(io->file), nbytes,
+ length, (unsigned) io->filepos),
+ errhint("Check free disk space.")));
+
+ io->filepos += nbytes;
+ return nbytes;
+}
+
+/*
+ * Error-reporting callback for use with CreateBlockRefTableReader.
+ */
+void
+ReportWalSummaryError(void *callback_arg, char *fmt,...)
+{
+ StringInfoData buf;
+ va_list ap;
+ int needed;
+
+ initStringInfo(&buf);
+ for (;;)
+ {
+ va_start(ap, fmt);
+ needed = appendStringInfoVA(&buf, fmt, ap);
+ va_end(ap);
+ if (needed == 0)
+ break;
+ enlargeStringInfo(&buf, needed);
+ }
+ ereport(ERROR,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg_internal("%s", buf.data));
+}
+
+/*
+ * Comparator to sort a List of WalSummaryFile objects by start_lsn.
+ */
+static int
+ListComparatorForWalSummaryFiles(const ListCell *a, const ListCell *b)
+{
+ WalSummaryFile *ws1 = lfirst(a);
+ WalSummaryFile *ws2 = lfirst(b);
+
+ if (ws1->start_lsn < ws2->start_lsn)
+ return -1;
+ if (ws1->start_lsn > ws2->start_lsn)
+ return 1;
+ return 0;
+}
diff --git a/src/backend/backup/walsummaryfuncs.c b/src/backend/backup/walsummaryfuncs.c
new file mode 100644
index 0000000000..2e77d38b4a
--- /dev/null
+++ b/src/backend/backup/walsummaryfuncs.c
@@ -0,0 +1,169 @@
+/*-------------------------------------------------------------------------
+ *
+ * walsummaryfuncs.c
+ * SQL-callable functions for accessing WAL summary data.
+ *
+ * Portions Copyright (c) 2010-2022, PostgreSQL Global Development Group
+ *
+ * src/backend/backup/walsummaryfuncs.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "backup/walsummary.h"
+#include "common/blkreftable.h"
+#include "funcapi.h"
+#include "miscadmin.h"
+#include "utils/fmgrprotos.h"
+#include "utils/pg_lsn.h"
+
+#define NUM_WS_ATTS 3
+#define NUM_SUMMARY_ATTS 6
+#define MAX_BLOCKS_PER_CALL 256
+
+/*
+ * List the WAL summary files available in pg_wal/summaries.
+ */
+Datum
+pg_available_wal_summaries(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsi;
+ List *wslist;
+ ListCell *lc;
+ Datum values[NUM_WS_ATTS];
+ bool nulls[NUM_WS_ATTS];
+
+ InitMaterializedSRF(fcinfo, 0);
+ rsi = (ReturnSetInfo *) fcinfo->resultinfo;
+
+ memset(nulls, 0, sizeof(nulls));
+
+ wslist = GetWalSummaries(0, InvalidXLogRecPtr, InvalidXLogRecPtr);
+ foreach(lc, wslist)
+ {
+ WalSummaryFile *ws = (WalSummaryFile *) lfirst(lc);
+ HeapTuple tuple;
+
+ CHECK_FOR_INTERRUPTS();
+
+ values[0] = Int64GetDatum((int64) ws->tli);
+ values[1] = LSNGetDatum(ws->start_lsn);
+ values[2] = LSNGetDatum(ws->end_lsn);
+
+ tuple = heap_form_tuple(rsi->setDesc, values, nulls);
+ tuplestore_puttuple(rsi->setResult, tuple);
+ }
+
+ return (Datum) 0;
+}
+
+/*
+ * List the contents of a WAL summary file identified by TLI, start LSN,
+ * and end LSN.
+ */
+Datum
+pg_wal_summary_contents(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsi;
+ Datum values[NUM_SUMMARY_ATTS];
+ bool nulls[NUM_SUMMARY_ATTS];
+ WalSummaryFile ws;
+ WalSummaryIO io;
+ BlockRefTableReader *reader;
+ int64 raw_tli;
+ RelFileLocator rlocator;
+ ForkNumber forknum;
+ BlockNumber limit_block;
+
+ InitMaterializedSRF(fcinfo, 0);
+ rsi = (ReturnSetInfo *) fcinfo->resultinfo;
+ memset(nulls, 0, sizeof(nulls));
+
+ /*
+ * Since the timeline could at least in theory be more than 2^31, and
+ * since we don't have unsigned types at the SQL level, it is passed as a
+ * 64-bit integer. Test whether it's out of range.
+ */
+ raw_tli = PG_GETARG_INT64(0);
+ if (raw_tli < 1 || raw_tli > PG_INT32_MAX)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid timeline %lld", (long long) raw_tli));
+
+ /* Prepare to read the specified WAL summry file. */
+ ws.tli = (TimeLineID) raw_tli;
+ ws.start_lsn = PG_GETARG_LSN(1);
+ ws.end_lsn = PG_GETARG_LSN(2);
+ io.filepos = 0;
+ io.file = OpenWalSummaryFile(&ws, false);
+ reader = CreateBlockRefTableReader(ReadWalSummary, &io,
+ FilePathName(io.file),
+ ReportWalSummaryError, NULL);
+
+ /* Loop over relation forks. */
+ while (BlockRefTableReaderNextRelation(reader, &rlocator, &forknum,
+ &limit_block))
+ {
+ BlockNumber blocks[MAX_BLOCKS_PER_CALL];
+ HeapTuple tuple;
+
+ CHECK_FOR_INTERRUPTS();
+
+ values[0] = ObjectIdGetDatum(rlocator.relNumber);
+ values[1] = ObjectIdGetDatum(rlocator.spcOid);
+ values[2] = ObjectIdGetDatum(rlocator.dbOid);
+ values[3] = Int16GetDatum((int16) forknum);
+
+ /* Loop over blocks within the current relation fork. */
+ while (true)
+ {
+ unsigned nblocks;
+ unsigned i;
+
+ CHECK_FOR_INTERRUPTS();
+
+ nblocks = BlockRefTableReaderGetBlocks(reader, blocks,
+ MAX_BLOCKS_PER_CALL);
+ if (nblocks == 0)
+ break;
+
+ /*
+ * For each block that we specifically know to have been modified,
+ * emit a row with that block number and limit_block = false.
+ */
+ values[5] = BoolGetDatum(false);
+ for (i = 0; i < nblocks; ++i)
+ {
+ values[4] = Int64GetDatum((int64) blocks[i]);
+
+ tuple = heap_form_tuple(rsi->setDesc, values, nulls);
+ tuplestore_puttuple(rsi->setResult, tuple);
+ }
+
+ /*
+ * If the limit block is not InvalidBlockNumber, emit an exta row
+ * with that block number and limit_block = true.
+ *
+ * There is no point in doing this when the limit_block is
+ * InvalidBlockNumber, because no block with that number or any
+ * higher number can ever exist.
+ */
+ if (BlockNumberIsValid(limit_block))
+ {
+ values[4] = Int64GetDatum((int64) limit_block);
+ values[5] = BoolGetDatum(true);
+
+ tuple = heap_form_tuple(rsi->setDesc, values, nulls);
+ tuplestore_puttuple(rsi->setResult, tuple);
+ }
+ }
+ }
+
+ /* Cleanup */
+ DestroyBlockRefTableReader(reader);
+ FileClose(io.file);
+
+ return (Datum) 0;
+}
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile
index 047448b34e..367a46c617 100644
--- a/src/backend/postmaster/Makefile
+++ b/src/backend/postmaster/Makefile
@@ -24,6 +24,7 @@ OBJS = \
postmaster.o \
startup.o \
syslogger.o \
+ walsummarizer.o \
walwriter.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/postmaster/auxprocess.c b/src/backend/postmaster/auxprocess.c
index cae6feb356..0c15c1777d 100644
--- a/src/backend/postmaster/auxprocess.c
+++ b/src/backend/postmaster/auxprocess.c
@@ -21,6 +21,7 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/startup.h"
+#include "postmaster/walsummarizer.h"
#include "postmaster/walwriter.h"
#include "replication/walreceiver.h"
#include "storage/bufmgr.h"
@@ -80,6 +81,9 @@ AuxiliaryProcessMain(AuxProcType auxtype)
case WalReceiverProcess:
MyBackendType = B_WAL_RECEIVER;
break;
+ case WalSummarizerProcess:
+ MyBackendType = B_WAL_SUMMARIZER;
+ break;
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
MyBackendType = B_INVALID;
@@ -161,6 +165,10 @@ AuxiliaryProcessMain(AuxProcType auxtype)
WalReceiverMain();
proc_exit(1);
+ case WalSummarizerProcess:
+ WalSummarizerMain();
+ proc_exit(1);
+
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
proc_exit(1);
diff --git a/src/backend/postmaster/meson.build b/src/backend/postmaster/meson.build
index cda921fd10..a30eb6692f 100644
--- a/src/backend/postmaster/meson.build
+++ b/src/backend/postmaster/meson.build
@@ -12,5 +12,6 @@ backend_sources += files(
'postmaster.c',
'startup.c',
'syslogger.c',
+ 'walsummarizer.c',
'walwriter.c',
)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 7b6b613c4a..7952fd5c4b 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -115,6 +115,7 @@
#include "postmaster/pgarch.h"
#include "postmaster/postmaster.h"
#include "postmaster/syslogger.h"
+#include "postmaster/walsummarizer.h"
#include "replication/logicallauncher.h"
#include "replication/walsender.h"
#include "storage/fd.h"
@@ -252,6 +253,7 @@ static pid_t StartupPID = 0,
CheckpointerPID = 0,
WalWriterPID = 0,
WalReceiverPID = 0,
+ WalSummarizerPID = 0,
AutoVacPID = 0,
PgArchPID = 0,
SysLoggerPID = 0;
@@ -443,6 +445,7 @@ static bool CreateOptsFile(int argc, char *argv[], char *fullprogname);
static pid_t StartChildProcess(AuxProcType type);
static void StartAutovacuumWorker(void);
static void MaybeStartWalReceiver(void);
+static void MaybeStartWalSummarizer(void);
static void InitPostmasterDeathWatchHandle(void);
/*
@@ -562,6 +565,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
#define StartWalReceiver() StartChildProcess(WalReceiverProcess)
+#define StartWalSummarizer() StartChildProcess(WalSummarizerProcess)
/* Macros to check exit status of a child process */
#define EXIT_STATUS_0(st) ((st) == 0)
@@ -931,6 +935,9 @@ PostmasterMain(int argc, char *argv[])
if (max_wal_senders > 0 && wal_level == WAL_LEVEL_MINIMAL)
ereport(ERROR,
(errmsg("WAL streaming (max_wal_senders > 0) requires wal_level \"replica\" or \"logical\"")));
+ if (summarize_wal && wal_level == WAL_LEVEL_MINIMAL)
+ ereport(ERROR,
+ (errmsg("WAL cannot be summarized when wal_level is \"minimal\"")));
/*
* Other one-time internal sanity checks can go here, if they are fast.
@@ -1833,6 +1840,9 @@ ServerLoop(void)
if (WalReceiverRequested)
MaybeStartWalReceiver();
+ /* If we need to start a WAL summarizer, try to do that now */
+ MaybeStartWalSummarizer();
+
/* Get other worker processes running, if needed */
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
@@ -2657,6 +2667,8 @@ process_pm_reload_request(void)
signal_child(WalWriterPID, SIGHUP);
if (WalReceiverPID != 0)
signal_child(WalReceiverPID, SIGHUP);
+ if (WalSummarizerPID != 0)
+ signal_child(WalSummarizerPID, SIGHUP);
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGHUP);
if (PgArchPID != 0)
@@ -3010,6 +3022,7 @@ process_pm_child_exit(void)
BgWriterPID = StartBackgroundWriter();
if (WalWriterPID == 0)
WalWriterPID = StartWalWriter();
+ MaybeStartWalSummarizer();
/*
* Likewise, start other special children as needed. In a restart
@@ -3128,6 +3141,20 @@ process_pm_child_exit(void)
continue;
}
+ /*
+ * Was it the wal summarizer? Normal exit can be ignored; we'll start
+ * a new one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
+ */
+ if (pid == WalSummarizerPID)
+ {
+ WalSummarizerPID = 0;
+ if (!EXIT_STATUS_0(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("WAL summarizer process"));
+ continue;
+ }
+
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@@ -3523,6 +3550,12 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
else if (WalReceiverPID != 0 && take_action)
sigquit_child(WalReceiverPID);
+ /* Take care of the walsummarizer too */
+ if (pid == WalSummarizerPID)
+ WalSummarizerPID = 0;
+ else if (WalSummarizerPID != 0 && take_action)
+ sigquit_child(WalSummarizerPID);
+
/* Take care of the autovacuum launcher too */
if (pid == AutoVacPID)
AutoVacPID = 0;
@@ -3673,6 +3706,8 @@ PostmasterStateMachine(void)
signal_child(StartupPID, SIGTERM);
if (WalReceiverPID != 0)
signal_child(WalReceiverPID, SIGTERM);
+ if (WalSummarizerPID != 0)
+ signal_child(WalSummarizerPID, SIGTERM);
/* checkpointer, archiver, stats, and syslogger may continue for now */
/* Now transition to PM_WAIT_BACKENDS state to wait for them to die */
@@ -3699,6 +3734,7 @@ PostmasterStateMachine(void)
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
WalReceiverPID == 0 &&
+ WalSummarizerPID == 0 &&
BgWriterPID == 0 &&
(CheckpointerPID == 0 ||
(!FatalError && Shutdown < ImmediateShutdown)) &&
@@ -3796,6 +3832,7 @@ PostmasterStateMachine(void)
/* These other guys should be dead already */
Assert(StartupPID == 0);
Assert(WalReceiverPID == 0);
+ Assert(WalSummarizerPID == 0);
Assert(BgWriterPID == 0);
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
@@ -4017,6 +4054,8 @@ TerminateChildren(int signal)
signal_child(WalWriterPID, signal);
if (WalReceiverPID != 0)
signal_child(WalReceiverPID, signal);
+ if (WalSummarizerPID != 0)
+ signal_child(WalSummarizerPID, signal);
if (AutoVacPID != 0)
signal_child(AutoVacPID, signal);
if (PgArchPID != 0)
@@ -5364,6 +5403,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork WAL receiver process: %m")));
break;
+ case WalSummarizerProcess:
+ ereport(LOG,
+ (errmsg("could not fork WAL summarizer process: %m")));
+ break;
default:
ereport(LOG,
(errmsg("could not fork process: %m")));
@@ -5500,6 +5543,19 @@ MaybeStartWalReceiver(void)
}
}
+/*
+ * MaybeStartWalSummarizer
+ * Start the WAL summarizer process, if not running and our state allows.
+ */
+static void
+MaybeStartWalSummarizer(void)
+{
+ if (summarize_wal && WalSummarizerPID == 0 &&
+ (pmState == PM_RUN || pmState == PM_HOT_STANDBY) &&
+ Shutdown <= SmartShutdown)
+ WalSummarizerPID = StartWalSummarizer();
+}
+
/*
* Create the opts file
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
new file mode 100644
index 0000000000..505351f663
--- /dev/null
+++ b/src/backend/postmaster/walsummarizer.c
@@ -0,0 +1,1374 @@
+/*-------------------------------------------------------------------------
+ *
+ * walsummarizer.c
+ *
+ * Background process to perform WAL summarization, if it is enabled.
+ * It continuously scans the write-ahead log and periodically emits a
+ * summary file which indicates which blocks in which relation forks
+ * were modified by WAL records in the LSN range covered by the summary
+ * file. See walsummary.c and blkreftable.c for more details on the
+ * naming and contents of WAL summary files.
+ *
+ * If configured to do, this background process will also remove WAL
+ * summary files when the file timestamp is older than a configurable
+ * threshold (but only if the WAL has been removed first).
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/postmaster/walsummarizer.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/timeline.h"
+#include "access/xlog.h"
+#include "access/xlog_internal.h"
+#include "access/xlogrecovery.h"
+#include "access/xlogutils.h"
+#include "backup/walsummary.h"
+#include "catalog/storage_xlog.h"
+#include "common/blkreftable.h"
+#include "libpq/pqsignal.h"
+#include "miscadmin.h"
+#include "postmaster/bgwriter.h"
+#include "postmaster/interrupt.h"
+#include "postmaster/walsummarizer.h"
+#include "replication/walreceiver.h"
+#include "storage/fd.h"
+#include "storage/ipc.h"
+#include "storage/lwlock.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
+#include "storage/procsignal.h"
+#include "storage/shmem.h"
+#include "storage/spin.h"
+#include "utils/guc.h"
+#include "utils/memutils.h"
+#include "utils/wait_event.h"
+
+/*
+ * Data in shared memory related to WAL summarization.
+ */
+typedef struct
+{
+ /*
+ * These fields are protected by WALSummarizerLock.
+ *
+ * Until we've discovered what summary files already exist on disk and
+ * stored that information in shared memory, initialized is false and the
+ * other fields here contain no meaningful information. After that has
+ * been done, initialized is true.
+ *
+ * summarized_tli and summarized_lsn indicate the last LSN and TLI at
+ * which the next summary file will start. Normally, these are the LSN and
+ * TLI at which the last file ended; in such case, lsn_is_exact is true.
+ * If, however, the LSN is just an approximation, then lsn_is_exact is
+ * false. This can happen if, for example, there are no existing WAL
+ * summary files at startup. In that case, we have to derive the position
+ * at which to start summarizing from the WAL files that exist on disk,
+ * and so the LSN might point to the start of the next file even though
+ * that might happen to be in the middle of a WAL record.
+ *
+ * summarizer_pgprocno is the pgprocno value for the summarizer process,
+ * if one is running, or else INVALID_PGPROCNO.
+ *
+ * pending_lsn is used by the summarizer to advertise the ending LSN of a
+ * record it has recently read. It shouldn't ever be less than
+ * summarized_lsn, but might be greater, because the summarizer buffers
+ * data for a range of LSNs in memory before writing out a new file.
+ */
+ bool initialized;
+ TimeLineID summarized_tli;
+ XLogRecPtr summarized_lsn;
+ bool lsn_is_exact;
+ int summarizer_pgprocno;
+ XLogRecPtr pending_lsn;
+
+ /*
+ * This field handles its own synchronizaton.
+ */
+ ConditionVariable summary_file_cv;
+} WalSummarizerData;
+
+/*
+ * Private data for our xlogreader's page read callback.
+ */
+typedef struct
+{
+ TimeLineID tli;
+ bool historic;
+ XLogRecPtr read_upto;
+ bool end_of_wal;
+ bool waited;
+} SummarizerReadLocalXLogPrivate;
+
+/* Pointer to shared memory state. */
+static WalSummarizerData *WalSummarizerCtl;
+
+/*
+ * When we reach end of WAL and need to read more, we sleep for a number of
+ * milliseconds that is a integer multiple of MS_PER_SLEEP_QUANTUM. This is
+ * the multiplier. It should vary between 1 and MAX_SLEEP_QUANTA, depending
+ * on system activity. See summarizer_wait_for_wal() for how we adjust this.
+ */
+static long sleep_quanta = 1;
+
+/*
+ * The sleep time will always be a multiple of 200ms and will not exceed
+ * thirty seconds (150 * 200 = 30 * 1000). Note that the timeout here needs
+ * to be substntially less than the maximum amount of time for which an
+ * incremental backup will wait for this process to catch up. Otherwise, an
+ * incremental backup might time out on an idle system just because we sleep
+ * for too long.
+ */
+#define MAX_SLEEP_QUANTA 150
+#define MS_PER_SLEEP_QUANTUM 200
+
+/*
+ * This is a count of the number of pages of WAL that we've read since the
+ * last time we waited for more WAL to appear.
+ */
+static long pages_read_since_last_sleep = 0;
+
+/*
+ * Most recent RedoRecPtr value observed by MaybeRemoveOldWalSummaries.
+ */
+static XLogRecPtr redo_pointer_at_last_summary_removal = InvalidXLogRecPtr;
+
+/*
+ * GUC parameters
+ */
+bool summarize_wal = false;
+int wal_summarize_keep_time = 10 * 24 * 60;
+
+static XLogRecPtr GetLatestLSN(TimeLineID *tli);
+static void HandleWalSummarizerInterrupts(void);
+static XLogRecPtr SummarizeWAL(TimeLineID tli, XLogRecPtr start_lsn,
+ bool exact, XLogRecPtr switch_lsn,
+ XLogRecPtr maximum_lsn);
+static void SummarizeSmgrRecord(XLogReaderState *xlogreader,
+ BlockRefTable *brtab);
+static void SummarizeXactRecord(XLogReaderState *xlogreader,
+ BlockRefTable *brtab);
+static bool SummarizeXlogRecord(XLogReaderState *xlogreader);
+static int summarizer_read_local_xlog_page(XLogReaderState *state,
+ XLogRecPtr targetPagePtr,
+ int reqLen,
+ XLogRecPtr targetRecPtr,
+ char *cur_page);
+static void summarizer_wait_for_wal(void);
+static void MaybeRemoveOldWalSummaries(void);
+
+/*
+ * Amount of shared memory required for this module.
+ */
+Size
+WalSummarizerShmemSize(void)
+{
+ return sizeof(WalSummarizerData);
+}
+
+/*
+ * Create or attach to shared memory segment for this module.
+ */
+void
+WalSummarizerShmemInit(void)
+{
+ bool found;
+
+ WalSummarizerCtl = (WalSummarizerData *)
+ ShmemInitStruct("Wal Summarizer Ctl", WalSummarizerShmemSize(),
+ &found);
+
+ if (!found)
+ {
+ /*
+ * First time through, so initialize.
+ *
+ * We're just filling in dummy values here -- the real initialization
+ * will happen when GetOldestUnsummarizedLSN() is called for the first
+ * time.
+ */
+ WalSummarizerCtl->initialized = false;
+ WalSummarizerCtl->summarized_tli = 0;
+ WalSummarizerCtl->summarized_lsn = InvalidXLogRecPtr;
+ WalSummarizerCtl->lsn_is_exact = false;
+ WalSummarizerCtl->summarizer_pgprocno = INVALID_PGPROCNO;
+ WalSummarizerCtl->pending_lsn = InvalidXLogRecPtr;
+ ConditionVariableInit(&WalSummarizerCtl->summary_file_cv);
+ }
+}
+
+/*
+ * Entry point for walsummarizer process.
+ */
+void
+WalSummarizerMain(void)
+{
+ sigjmp_buf local_sigjmp_buf;
+ MemoryContext context;
+
+ /*
+ * Within this function, 'current_lsn' and 'current_tli' refer to the
+ * point from which the next WAL summary file should start. 'exact' is
+ * true if 'current_lsn' is known to be the start of a WAL recod or WAL
+ * segment, and false if it might be in the middle of a record someplace.
+ *
+ * 'switch_lsn' and 'switch_tli', if set, are the LSN at which we need to
+ * switch to a new timeline and the timeline to which we need to switch.
+ * If not set, we either haven't figured out the answers yet or we're
+ * already on the latest timeline.
+ */
+ XLogRecPtr current_lsn;
+ TimeLineID current_tli;
+ bool exact;
+ XLogRecPtr switch_lsn = InvalidXLogRecPtr;
+ TimeLineID switch_tli = 0;
+
+ ereport(DEBUG1,
+ (errmsg_internal("WAL summarizer started")));
+
+ /*
+ * Properly accept or ignore signals the postmaster might send us
+ *
+ * We have no particular use for SIGINT at the moment, but seems
+ * reasonable to treat like SIGTERM.
+ */
+ pqsignal(SIGHUP, SignalHandlerForConfigReload);
+ pqsignal(SIGINT, SignalHandlerForShutdownRequest);
+ pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
+ /* SIGQUIT handler was already set up by InitPostmasterChild */
+ pqsignal(SIGALRM, SIG_IGN);
+ pqsignal(SIGPIPE, SIG_IGN);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
+ pqsignal(SIGUSR2, SIG_IGN); /* not used */
+
+ /* Advertise ourselves. */
+ LWLockAcquire(WALSummarizerLock, LW_EXCLUSIVE);
+ WalSummarizerCtl->summarizer_pgprocno = MyProc->pgprocno;
+ LWLockRelease(WALSummarizerLock);
+
+ /* Create and switch to a memory context that we can reset on error. */
+ context = AllocSetContextCreate(TopMemoryContext,
+ "Wal Summarizer",
+ ALLOCSET_DEFAULT_SIZES);
+ MemoryContextSwitchTo(context);
+
+ /*
+ * Reset some signals that are accepted by postmaster but not here
+ */
+ pqsignal(SIGCHLD, SIG_DFL);
+
+ /*
+ * If an exception is encountered, processing resumes here.
+ */
+ if (sigsetjmp(local_sigjmp_buf, 1) != 0)
+ {
+ /* Since not using PG_TRY, must reset error stack by hand */
+ error_context_stack = NULL;
+
+ /* Prevent interrupts while cleaning up */
+ HOLD_INTERRUPTS();
+
+ /* Report the error to the server log */
+ EmitErrorReport();
+
+ /* Release resources we might have acquired. */
+ LWLockReleaseAll();
+ ConditionVariableCancelSleep();
+ pgstat_report_wait_end();
+ ReleaseAuxProcessResources(false);
+ AtEOXact_Files(false);
+ AtEOXact_HashTables(false);
+
+ /*
+ * Now return to normal top-level context and clear ErrorContext for
+ * next time.
+ */
+ MemoryContextSwitchTo(context);
+ FlushErrorState();
+
+ /* Flush any leaked data in the top-level context */
+ MemoryContextResetAndDeleteChildren(context);
+
+ /* Now we can allow interrupts again */
+ RESUME_INTERRUPTS();
+
+ /*
+ * Sleep for 10 seconds before attempting to resume operations in
+ * order to avoid excessing logging.
+ *
+ * Many of the likely error conditions are things that will repeat
+ * every time. For example, if the WAL can't be read or the summary
+ * can't be written, only administrator action will cure the problem.
+ * So a really fast retry time doesn't seem to be especially
+ * beneficial, and it will clutter the logs.
+ */
+ (void) WaitLatch(MyLatch,
+ WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ 10000,
+ WAIT_EVENT_WAL_SUMMARIZER_ERROR);
+ }
+
+ /* We can now handle ereport(ERROR) */
+ PG_exception_stack = &local_sigjmp_buf;
+
+ /*
+ * Unblock signals (they were blocked when the postmaster forked us)
+ */
+ sigprocmask(SIG_SETMASK, &UnBlockSig, NULL);
+
+ /*
+ * Fetch information about previous progress from shared memory.
+ *
+ * If we discover that WAL summarization is not enabled, just exit.
+ */
+ current_lsn = GetOldestUnsummarizedLSN(¤t_tli, &exact);
+ if (XLogRecPtrIsInvalid(current_lsn))
+ proc_exit(0);
+
+ /*
+ * Loop forever
+ */
+ for (;;)
+ {
+ XLogRecPtr latest_lsn;
+ TimeLineID latest_tli;
+ XLogRecPtr end_of_summary_lsn;
+
+ /* Flush any leaked data in the top-level context */
+ MemoryContextResetAndDeleteChildren(context);
+
+ /* Process any signals received recently. */
+ HandleWalSummarizerInterrupts();
+
+ /* If it's time to remove any old WAL summaries, do that now. */
+ MaybeRemoveOldWalSummaries();
+
+ /* Find the LSN and TLI up to which we can safely summarize. */
+ latest_lsn = GetLatestLSN(&latest_tli);
+
+ /*
+ * If we're summarizing a historic timeline and we haven't yet
+ * computed the point at which to switch to the next timeline, do that
+ * now.
+ *
+ * Note that if this is a standby, what was previously the current
+ * timeline could become historic at any time.
+ *
+ * We could try to make this more efficient by caching the results of
+ * readTimeLineHistory when latest_tli has not changed, but since we
+ * only have to do this once per timeline switch, we probably wouldn't
+ * save any significant amount of work in practice.
+ */
+ if (current_tli != latest_tli && XLogRecPtrIsInvalid(switch_lsn))
+ {
+ List *tles = readTimeLineHistory(latest_tli);
+
+ switch_lsn = tliSwitchPoint(current_tli, tles, &switch_tli);
+ ereport(DEBUG1,
+ errmsg("switch point from TLI %u to TLI %u is at %X/%X",
+ current_tli, switch_tli, LSN_FORMAT_ARGS(switch_lsn)));
+ }
+
+ /*
+ * If we've reached the switch LSN, we can't summarize anything else
+ * on this timeline. Switch to the next timeline and go around again.
+ */
+ if (!XLogRecPtrIsInvalid(switch_lsn) && current_lsn >= switch_lsn)
+ {
+ current_tli = switch_tli;
+ switch_lsn = InvalidXLogRecPtr;
+ switch_tli = 0;
+ continue;
+ }
+
+ /* Summarize WAL. */
+ end_of_summary_lsn = SummarizeWAL(current_tli,
+ current_lsn, exact,
+ switch_lsn, latest_lsn);
+ Assert(!XLogRecPtrIsInvalid(end_of_summary_lsn));
+ Assert(end_of_summary_lsn >= current_lsn);
+
+ /*
+ * Update state for next loop iteration.
+ *
+ * Next summary file should start from exactly where this one ended.
+ */
+ current_lsn = end_of_summary_lsn;
+ exact = true;
+
+ /* Update state in shared memory. */
+ LWLockAcquire(WALSummarizerLock, LW_EXCLUSIVE);
+ Assert(WalSummarizerCtl->pending_lsn <= end_of_summary_lsn);
+ WalSummarizerCtl->summarized_lsn = end_of_summary_lsn;
+ WalSummarizerCtl->summarized_tli = current_tli;
+ WalSummarizerCtl->lsn_is_exact = true;
+ WalSummarizerCtl->pending_lsn = end_of_summary_lsn;
+ LWLockRelease(WALSummarizerLock);
+
+ /* Wake up anyone waiting for more summary files to be written. */
+ ConditionVariableBroadcast(&WalSummarizerCtl->summary_file_cv);
+ }
+}
+
+/*
+ * Get the oldest LSN in this server's timeline history that has not yet been
+ * summarized.
+ *
+ * If *tli != NULL, it will be set to the TLI for the LSN that is returned.
+ *
+ * If *lsn_is_exact != NULL, it will be set to true if the returned LSN is
+ * necessarily the start of a WAL record and false if it's just the beginning
+ * of a WAL segment.
+ */
+XLogRecPtr
+GetOldestUnsummarizedLSN(TimeLineID *tli, bool *lsn_is_exact)
+{
+ TimeLineID latest_tli;
+ LWLockMode mode = LW_SHARED;
+ int n;
+ List *tles;
+ XLogRecPtr unsummarized_lsn;
+ TimeLineID unsummarized_tli = 0;
+ bool should_make_exact = false;
+ List *existing_summaries;
+ ListCell *lc;
+
+ /* If not summarizing WAL, do nothing. */
+ if (!summarize_wal)
+ return InvalidXLogRecPtr;
+
+ /*
+ * Initially, we acquire the lock in shared mode and try to fetch the
+ * required information. If the data structure hasn't been initialized, we
+ * reacquire the lock in shared mode so that we can initialize it.
+ * However, if someone else does that first before we get the lock, then
+ * we can just return the requested information after all.
+ */
+ while (true)
+ {
+ LWLockAcquire(WALSummarizerLock, mode);
+
+ if (WalSummarizerCtl->initialized)
+ {
+ unsummarized_lsn = WalSummarizerCtl->summarized_lsn;
+ if (tli != NULL)
+ *tli = WalSummarizerCtl->summarized_tli;
+ if (lsn_is_exact != NULL)
+ *lsn_is_exact = WalSummarizerCtl->lsn_is_exact;
+ LWLockRelease(WALSummarizerLock);
+ return unsummarized_lsn;
+ }
+
+ if (mode == LW_EXCLUSIVE)
+ break;
+
+ LWLockRelease(WALSummarizerLock);
+ mode = LW_EXCLUSIVE;
+ }
+
+ /*
+ * The data structure needs to be initialized, and we are the first to
+ * obtain the lock in exclusive mode, so it's our job to do that
+ * initialization.
+ *
+ * So, find the oldest timeline on which WAL still exists, and the
+ * earliest segment for which it exists.
+ */
+ (void) GetLatestLSN(&latest_tli);
+ tles = readTimeLineHistory(latest_tli);
+ for (n = list_length(tles) - 1; n >= 0; --n)
+ {
+ TimeLineHistoryEntry *tle = list_nth(tles, n);
+ XLogSegNo oldest_segno;
+
+ oldest_segno = XLogGetOldestSegno(tle->tli);
+ if (oldest_segno != 0)
+ {
+ /* Compute oldest LSN that still exists on disk. */
+ XLogSegNoOffsetToRecPtr(oldest_segno, 0, wal_segment_size,
+ unsummarized_lsn);
+
+ unsummarized_tli = tle->tli;
+ break;
+ }
+ }
+
+ /* It really should not be possible for us to find no WAL. */
+ if (unsummarized_tli == 0)
+ ereport(ERROR,
+ errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg_internal("no WAL found on timeline %d", latest_tli));
+
+ /*
+ * Don't try to summarize anything older than the end LSN of the newest
+ * summary file that exists for this timeline.
+ */
+ existing_summaries =
+ GetWalSummaries(unsummarized_tli,
+ InvalidXLogRecPtr, InvalidXLogRecPtr);
+ foreach(lc, existing_summaries)
+ {
+ WalSummaryFile *ws = lfirst(lc);
+
+ if (ws->end_lsn > unsummarized_lsn)
+ {
+ unsummarized_lsn = ws->end_lsn;
+ should_make_exact = true;
+ }
+ }
+
+ /* Update shared memory with the discovered values. */
+ WalSummarizerCtl->initialized = true;
+ WalSummarizerCtl->summarized_lsn = unsummarized_lsn;
+ WalSummarizerCtl->summarized_tli = unsummarized_tli;
+ WalSummarizerCtl->lsn_is_exact = should_make_exact;
+ WalSummarizerCtl->pending_lsn = unsummarized_lsn;
+
+ /* Also return the to the caller as required. */
+ if (tli != NULL)
+ *tli = WalSummarizerCtl->summarized_tli;
+ if (lsn_is_exact != NULL)
+ *lsn_is_exact = WalSummarizerCtl->lsn_is_exact;
+ LWLockRelease(WALSummarizerLock);
+
+ return unsummarized_lsn;
+}
+
+/*
+ * Attempt to set the WAL summarizer's latch.
+ *
+ * This might not work, because there's no guarantee that the WAL summarizer
+ * process was successfully started, and it also might have started but
+ * subsequently terminated. So, under normal circumstances, this will get the
+ * latch set, but there's no guarantee.
+ */
+void
+SetWalSummarizerLatch(void)
+{
+ int pgprocno;
+
+ if (WalSummarizerCtl == NULL)
+ return;
+
+ LWLockAcquire(WALSummarizerLock, LW_EXCLUSIVE);
+ pgprocno = WalSummarizerCtl->summarizer_pgprocno;
+ LWLockRelease(WALSummarizerLock);
+
+ if (pgprocno != INVALID_PGPROCNO)
+ SetLatch(&ProcGlobal->allProcs[pgprocno].procLatch);
+}
+
+/*
+ * Wait until WAL summarization reaches the given LSN, but not longer than
+ * the given timeout.
+ *
+ * The return value is the first still-unsummarized LSN. If it's greater than
+ * or equal to the passed LSN, then that LSN was reached. If not, we timed out.
+ */
+XLogRecPtr
+WaitForWalSummarization(XLogRecPtr lsn, long timeout)
+{
+ TimestampTz start_time = GetCurrentTimestamp();
+ TimestampTz deadline = TimestampTzPlusMilliseconds(start_time, timeout);
+ XLogRecPtr summarized_lsn;
+
+ Assert(!XLogRecPtrIsInvalid(lsn));
+ Assert(timeout > 0);
+
+ while (1)
+ {
+ TimestampTz now;
+ long remaining_timeout;
+
+ /*
+ * If the LSN summarized on disk has reached the target value, stop.
+ */
+ LWLockAcquire(WALSummarizerLock, LW_EXCLUSIVE);
+ summarized_lsn = WalSummarizerCtl->summarized_lsn;
+ LWLockRelease(WALSummarizerLock);
+ if (summarized_lsn >= lsn)
+ break;
+
+ /* Timeout reached? If yes, stop. */
+ now = GetCurrentTimestamp();
+ remaining_timeout = TimestampDifferenceMilliseconds(now, deadline);
+ if (remaining_timeout <= 0)
+ break;
+
+ /* Wait and see. */
+ ConditionVariableTimedSleep(&WalSummarizerCtl->summary_file_cv,
+ remaining_timeout,
+ WAIT_EVENT_WAL_SUMMARY_READY);
+ }
+
+ return summarized_lsn;
+}
+
+/*
+ * Get the latest LSN that is eligible to be summarized, and set *tli to the
+ * corresponding timeline.
+ */
+static XLogRecPtr
+GetLatestLSN(TimeLineID *tli)
+{
+ if (!RecoveryInProgress())
+ {
+ /* Don't summarize WAL before it's flushed. */
+ return GetFlushRecPtr(tli);
+ }
+ else
+ {
+ XLogRecPtr flush_lsn;
+ TimeLineID flush_tli;
+ XLogRecPtr replay_lsn;
+ TimeLineID replay_tli;
+
+ /*
+ * What we really want to know is how much WAL has been flushed to
+ * disk, but the only flush position available is the one provided by
+ * the walreceiver, which may not be running, because this could be
+ * crash recovery or recovery via restore_command. So use either the
+ * WAL receiver's flush position or the replay position, whichever is
+ * further ahead, on the theory that if the WAL has been replayed then
+ * it must also have been flushed to disk.
+ */
+ flush_lsn = GetWalRcvFlushRecPtr(NULL, &flush_tli);
+ replay_lsn = GetXLogReplayRecPtr(&replay_tli);
+ if (flush_lsn > replay_lsn)
+ {
+ *tli = flush_tli;
+ return flush_lsn;
+ }
+ else
+ {
+ *tli = replay_tli;
+ return replay_lsn;
+ }
+ }
+}
+
+/*
+ * Interrupt handler for main loop of WAL summarizer process.
+ */
+static void
+HandleWalSummarizerInterrupts(void)
+{
+ if (ProcSignalBarrierPending)
+ ProcessProcSignalBarrier();
+
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+
+ if (ShutdownRequestPending || !summarize_wal)
+ {
+ ereport(DEBUG1,
+ errmsg_internal("WAL summarizer shutting down"));
+ proc_exit(0);
+ }
+
+ /* Perform logging of memory contexts of this process */
+ if (LogMemoryContextPending)
+ ProcessLogMemoryContextInterrupt();
+}
+
+/*
+ * Summarize a range of WAL records on a single timeline.
+ *
+ * 'tli' is the timeline to be summarized.
+ *
+ * 'start_lsn' is the point at which we should start summarizing. If this
+ * value comes from the end LSN of the previous record as returned by the
+ * xlograder machinery, 'exact' should be true; otherwise, 'exact' should
+ * be false, and this function will search forward for the start of a valid
+ * WAL record.
+ *
+ * 'switch_lsn' is the point at which we should switch to a later timeline,
+ * if we're summarizing a historic timeline.
+ *
+ * 'maximum_lsn' identifies the point beyond which we can't count on being
+ * able to read any more WAL. It should be the switch point when reading a
+ * historic timeline, or the most-recently-measured end of WAL when reading
+ * the current timeline.
+ *
+ * The return value is the LSN at which the WAL summary actually ends. Most
+ * often, a summary file ends because we notice that a checkpoint has
+ * occurred and reach the redo pointer of that checkpoint, but sometimes
+ * we stop for other reasons, such as a timeline switch.
+ */
+static XLogRecPtr
+SummarizeWAL(TimeLineID tli, XLogRecPtr start_lsn, bool exact,
+ XLogRecPtr switch_lsn, XLogRecPtr maximum_lsn)
+{
+ SummarizerReadLocalXLogPrivate *private_data;
+ XLogReaderState *xlogreader;
+ XLogRecPtr summary_start_lsn;
+ XLogRecPtr summary_end_lsn = switch_lsn;
+ char temp_path[MAXPGPATH];
+ char final_path[MAXPGPATH];
+ WalSummaryIO io;
+ BlockRefTable *brtab = CreateEmptyBlockRefTable();
+
+ /* Initialize private data for xlogreader. */
+ private_data = (SummarizerReadLocalXLogPrivate *)
+ palloc0(sizeof(SummarizerReadLocalXLogPrivate));
+ private_data->tli = tli;
+ private_data->historic = !XLogRecPtrIsInvalid(switch_lsn);
+ private_data->read_upto = maximum_lsn;
+
+ /* Create xlogreader. */
+ xlogreader = XLogReaderAllocate(wal_segment_size, NULL,
+ XL_ROUTINE(.page_read = &summarizer_read_local_xlog_page,
+ .segment_open = &wal_segment_open,
+ .segment_close = &wal_segment_close),
+ private_data);
+ if (xlogreader == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("out of memory"),
+ errdetail("Failed while allocating a WAL reading processor.")));
+
+ /*
+ * When exact = false, we're starting from an arbitrary point in the WAL
+ * and must search forward for the start of the next record.
+ *
+ * When exact = true, start_lsn should be either the LSN where a record
+ * begins, or the LSN of a page where the page header is immediately
+ * followed by the start of a new record. XLogBeginRead should tolerate
+ * either case.
+ *
+ * We need to allow for both cases because the behavior of xlogreader
+ * varies. When a record spans two or more xlog pages, the ending LSN
+ * reported by xlogreader will be the starting LSN of the following
+ * record, but when an xlog page boundary falls between two records, the
+ * end LSN for the first will be reported as the first byte of the
+ * following page. We can't know until we read that page how large the
+ * header will be, but we'll have to skip over it to find the next record.
+ */
+ if (exact)
+ {
+ /*
+ * Even if start_lsn is the beginning of a page rather than the
+ * beginning of the first record on that page, we should still use it
+ * as the start LSN for the summary file. That's because we detect
+ * missing summary files by looking for cases where the end LSN of one
+ * file is less than the start LSN of the next file. When only a page
+ * header is skipped, nothing has been missed.
+ */
+ XLogBeginRead(xlogreader, start_lsn);
+ summary_start_lsn = start_lsn;
+ }
+ else
+ {
+ summary_start_lsn = XLogFindNextRecord(xlogreader, start_lsn);
+ if (XLogRecPtrIsInvalid(summary_start_lsn))
+ {
+ /*
+ * If we hit end-of-WAL while trying to find the next valid
+ * record, we must be on a historic timeline that has no valid
+ * records that begin after start_lsn and before end of WAL.
+ */
+ if (private_data->end_of_wal)
+ {
+ ereport(DEBUG1,
+ errmsg_internal("could not read WAL from timeline %u at %X/%X: end of WAL at %X/%X",
+ tli,
+ LSN_FORMAT_ARGS(start_lsn),
+ LSN_FORMAT_ARGS(private_data->read_upto)));
+
+ /*
+ * The timeline ends at or after start_lsn, without containing
+ * any records. Thus, we must make sure the main loop does not
+ * iterate. If start_lsn is the end of the timeline, then we
+ * won't actually emit an empty summary file, but otherwise,
+ * we must, to capture the fact that the LSN range in question
+ * contains no interesting WAL records.
+ */
+ summary_start_lsn = start_lsn;
+ summary_end_lsn = private_data->read_upto;
+ switch_lsn = xlogreader->EndRecPtr;
+ }
+ else
+ ereport(ERROR,
+ (errmsg("could not find a valid record after %X/%X",
+ LSN_FORMAT_ARGS(start_lsn))));
+ }
+
+ /* We shouldn't go backward. */
+ Assert(summary_start_lsn >= start_lsn);
+ }
+
+ /*
+ * Main loop: read xlog records one by one.
+ */
+ while (1)
+ {
+ int block_id;
+ char *errormsg;
+ XLogRecord *record;
+ bool stop_requested = false;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /* We shouldn't go backward. */
+ Assert(summary_start_lsn <= xlogreader->EndRecPtr);
+
+ /*
+ * This flag tracks whether the read of a particular record had to
+ * wait for more WAL to arrive, so reset it before reading the next
+ * record.
+ */
+ private_data->waited = false;
+
+ /* Now read the next record. */
+ record = XLogReadRecord(xlogreader, &errormsg);
+ if (record == NULL)
+ {
+ SummarizerReadLocalXLogPrivate *private_data;
+
+ private_data = (SummarizerReadLocalXLogPrivate *)
+ xlogreader->private_data;
+ if (private_data->end_of_wal)
+ {
+ /*
+ * This timeline must be historic and must end before we were
+ * able to read a complete record.
+ */
+ ereport(DEBUG1,
+ errmsg_internal("could not read WAL from timeline %d at %X/%X: end of WAL at %X/%X",
+ tli,
+ LSN_FORMAT_ARGS(xlogreader->EndRecPtr),
+ LSN_FORMAT_ARGS(private_data->read_upto)));
+ /* Summary ends at end of WAL. */
+ summary_end_lsn = private_data->read_upto;
+ break;
+ }
+ if (errormsg)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read WAL from timeline %u at %X/%X: %s",
+ tli, LSN_FORMAT_ARGS(xlogreader->EndRecPtr),
+ errormsg)));
+ else
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read WAL from timeline %u at %X/%X",
+ tli, LSN_FORMAT_ARGS(xlogreader->EndRecPtr))));
+ }
+
+ /* We shouldn't go backward. */
+ Assert(summary_start_lsn <= xlogreader->EndRecPtr);
+
+ if (!XLogRecPtrIsInvalid(switch_lsn) &&
+ xlogreader->ReadRecPtr >= switch_lsn)
+ {
+ /*
+ * Woops! We've read a record that *starts* after the switch LSN,
+ * contrary to our goal of reading only until we hit the first
+ * record that ends at or after the switch LSN. Pretend we didn't
+ * read it after all by bailing out of this loop right here,
+ * before we do anything with this record.
+ *
+ * This can happen because the last record before the switch LSN
+ * might be continued across multiple pages, and then we might
+ * come to a page with XLP_FIRST_IS_OVERWRITE_CONTRECORD set. In
+ * that case, the record that was continued across multiple pages
+ * is incomplete and will be disregarded, and the read will
+ * restart from the beginning of the page that is flagged
+ * XLP_FIRST_IS_OVERWRITE_CONTRECORD.
+ *
+ * If this case occurs, we can fairly say that the current summary
+ * file ends at the switch LSN exactly. The first record on the
+ * page marked XLP_FIRST_IS_OVERWRITE_CONTRECORD will be
+ * discovered when generating the next summary file.
+ */
+ summary_end_lsn = switch_lsn;
+ break;
+ }
+
+ /* Special handling for particular types of WAL records. */
+ switch (XLogRecGetRmid(xlogreader))
+ {
+ case RM_SMGR_ID:
+ SummarizeSmgrRecord(xlogreader, brtab);
+ break;
+ case RM_XACT_ID:
+ SummarizeXactRecord(xlogreader, brtab);
+ break;
+ case RM_XLOG_ID:
+ stop_requested = SummarizeXlogRecord(xlogreader);
+ break;
+ default:
+ break;
+ }
+
+ /*
+ * If we've been told that it's time to end this WAL summary file, do
+ * so. As an exception, if there's nothing included in this WAL
+ * summary file yet, then stoppng doesn't make any sense, and we
+ * should wait until the next stop point instead.
+ */
+ if (stop_requested && xlogreader->ReadRecPtr > summary_start_lsn)
+ {
+ summary_end_lsn = xlogreader->ReadRecPtr;
+ break;
+ }
+
+ /* Feed block references from xlog record to block reference table. */
+ for (block_id = 0; block_id <= XLogRecMaxBlockId(xlogreader);
+ block_id++)
+ {
+ RelFileLocator rlocator;
+ ForkNumber forknum;
+ BlockNumber blocknum;
+
+ if (!XLogRecGetBlockTagExtended(xlogreader, block_id, &rlocator,
+ &forknum, &blocknum, NULL))
+ continue;
+
+ /*
+ * As we do elsewhere, ignore the FSM fork, because it's not fully
+ * WAL-logged.
+ */
+ if (forknum != FSM_FORKNUM)
+ BlockRefTableMarkBlockModified(brtab, &rlocator, forknum,
+ blocknum);
+ }
+
+ /* Update our notion of where this summary file ends. */
+ summary_end_lsn = xlogreader->EndRecPtr;
+
+ /* Also update shared memory. */
+ LWLockAcquire(WALSummarizerLock, LW_EXCLUSIVE);
+ Assert(summary_end_lsn >= WalSummarizerCtl->pending_lsn);
+ Assert(summary_end_lsn >= WalSummarizerCtl->summarized_lsn);
+ WalSummarizerCtl->pending_lsn = summary_end_lsn;
+ LWLockRelease(WALSummarizerLock);
+
+ /*
+ * If we have a switch LSN and have reached it, stop before reading
+ * the next record.
+ */
+ if (!XLogRecPtrIsInvalid(switch_lsn) &&
+ xlogreader->EndRecPtr >= switch_lsn)
+ break;
+ }
+
+ /* Destroy xlogreader. */
+ pfree(xlogreader->private_data);
+ XLogReaderFree(xlogreader);
+
+ /*
+ * If a timeline switch occurs, we may fail to make any progress at all
+ * before exiting the loop above. If that happens, we don't write a WAL
+ * summary file at all.
+ */
+ if (summary_end_lsn > summary_start_lsn)
+ {
+ /* Generate temporary and final path name. */
+ snprintf(temp_path, MAXPGPATH,
+ XLOGDIR "/summaries/temp.summary");
+ snprintf(final_path, MAXPGPATH,
+ XLOGDIR "/summaries/%08X%08X%08X%08X%08X.summary",
+ tli,
+ LSN_FORMAT_ARGS(summary_start_lsn),
+ LSN_FORMAT_ARGS(summary_end_lsn));
+
+ /* Open the temporary file for writing. */
+ io.filepos = 0;
+ io.file = PathNameOpenFile(temp_path, O_WRONLY | O_CREAT | O_TRUNC);
+ if (io.file < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m", temp_path)));
+
+ /* Write the data. */
+ WriteBlockRefTable(brtab, WriteWalSummary, &io);
+
+ /* Close temporary file and shut down xlogreader. */
+ FileClose(io.file);
+
+ /* Tell the user what we did. */
+ ereport(DEBUG1,
+ errmsg("summarized WAL on TLI %d from %X/%X to %X/%X",
+ tli,
+ LSN_FORMAT_ARGS(summary_start_lsn),
+ LSN_FORMAT_ARGS(summary_end_lsn)));
+
+ /* Durably rename the new summary into place. */
+ durable_rename(temp_path, final_path, ERROR);
+ }
+
+ return summary_end_lsn;
+}
+
+/*
+ * Special handling for WAL records with RM_SMGR_ID.
+ */
+static void
+SummarizeSmgrRecord(XLogReaderState *xlogreader, BlockRefTable *brtab)
+{
+ uint8 info = XLogRecGetInfo(xlogreader) & ~XLR_INFO_MASK;
+
+ if (info == XLOG_SMGR_CREATE)
+ {
+ xl_smgr_create *xlrec;
+
+ /*
+ * If a new relation fork is created on disk, there is no point
+ * tracking anything about which blocks have been modified, because
+ * the whole thing will be new. Hence, set the limit block for this
+ * fork to 0.
+ *
+ * Ignore the FSM fork, which is not fully WAL-logged.
+ */
+ xlrec = (xl_smgr_create *) XLogRecGetData(xlogreader);
+
+ if (xlrec->forkNum != FSM_FORKNUM)
+ BlockRefTableSetLimitBlock(brtab, &xlrec->rlocator,
+ xlrec->forkNum, 0);
+ }
+ else if (info == XLOG_SMGR_TRUNCATE)
+ {
+ xl_smgr_truncate *xlrec;
+
+ xlrec = (xl_smgr_truncate *) XLogRecGetData(xlogreader);
+
+ /*
+ * If a relation fork is truncated on disk, there is in point in
+ * tracking anything about block modifications beyond the truncation
+ * point.
+ *
+ * We ignore SMGR_TRUNCATE_FSM here because the FSM isn't fully
+ * WAL-logged and thus we can't track modified blocks for it anyway.
+ */
+ if ((xlrec->flags & SMGR_TRUNCATE_HEAP) != 0)
+ BlockRefTableSetLimitBlock(brtab, &xlrec->rlocator,
+ MAIN_FORKNUM, xlrec->blkno);
+ if ((xlrec->flags & SMGR_TRUNCATE_VM) != 0)
+ BlockRefTableSetLimitBlock(brtab, &xlrec->rlocator,
+ VISIBILITYMAP_FORKNUM, xlrec->blkno);
+ }
+}
+
+/*
+ * Special handling for WAL recods with RM_XACT_ID.
+ */
+static void
+SummarizeXactRecord(XLogReaderState *xlogreader, BlockRefTable *brtab)
+{
+ uint8 info = XLogRecGetInfo(xlogreader) & ~XLR_INFO_MASK;
+ uint8 xact_info = info & XLOG_XACT_OPMASK;
+
+ if (xact_info == XLOG_XACT_COMMIT ||
+ xact_info == XLOG_XACT_COMMIT_PREPARED)
+ {
+ xl_xact_commit *xlrec = (xl_xact_commit *) XLogRecGetData(xlogreader);
+ xl_xact_parsed_commit parsed;
+ int i;
+
+ ParseCommitRecord(XLogRecGetInfo(xlogreader), xlrec, &parsed);
+ for (i = 0; i < parsed.nrels; ++i)
+ {
+ ForkNumber forknum;
+
+ for (forknum = 0; forknum <= MAX_FORKNUM; ++forknum)
+ if (forknum != FSM_FORKNUM)
+ BlockRefTableSetLimitBlock(brtab, &parsed.xlocators[i],
+ forknum, 0);
+ }
+ }
+ else if (xact_info == XLOG_XACT_ABORT ||
+ xact_info == XLOG_XACT_ABORT_PREPARED)
+ {
+ xl_xact_abort *xlrec = (xl_xact_abort *) XLogRecGetData(xlogreader);
+ xl_xact_parsed_abort parsed;
+ int i;
+
+ ParseAbortRecord(XLogRecGetInfo(xlogreader), xlrec, &parsed);
+ for (i = 0; i < parsed.nrels; ++i)
+ {
+ ForkNumber forknum;
+
+ for (forknum = 0; forknum <= MAX_FORKNUM; ++forknum)
+ if (forknum != FSM_FORKNUM)
+ BlockRefTableSetLimitBlock(brtab, &parsed.xlocators[i],
+ forknum, 0);
+ }
+ }
+}
+
+/*
+ * Special handling for WAL recods with RM_XLOG_ID.
+ */
+static bool
+SummarizeXlogRecord(XLogReaderState *xlogreader)
+{
+ uint8 info = XLogRecGetInfo(xlogreader) & ~XLR_INFO_MASK;
+
+ if (info == XLOG_CHECKPOINT_REDO || info == XLOG_CHECKPOINT_SHUTDOWN)
+ {
+ /*
+ * This is an LSN at which redo might begin, so we'd like
+ * summarization to stop just before this WAL record.
+ */
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * Similar to read_local_xlog_page, but limited to read from one particular
+ * timeline. If the end of WAL is reached, it will wait for more if reading
+ * from the current timeline, or give up if reading from a historic timeline.
+ * In the latter case, it will also set private_data->end_of_wal = true.
+ *
+ * Caller must set private_data->tli to the TLI of interest,
+ * private_data->read_upto to the lowest LSN that is not known to be safe
+ * to read on that timeline, and private_data->historic to true if and only
+ * if the timeline is not the current timeline. This function will update
+ * private_data->read_upto and private_data->historic if more WAL appears
+ * on the current timeline or if the current timeline becomes historic.
+ */
+static int
+summarizer_read_local_xlog_page(XLogReaderState *state,
+ XLogRecPtr targetPagePtr, int reqLen,
+ XLogRecPtr targetRecPtr, char *cur_page)
+{
+ int count;
+ WALReadError errinfo;
+ SummarizerReadLocalXLogPrivate *private_data;
+
+ CHECK_FOR_INTERRUPTS();
+
+ private_data = (SummarizerReadLocalXLogPrivate *)
+ state->private_data;
+
+ while (true)
+ {
+ if (targetPagePtr + XLOG_BLCKSZ <= private_data->read_upto)
+ {
+ /*
+ * more than one block available; read only that block, have
+ * caller come back if they need more.
+ */
+ count = XLOG_BLCKSZ;
+ break;
+ }
+ else if (targetPagePtr + reqLen > private_data->read_upto)
+ {
+ /* We don't seem to have enough data. */
+ if (private_data->historic)
+ {
+ /*
+ * This is a historic timeline, so there will never be any
+ * more data than we have currently.
+ */
+ private_data->end_of_wal = true;
+ return -1;
+ }
+ else
+ {
+ XLogRecPtr latest_lsn;
+ TimeLineID latest_tli;
+
+ /*
+ * This is - or at least was up until very recently - the
+ * current timeline, so more data might show up. Delay here
+ * so we don't tight-loop.
+ */
+ HandleWalSummarizerInterrupts();
+ summarizer_wait_for_wal();
+ private_data->waited = true;
+
+ /* Recheck end-of-WAL. */
+ latest_lsn = GetLatestLSN(&latest_tli);
+ if (private_data->tli == latest_tli)
+ {
+ /* Still the current timeline, update max LSN. */
+ Assert(latest_lsn >= private_data->read_upto);
+ private_data->read_upto = latest_lsn;
+ }
+ else
+ {
+ List *tles = readTimeLineHistory(latest_tli);
+ XLogRecPtr switchpoint;
+
+ /*
+ * The timeline we're scanning is no longer the latest
+ * one. Figure out when it ended and allow reads up to
+ * exactly that point.
+ */
+ private_data->historic = true;
+ switchpoint = tliSwitchPoint(private_data->tli, tles,
+ NULL);
+ Assert(switchpoint >= private_data->read_upto);
+ private_data->read_upto = switchpoint;
+
+ /* Debugging output. */
+ ereport(DEBUG1,
+ errmsg("timeline %u became historic, can read up to %X/%X",
+ private_data->tli, LSN_FORMAT_ARGS(private_data->read_upto)));
+ }
+
+ /* Go around and try again. */
+ }
+ }
+ else
+ {
+ /* enough bytes available to satisfy the request */
+ count = private_data->read_upto - targetPagePtr;
+ break;
+ }
+ }
+
+ /*
+ * Even though we just determined how much of the page can be validly read
+ * as 'count', read the whole page anyway. It's guaranteed to be
+ * zero-padded up to the page boundary if it's incomplete.
+ */
+ if (!WALRead(state, cur_page, targetPagePtr, XLOG_BLCKSZ,
+ private_data->tli, &errinfo))
+ WALReadRaiseError(&errinfo);
+
+ /* Track that we read a page, for sleep time calculation. */
+ ++pages_read_since_last_sleep;
+
+ /* number of valid bytes in the buffer */
+ return count;
+}
+
+/*
+ * Sleep for long enough that we believe it's likely that more WAL will
+ * be available afterwards.
+ */
+static void
+summarizer_wait_for_wal(void)
+{
+ if (pages_read_since_last_sleep == 0)
+ {
+ /*
+ * No pages were read since the last sleep, so double the sleep time,
+ * but not beyond the maximum allowable value.
+ */
+ sleep_quanta = Min(sleep_quanta * 2, MAX_SLEEP_QUANTA);
+ }
+ else if (pages_read_since_last_sleep > 1)
+ {
+ /*
+ * Multiple pages were read since the last sleep, so reduce the sleep
+ * time.
+ *
+ * A large burst of activity should be able to quickly reduce the
+ * sleep time to the minimum, but we don't want a handful of extra WAL
+ * records to provoke a strong reaction. We choose to reduce the sleep
+ * time by 1 quantum for each page read beyond the first, which is a
+ * fairly arbitrary way of trying to be reactive without
+ * overrreacting.
+ */
+ if (pages_read_since_last_sleep > sleep_quanta - 1)
+ sleep_quanta = 1;
+ else
+ sleep_quanta -= pages_read_since_last_sleep;
+ }
+
+ /* OK, now sleep. */
+ (void) WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ sleep_quanta * MS_PER_SLEEP_QUANTUM,
+ WAIT_EVENT_WAL_SUMMARIZER_WAL);
+ ResetLatch(MyLatch);
+
+ /* Reset count of pages read. */
+ pages_read_since_last_sleep = 0;
+}
+
+/*
+ * Most recent RedoRecPtr value observed by RemoveOldWalSummaries.
+ */
+static void
+MaybeRemoveOldWalSummaries(void)
+{
+ XLogRecPtr redo_pointer = GetRedoRecPtr();
+ List *wslist;
+ time_t cutoff_time;
+
+ /* If WAL summary removal is disabled, don't do anything. */
+ if (wal_summarize_keep_time == 0)
+ return;
+
+ /*
+ * If the redo pointer has not advanced, don't do anything.
+ *
+ * This has the effect that we only try to remove old WAL summary files
+ * once per checkpoint cycle.
+ */
+ if (redo_pointer == redo_pointer_at_last_summary_removal)
+ return;
+ redo_pointer_at_last_summary_removal = redo_pointer;
+
+ /*
+ * Files should only be removed if the last modification time precedes the
+ * cutoff time we compute here.
+ */
+ cutoff_time = time(NULL) - 60 * wal_summarize_keep_time;
+
+ /* Get all the summaries that currently exist. */
+ wslist = GetWalSummaries(0, InvalidXLogRecPtr, InvalidXLogRecPtr);
+
+ /* Loop until all summaries have been considered for removal. */
+ while (wslist != NIL)
+ {
+ ListCell *lc;
+ XLogSegNo oldest_segno;
+ XLogRecPtr oldest_lsn = InvalidXLogRecPtr;
+ TimeLineID selected_tli;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /*
+ * Pick a timeline for which some summary files still exist on disk,
+ * and find the oldest LSN that still exists on disk for that
+ * timeline.
+ */
+ selected_tli = ((WalSummaryFile *) linitial(wslist))->tli;
+ oldest_segno = XLogGetOldestSegno(selected_tli);
+ if (oldest_segno != 0)
+ XLogSegNoOffsetToRecPtr(oldest_segno, 0, wal_segment_size,
+ oldest_lsn);
+
+
+ /* Consider each WAL file on the selected timeline in turn. */
+ foreach(lc, wslist)
+ {
+ WalSummaryFile *ws = lfirst(lc);
+
+ CHECK_FOR_INTERRUPTS();
+
+ /* If it's not on this timeline, it's not time to consider it. */
+ if (selected_tli != ws->tli)
+ continue;
+
+ /*
+ * If the WAL doesn't exist any more, we can remove it if the file
+ * modification time is old enough.
+ */
+ if (XLogRecPtrIsInvalid(oldest_lsn) || ws->end_lsn <= oldest_lsn)
+ RemoveWalSummaryIfOlderThan(ws, cutoff_time);
+
+ /*
+ * Whether we we removed the file or not, we need not consider it
+ * again.
+ */
+ wslist = foreach_delete_current(wslist, lc);
+ pfree(ws);
+ }
+ }
+}
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index f72f2906ce..d621f5507f 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -54,3 +54,4 @@ XactTruncationLock 44
WrapLimitsVacuumLock 46
NotifyQueueTailLock 47
WaitEventExtensionLock 48
+WALSummarizerLock 49
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 490d5a9ab7..8109aee6f0 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -296,7 +296,8 @@ pgstat_io_snapshot_cb(void)
* - Syslogger because it is not connected to shared memory
* - Archiver because most relevant archiving IO is delegated to a
* specialized command or module
-* - WAL Receiver and WAL Writer IO is not tracked in pg_stat_io for now
+* - WAL Receiver, WAL Writer, and WAL Summarizer IO are not tracked in
+* pg_stat_io for now
*
* Function returns true if BackendType participates in the cumulative stats
* subsystem for IO and false if it does not.
@@ -318,6 +319,7 @@ pgstat_tracks_io_bktype(BackendType bktype)
case B_LOGGER:
case B_WAL_RECEIVER:
case B_WAL_WRITER:
+ case B_WAL_SUMMARIZER:
return false;
case B_AUTOVAC_LAUNCHER:
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index d7995931bd..7e79163466 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -56,6 +56,7 @@ RECOVERY_WAL_STREAM "Waiting in main loop of startup process for WAL to arrive,
SYSLOGGER_MAIN "Waiting in main loop of syslogger process."
WAL_RECEIVER_MAIN "Waiting in main loop of WAL receiver process."
WAL_SENDER_MAIN "Waiting in main loop of WAL sender process."
+WAL_SUMMARIZER_WAL "Waiting in WAL summarizer for more WAL to be generated."
WAL_WRITER_MAIN "Waiting in main loop of WAL writer process."
@@ -142,6 +143,7 @@ SAFE_SNAPSHOT "Waiting to obtain a valid snapshot for a <literal>READ ONLY DEFER
SYNC_REP "Waiting for confirmation from a remote server during synchronous replication."
WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
+WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at end of a parallel operation."
@@ -162,6 +164,7 @@ REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the che
SPIN_DELAY "Waiting while acquiring a contended spinlock."
VACUUM_DELAY "Waiting in a cost-based vacuum delay point."
VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed."
+WAL_SUMMARIZER_ERROR "Waiting after a WAL summarizer error."
#
@@ -243,6 +246,8 @@ WAL_COPY_WRITE "Waiting for a write when creating a new WAL segment by copying a
WAL_INIT_SYNC "Waiting for a newly initialized WAL file to reach durable storage."
WAL_INIT_WRITE "Waiting for a write while initializing a new WAL file."
WAL_READ "Waiting for a read from a WAL file."
+WAL_SUMMARY_READ "Waiting for a read from a WAL summary file."
+WAL_SUMMARY_WRITE "Waiting for a write to a WAL summary file."
WAL_SYNC "Waiting for a WAL file to reach durable storage."
WAL_SYNC_METHOD_ASSIGN "Waiting for data to reach durable storage while assigning a new WAL sync method."
WAL_WRITE "Waiting for a write to a WAL file."
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index cfc5afaa6f..ef2a3a2bfd 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -306,6 +306,9 @@ GetBackendTypeDesc(BackendType backendType)
case B_WAL_SENDER:
backendDesc = "walsender";
break;
+ case B_WAL_SUMMARIZER:
+ backendDesc = "walsummarizer";
+ break;
case B_WAL_WRITER:
backendDesc = "walwriter";
break;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 7605eff9b9..cf39f8a651 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -63,6 +63,7 @@
#include "postmaster/postmaster.h"
#include "postmaster/startup.h"
#include "postmaster/syslogger.h"
+#include "postmaster/walsummarizer.h"
#include "postmaster/walwriter.h"
#include "replication/logicallauncher.h"
#include "replication/slot.h"
@@ -704,6 +705,8 @@ const char *const config_group_names[] =
gettext_noop("Write-Ahead Log / Archive Recovery"),
/* WAL_RECOVERY_TARGET */
gettext_noop("Write-Ahead Log / Recovery Target"),
+ /* WAL_SUMMARIZATION */
+ gettext_noop("Write-Ahead Log / Summarization"),
/* REPLICATION_SENDING */
gettext_noop("Replication / Sending Servers"),
/* REPLICATION_PRIMARY */
@@ -1787,6 +1790,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"summarize_wal", PGC_SIGHUP, WAL_SUMMARIZATION,
+ gettext_noop("Starts the WAL summarizer process to enable incremental backup."),
+ NULL
+ },
+ &summarize_wal,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"hot_standby", PGC_POSTMASTER, REPLICATION_STANDBY,
gettext_noop("Allows connections and queries during recovery."),
@@ -3191,6 +3204,19 @@ struct config_int ConfigureNamesInt[] =
check_wal_segment_size, NULL, NULL
},
+ {
+ {"wal_summarize_keep_time", PGC_SIGHUP, WAL_SUMMARIZATION,
+ gettext_noop("Time for which WAL summary files should be kept."),
+ NULL,
+ GUC_UNIT_MIN,
+ },
+ &wal_summarize_keep_time,
+ 10 * 24 * 60, /* 10 days */
+ 0,
+ INT_MAX,
+ NULL, NULL, NULL
+ },
+
{
{"autovacuum_naptime", PGC_SIGHUP, AUTOVACUUM,
gettext_noop("Time to sleep between autovacuum runs."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e48c066a5b..01c0428990 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -299,6 +299,11 @@
#recovery_target_action = 'pause' # 'pause', 'promote', 'shutdown'
# (change requires restart)
+# - WAL Summarization -
+
+#summarize_wal = off # run WAL summarizer process?
+#wal_summarize_keep_time = '10d' # when to remove old summary files, 0 = never
+
#------------------------------------------------------------------------------
# REPLICATION
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 0c6f5ceb0a..e68b40d2b5 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -227,6 +227,7 @@ static char *extra_options = "";
static const char *const subdirs[] = {
"global",
"pg_wal/archive_status",
+ "pg_wal/summaries",
"pg_commit_ts",
"pg_dynshmem",
"pg_notify",
diff --git a/src/common/Makefile b/src/common/Makefile
index 1092dc63df..23e5a3db47 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -49,6 +49,7 @@ OBJS_COMMON = \
archive.o \
base64.o \
binaryheap.o \
+ blkreftable.o \
checksum_helper.o \
compression.o \
config_info.o \
diff --git a/src/common/blkreftable.c b/src/common/blkreftable.c
new file mode 100644
index 0000000000..4d32da0507
--- /dev/null
+++ b/src/common/blkreftable.c
@@ -0,0 +1,1309 @@
+/*-------------------------------------------------------------------------
+ *
+ * blkreftable.c
+ * Block reference tables.
+ *
+ * A block reference table is used to keep track of which blocks have
+ * been modified by WAL records within a certain LSN range.
+ *
+ * For each relation fork, we keep track of all blocks that have appeared
+ * in block reference in the WAL. We also keep track of the "limit block",
+ * which is the smallest relation length in blocks known to have occurred
+ * during that range of WAL records. This should be set to 0 if the relation
+ * fork is created or destroyed, and to the post-truncation length if
+ * truncated.
+ *
+ * Whenever we set the limit block, we also forget about any modified blocks
+ * beyond that point. Those blocks don't exist any more. Such blocks can
+ * later be marked as modified again; if that happens, it means the relation
+ * was re-extended.
+ *
+ * Portions Copyright (c) 2010-2022, PostgreSQL Global Development Group
+ *
+ * src/common/blkreftable.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+
+#ifndef FRONTEND
+#include "postgres.h"
+#else
+#include "postgres_fe.h"
+#endif
+
+#ifdef FRONTEND
+#include "common/logging.h"
+#endif
+
+#include "common/blkreftable.h"
+#include "common/hashfn.h"
+#include "port/pg_crc32c.h"
+
+/*
+ * A block reference table keeps track of the status of each relation
+ * fork individually.
+ */
+typedef struct BlockRefTableKey
+{
+ RelFileLocator rlocator;
+ ForkNumber forknum;
+} BlockRefTableKey;
+
+/*
+ * We could need to store data either for a relation in which only a
+ * tiny fraction of the blocks have been modified or for a relation in
+ * which nearly every block has been modified, and we want a
+ * space-efficient representation in both cases. To accomplish this,
+ * we divide the relation into chunks of 2^16 blocks and choose between
+ * an array representation and a bitmap representation for each chunk.
+ *
+ * When the number of modified blocks in a given chunk is small, we
+ * essentially store an array of block numbers, but we need not store the
+ * entire block number: instead, we store each block number as a 2-byte
+ * offset from the start of the chunk.
+ *
+ * When the number of modified blocks in a given chunk is large, we switch
+ * to a bitmap representation.
+ *
+ * These same basic representational choices are used both when a block
+ * reference table is stored in memory and when it is serialized to disk.
+ *
+ * In the in-memory representation, we initially allocate each chunk with
+ * space for a number of entries given by INITIAL_ENTRIES_PER_CHUNK and
+ * increase that as necessary until we reach MAX_ENTRIES_PER_CHUNK.
+ * Any chunk whose allocated size reaches MAX_ENTRIES_PER_CHUNK is converted
+ * to a bitmap, and thus never needs to grow further.
+ */
+#define BLOCKS_PER_CHUNK (1 << 16)
+#define BLOCKS_PER_ENTRY (BITS_PER_BYTE * sizeof(uint16))
+#define MAX_ENTRIES_PER_CHUNK (BLOCKS_PER_CHUNK / BLOCKS_PER_ENTRY)
+#define INITIAL_ENTRIES_PER_CHUNK 16
+typedef uint16 *BlockRefTableChunk;
+
+/*
+ * State for one relation fork.
+ *
+ * 'rlocator' and 'forknum' identify the relation fork to which this entry
+ * pertains.
+ *
+ * 'limit_block' is the shortest known length of the relation in blocks
+ * within the LSN range covered by a particular block reference table.
+ * It should be set to 0 if the relation fork is created or dropped. If the
+ * relation fork is truncated, it should be set to the number of blocks that
+ * remain after truncation.
+ *
+ * 'nchunks' is the allocated length of each of the three arrays that follow.
+ * We can only represent the status of block numbers less than nchunks *
+ * BLOCKS_PER_CHUNK.
+ *
+ * 'chunk_size' is an array storing the allocated size of each chunk.
+ *
+ * 'chunk_usage' is an array storing the number of elements used in each
+ * chunk. If that value is less than MAX_ENTRIES_PER_CHUNK, the corresonding
+ * chunk is used as an array; else the corresponding chunk is used as a bitmap.
+ * When used as a bitmap, the least significant bit of the first array element
+ * is the status of the lowest-numbered block covered by this chunk.
+ *
+ * 'chunk_data' is the array of chunks.
+ */
+struct BlockRefTableEntry
+{
+ BlockRefTableKey key;
+ BlockNumber limit_block;
+ char status;
+ uint32 nchunks;
+ uint16 *chunk_size;
+ uint16 *chunk_usage;
+ BlockRefTableChunk *chunk_data;
+};
+
+/* Declare and define a hash table over type BlockRefTableEntry. */
+#define SH_PREFIX blockreftable
+#define SH_ELEMENT_TYPE BlockRefTableEntry
+#define SH_KEY_TYPE BlockRefTableKey
+#define SH_KEY key
+#define SH_HASH_KEY(tb, key) \
+ hash_bytes((const unsigned char *) &key, sizeof(BlockRefTableKey))
+#define SH_EQUAL(tb, a, b) memcmp(&a, &b, sizeof(BlockRefTableKey)) == 0
+#define SH_SCOPE static inline
+#ifdef FRONTEND
+#define SH_RAW_ALLOCATOR pg_malloc0
+#endif
+#define SH_DEFINE
+#define SH_DECLARE
+#include "lib/simplehash.h"
+
+/*
+ * A block reference table is basically just the hash table, but we don't
+ * want to expose that to outside callers.
+ *
+ * We keep track of the memory context in use explicitly too, so that it's
+ * easy to place all of our allocations in the same context.
+ */
+struct BlockRefTable
+{
+ blockreftable_hash *hash;
+#ifndef FRONTEND
+ MemoryContext mcxt;
+#endif
+};
+
+/*
+ * On-disk serialization format for block reference table entries.
+ */
+typedef struct BlockRefTableSerializedEntry
+{
+ RelFileLocator rlocator;
+ ForkNumber forknum;
+ BlockNumber limit_block;
+ uint32 nchunks;
+} BlockRefTableSerializedEntry;
+
+/*
+ * Buffer size, so that we avoid doing many small I/Os.
+ */
+#define BUFSIZE 65536
+
+/*
+ * Ad-hoc buffer for file I/O.
+ */
+typedef struct BlockRefTableBuffer
+{
+ io_callback_fn io_callback;
+ void *io_callback_arg;
+ char data[BUFSIZE];
+ int used;
+ int cursor;
+ pg_crc32c crc;
+} BlockRefTableBuffer;
+
+/*
+ * State for keeping track of progress while incrementally reading a block
+ * table reference file from disk.
+ *
+ * total_chunks means the number of chunks for the RelFileLocator/ForkNumber
+ * combination that is curently being read, and consumed_chunks is the number
+ * of those that have been read. (We always read all the information for
+ * a single chunk at one time, so we don't need to be able to represent the
+ * state where a chunk has been partially read.)
+ *
+ * chunk_size is the array of chunk sizes. The length is given by total_chunks.
+ *
+ * chunk_data holds the current chunk.
+ *
+ * chunk_position helps us figure out how much progress we've made in returning
+ * the block numbers for the current chunk to the caller. If the chunk is a
+ * bitmap, it's the number of bits we've scanned; otherwise, it's the number
+ * of chunk entries we've scanned.
+ */
+struct BlockRefTableReader
+{
+ BlockRefTableBuffer buffer;
+ char *error_filename;
+ report_error_fn error_callback;
+ void *error_callback_arg;
+ uint32 total_chunks;
+ uint32 consumed_chunks;
+ uint16 *chunk_size;
+ uint16 chunk_data[MAX_ENTRIES_PER_CHUNK];
+ uint32 chunk_position;
+};
+
+/*
+ * State for keeping track of progress while incrementally writing a block
+ * reference table file to disk.
+ */
+struct BlockRefTableWriter
+{
+ BlockRefTableBuffer buffer;
+};
+
+/* Function prototypes. */
+static int BlockRefTableComparator(const void *a, const void *b);
+static void BlockRefTableFlush(BlockRefTableBuffer *buffer);
+static void BlockRefTableRead(BlockRefTableReader *reader, void *data,
+ int length);
+static void BlockRefTableWrite(BlockRefTableBuffer *buffer, void *data,
+ int length);
+static void BlockRefTableFileTerminate(BlockRefTableBuffer *buffer);
+
+/*
+ * Create an empty block reference table.
+ */
+BlockRefTable *
+CreateEmptyBlockRefTable(void)
+{
+ BlockRefTable *brtab = palloc(sizeof(BlockRefTable));
+
+ /*
+ * Even completely empty database has a few hundred relation forks, so it
+ * seems best to size the hash on the assumption that we're going to have
+ * at least a few thousand entries.
+ */
+#ifdef FRONTEND
+ brtab->hash = blockreftable_create(4096, NULL);
+#else
+ brtab->mcxt = CurrentMemoryContext;
+ brtab->hash = blockreftable_create(brtab->mcxt, 4096, NULL);
+#endif
+
+ return brtab;
+}
+
+/*
+ * Set the "limit block" for a relation fork and forget any modified blocks
+ * with equal or higher block numbers.
+ *
+ * The "limit block" is the shortest known length of the relation within the
+ * range of WAL records covered by this block reference table.
+ */
+void
+BlockRefTableSetLimitBlock(BlockRefTable *brtab,
+ const RelFileLocator *rlocator,
+ ForkNumber forknum,
+ BlockNumber limit_block)
+{
+ BlockRefTableEntry *brtentry;
+ BlockRefTableKey key;
+ bool found;
+
+ memcpy(&key.rlocator, rlocator, sizeof(RelFileLocator));
+ key.forknum = forknum;
+ brtentry = blockreftable_insert(brtab->hash, key, &found);
+
+ if (!found)
+ {
+ /*
+ * We have no existing data about this relation fork, so just record
+ * the limit_block value supplied by the caller, and make sure other
+ * parts of the entry are properly initialized.
+ */
+ brtentry->limit_block = limit_block;
+ brtentry->nchunks = 0;
+ brtentry->chunk_size = NULL;
+ brtentry->chunk_usage = NULL;
+ brtentry->chunk_data = NULL;
+ return;
+ }
+
+ BlockRefTableEntrySetLimitBlock(brtentry, limit_block);
+}
+
+/*
+ * Mark a block in a given relation fork as known to have been modified.
+ */
+void
+BlockRefTableMarkBlockModified(BlockRefTable *brtab,
+ const RelFileLocator *rlocator,
+ ForkNumber forknum,
+ BlockNumber blknum)
+{
+ BlockRefTableEntry *brtentry;
+ BlockRefTableKey key;
+ bool found;
+#ifndef FRONTEND
+ MemoryContext oldcontext = MemoryContextSwitchTo(brtab->mcxt);
+#endif
+
+ memcpy(&key.rlocator, rlocator, sizeof(RelFileLocator));
+ key.forknum = forknum;
+ brtentry = blockreftable_insert(brtab->hash, key, &found);
+
+ if (!found)
+ {
+ /*
+ * We want to set the initial limit block value to something higher
+ * than any legal block number. InvalidBlockNumber fits the bill.
+ */
+ brtentry->limit_block = InvalidBlockNumber;
+ brtentry->nchunks = 0;
+ brtentry->chunk_size = NULL;
+ brtentry->chunk_usage = NULL;
+ brtentry->chunk_data = NULL;
+ }
+
+ BlockRefTableEntryMarkBlockModified(brtentry, forknum, blknum);
+
+#ifndef FRONTEND
+ MemoryContextSwitchTo(oldcontext);
+#endif
+}
+
+/*
+ * Get an entry from a block reference table.
+ *
+ * If the entry does not exist, this function returns NULL. Otherwise, it
+ * returns the entry and sets *limit_block to the value from the entry.
+ */
+BlockRefTableEntry *
+BlockRefTableGetEntry(BlockRefTable *brtab, const RelFileLocator *rlocator,
+ ForkNumber forknum, BlockNumber *limit_block)
+{
+ BlockRefTableKey key;
+ BlockRefTableEntry *entry;
+
+ Assert(limit_block != NULL);
+
+ memcpy(&key.rlocator, rlocator, sizeof(RelFileLocator));
+ key.forknum = forknum;
+ entry = blockreftable_lookup(brtab->hash, key);
+
+ if (entry != NULL)
+ *limit_block = entry->limit_block;
+
+ return entry;
+}
+
+/*
+ * Get block numbers from a table entry.
+ *
+ * 'blocks' must point to enough space to hold at least 'nblocks' block
+ * numbers, and any block numbers we manage to get will be written there.
+ * The return value is the number of block numbers actually written.
+ *
+ * We do not return block numbers unless they are greater than or equal to
+ * start_blkno and strictly less than stop_blkno.
+ */
+int
+BlockRefTableEntryGetBlocks(BlockRefTableEntry *entry,
+ BlockNumber start_blkno,
+ BlockNumber stop_blkno,
+ BlockNumber *blocks,
+ int nblocks)
+{
+ uint32 start_chunkno;
+ uint32 stop_chunkno;
+ uint32 chunkno;
+ int nresults = 0;
+
+ Assert(entry != NULL);
+
+ /*
+ * Figure out which chunks could potentially contain blocks of interest.
+ *
+ * We need to be careful about overflow here, because stop_blkno could be
+ * InvalidBlockNumber or something very close to it.
+ */
+ start_chunkno = start_blkno / BLOCKS_PER_CHUNK;
+ stop_chunkno = stop_blkno / BLOCKS_PER_CHUNK;
+ if ((stop_blkno % BLOCKS_PER_CHUNK) != 0)
+ ++stop_chunkno;
+ if (stop_chunkno > entry->nchunks)
+ stop_chunkno = entry->nchunks;
+
+ /*
+ * Loop over chunks.
+ */
+ for (chunkno = start_chunkno; chunkno < stop_chunkno; ++chunkno)
+ {
+ uint16 chunk_usage = entry->chunk_usage[chunkno];
+ BlockRefTableChunk chunk_data = entry->chunk_data[chunkno];
+ unsigned start_offset = 0;
+ unsigned stop_offset = BLOCKS_PER_CHUNK;
+
+ /*
+ * If the start and/or stop block number falls within this chunk, the
+ * whole chunk may not be of interest. Figure out which portion we
+ * care about, if it's not the whole thing.
+ */
+ if (chunkno == start_chunkno)
+ start_offset = start_blkno % BLOCKS_PER_CHUNK;
+ if (chunkno == stop_chunkno)
+ stop_offset = stop_blkno % BLOCKS_PER_CHUNK;
+
+ /*
+ * Handling differs depending on whether this is an array of offsets
+ * or a bitmap.
+ */
+ if (chunk_usage == MAX_ENTRIES_PER_CHUNK)
+ {
+ unsigned i;
+
+ /* It's a bitmap, so test every relevant bit. */
+ for (i = start_offset; i < BLOCKS_PER_CHUNK; ++i)
+ {
+ uint16 w = chunk_data[i / BLOCKS_PER_ENTRY];
+
+ if ((w & (1 << (i % BLOCKS_PER_ENTRY))) != 0)
+ {
+ BlockNumber blkno = chunkno * BLOCKS_PER_CHUNK + i;
+
+ blocks[nresults++] = blkno;
+
+ /* Early exit if we run out of output space. */
+ if (nresults == nblocks)
+ return nresults;
+ }
+ }
+ }
+ else
+ {
+ unsigned i;
+
+ /* It's an array of offsets, so check each one. */
+ for (i = 0; i < chunk_usage; ++i)
+ {
+ uint16 offset = chunk_data[i];
+
+ if (offset >= start_offset && offset < stop_offset)
+ {
+ BlockNumber blkno = chunkno * BLOCKS_PER_CHUNK + offset;
+
+ blocks[nresults++] = blkno;
+
+ /* Early exit if we run out of output space. */
+ if (nresults == nblocks)
+ return nresults;
+ }
+ }
+ }
+ }
+
+ return nresults;
+}
+
+/*
+ * Serialize a block reference table to a file.
+ */
+void
+WriteBlockRefTable(BlockRefTable *brtab,
+ io_callback_fn write_callback,
+ void *write_callback_arg)
+{
+ BlockRefTableSerializedEntry *sdata = NULL;
+ BlockRefTableBuffer buffer;
+ uint32 magic = BLOCKREFTABLE_MAGIC;
+
+ /* Prepare buffer. */
+ memset(&buffer, 0, sizeof(BlockRefTableBuffer));
+ buffer.io_callback = write_callback;
+ buffer.io_callback_arg = write_callback_arg;
+ INIT_CRC32C(buffer.crc);
+
+ /* Write magic number. */
+ BlockRefTableWrite(&buffer, &magic, sizeof(uint32));
+
+ /* Write the entries, assuming there are some. */
+ if (brtab->hash->members > 0)
+ {
+ unsigned i = 0;
+ blockreftable_iterator it;
+ BlockRefTableEntry *brtentry;
+
+ /* Extract entries into serializable format and sort them. */
+ sdata =
+ palloc(brtab->hash->members * sizeof(BlockRefTableSerializedEntry));
+ blockreftable_start_iterate(brtab->hash, &it);
+ while ((brtentry = blockreftable_iterate(brtab->hash, &it)) != NULL)
+ {
+ BlockRefTableSerializedEntry *sentry = &sdata[i++];
+
+ sentry->rlocator = brtentry->key.rlocator;
+ sentry->forknum = brtentry->key.forknum;
+ sentry->limit_block = brtentry->limit_block;
+ sentry->nchunks = brtentry->nchunks;
+
+ /* trim trailing zero entries */
+ while (sentry->nchunks > 0 &&
+ brtentry->chunk_usage[sentry->nchunks - 1] == 0)
+ sentry->nchunks--;
+ }
+ Assert(i == brtab->hash->members);
+ qsort(sdata, i, sizeof(BlockRefTableSerializedEntry),
+ BlockRefTableComparator);
+
+ /* Loop over entries in sorted order and serialize each one. */
+ for (i = 0; i < brtab->hash->members; ++i)
+ {
+ BlockRefTableSerializedEntry *sentry = &sdata[i];
+ BlockRefTableEntry *brtentry;
+ BlockRefTableKey key;
+ unsigned j;
+
+ /* Write the serialized entry itself. */
+ BlockRefTableWrite(&buffer, sentry,
+ sizeof(BlockRefTableSerializedEntry));
+
+ /* Look up the original entry so we can access the chunks. */
+ memcpy(&key.rlocator, &sentry->rlocator, sizeof(RelFileLocator));
+ key.forknum = sentry->forknum;
+ brtentry = blockreftable_lookup(brtab->hash, key);
+ Assert(brtentry != NULL);
+
+ /* Write the untruncated portion of the chunk length array. */
+ if (sentry->nchunks != 0)
+ BlockRefTableWrite(&buffer, brtentry->chunk_usage,
+ sentry->nchunks * sizeof(uint16));
+
+ /* Write the contents of each chunk. */
+ for (j = 0; j < brtentry->nchunks; ++j)
+ {
+ if (brtentry->chunk_usage[j] == 0)
+ continue;
+ BlockRefTableWrite(&buffer, brtentry->chunk_data[j],
+ brtentry->chunk_usage[j] * sizeof(uint16));
+ }
+ }
+ }
+
+ /* Write out appropriate terminator and CRC and flush buffer. */
+ BlockRefTableFileTerminate(&buffer);
+}
+
+/*
+ * Prepare to incrementally read a block reference table file.
+ *
+ * 'read_callback' is a function that can be called to read data from the
+ * underlying file (or other data source) into our internal buffer.
+ *
+ * 'read_callback_arg' is an opaque argument to be passed to read_callback.
+ *
+ * 'error_filename' is the filename that should be included in error messages
+ * if the file is found to be malformed. The value is not copied, so the
+ * caller should ensure that it remains valid until done with this
+ * BlockRefTableReader.
+ *
+ * 'error_callback' is a function to be called if the file is found to be
+ * malformed. This is not used for I/O errors, which must be handled internally
+ * by read_callback.
+ *
+ * 'error_callback_arg' is an opaque arguent to be passed to error_callback.
+ */
+BlockRefTableReader *
+CreateBlockRefTableReader(io_callback_fn read_callback,
+ void *read_callback_arg,
+ char *error_filename,
+ report_error_fn error_callback,
+ void *error_callback_arg)
+{
+ BlockRefTableReader *reader;
+ uint32 magic;
+
+ /* Initialize data structure. */
+ reader = palloc0(sizeof(BlockRefTableReader));
+ reader->buffer.io_callback = read_callback;
+ reader->buffer.io_callback_arg = read_callback_arg;
+ reader->error_filename = error_filename;
+ reader->error_callback = error_callback;
+ reader->error_callback_arg = error_callback_arg;
+ INIT_CRC32C(reader->buffer.crc);
+
+ /* Verify magic number. */
+ BlockRefTableRead(reader, &magic, sizeof(uint32));
+ if (magic != BLOCKREFTABLE_MAGIC)
+ error_callback(error_callback_arg,
+ "file \"%s\" has wrong magic number: expected %u, found %u",
+ error_filename,
+ BLOCKREFTABLE_MAGIC, magic);
+
+ return reader;
+}
+
+/*
+ * Read next relation fork covered by this block reference table file.
+ *
+ * After calling this function, you must call BlockRefTableReaderGetBlocks
+ * until it returns 0 before calling it again.
+ */
+bool
+BlockRefTableReaderNextRelation(BlockRefTableReader *reader,
+ RelFileLocator *rlocator,
+ ForkNumber *forknum,
+ BlockNumber *limit_block)
+{
+ BlockRefTableSerializedEntry sentry;
+ BlockRefTableSerializedEntry zentry = {{0}};
+
+ /*
+ * Sanity check: caller must read all blocks from all chunks before moving
+ * on to the next relation.
+ */
+ Assert(reader->total_chunks == reader->consumed_chunks);
+
+ /* Read serialized entry. */
+ BlockRefTableRead(reader, &sentry,
+ sizeof(BlockRefTableSerializedEntry));
+
+ /*
+ * If we just read the sentinel entry indicating that we've reached the
+ * end, read and check the CRC.
+ */
+ if (memcmp(&sentry, &zentry, sizeof(BlockRefTableSerializedEntry)) == 0)
+ {
+ pg_crc32c expected_crc;
+ pg_crc32c actual_crc;
+
+ /*
+ * We want to know the CRC of the file excluding the 4-byte CRC
+ * itself, so copy the current value of the CRC accumulator before
+ * reading those bytes, and use the copy to finalize the calculation.
+ */
+ expected_crc = reader->buffer.crc;
+ FIN_CRC32C(expected_crc);
+
+ /* Now we can read the actual value. */
+ BlockRefTableRead(reader, &actual_crc, sizeof(pg_crc32c));
+
+ /* Throw an error if there is a mismatch. */
+ if (!EQ_CRC32C(expected_crc, actual_crc))
+ reader->error_callback(reader->error_callback_arg,
+ "file \"%s\" has wrong checksum: expected %08X, found %08X",
+ reader->error_filename, expected_crc, actual_crc);
+
+ return false;
+ }
+
+ /* Read chunk size array. */
+ if (reader->chunk_size != NULL)
+ pfree(reader->chunk_size);
+ reader->chunk_size = palloc(sentry.nchunks * sizeof(uint16));
+ BlockRefTableRead(reader, reader->chunk_size,
+ sentry.nchunks * sizeof(uint16));
+
+ /* Set up for chunk scan. */
+ reader->total_chunks = sentry.nchunks;
+ reader->consumed_chunks = 0;
+
+ /* Return data to caller. */
+ memcpy(rlocator, &sentry.rlocator, sizeof(RelFileLocator));
+ *forknum = sentry.forknum;
+ *limit_block = sentry.limit_block;
+ return true;
+}
+
+/*
+ * Get modified blocks associated with the relation fork returned by
+ * the most recent call to BlockRefTableReaderNextRelation.
+ *
+ * On return, block numbers will be written into the 'blocks' array, whose
+ * length should be passed via 'nblocks'. The return value is the number of
+ * entries actually written into the 'blocks' array, which may be less than
+ * 'nblocks' if we run out of modified blocks in the relation fork before
+ * we run out of room in the array.
+ */
+unsigned
+BlockRefTableReaderGetBlocks(BlockRefTableReader *reader,
+ BlockNumber *blocks,
+ int nblocks)
+{
+ unsigned blocks_found = 0;
+
+ /* Must provide space for at least one block number to be returned. */
+ Assert(nblocks > 0);
+
+ /* Loop collecting blocks to return to caller. */
+ for (;;)
+ {
+ uint16 next_chunk_size;
+
+ /*
+ * If we've read at least one chunk, maybe it contains some block
+ * numbers that could satisfy caller's request.
+ */
+ if (reader->consumed_chunks > 0)
+ {
+ uint32 chunkno = reader->consumed_chunks - 1;
+ uint16 chunk_size = reader->chunk_size[chunkno];
+
+ if (chunk_size == MAX_ENTRIES_PER_CHUNK)
+ {
+ /* Bitmap format, so search for bits that are set. */
+ while (reader->chunk_position < BLOCKS_PER_CHUNK &&
+ blocks_found < nblocks)
+ {
+ uint16 chunkoffset = reader->chunk_position;
+ uint16 w;
+
+ w = reader->chunk_data[chunkoffset / BLOCKS_PER_ENTRY];
+ if ((w & (1u << (chunkoffset % BLOCKS_PER_ENTRY))) != 0)
+ blocks[blocks_found++] =
+ chunkno * BLOCKS_PER_CHUNK + chunkoffset;
+ ++reader->chunk_position;
+ }
+ }
+ else
+ {
+ /* Not in bitmap format, so each entry is a 2-byte offset. */
+ while (reader->chunk_position < chunk_size &&
+ blocks_found < nblocks)
+ {
+ blocks[blocks_found++] = chunkno * BLOCKS_PER_CHUNK
+ + reader->chunk_data[reader->chunk_position];
+ ++reader->chunk_position;
+ }
+ }
+ }
+
+ /* We found enough blocks, so we're done. */
+ if (blocks_found >= nblocks)
+ break;
+
+ /*
+ * We didn't find enough blocks, so we must need the next chunk. If
+ * there are none left, though, then we're done anyway.
+ */
+ if (reader->consumed_chunks == reader->total_chunks)
+ break;
+
+ /*
+ * Read data for next chunk and reset scan position to beginning of
+ * chunk. Note that the next chunk might be empty, in which case we
+ * consume the chunk without actually consuming any bytes from the
+ * underlying file.
+ */
+ next_chunk_size = reader->chunk_size[reader->consumed_chunks];
+ if (next_chunk_size > 0)
+ BlockRefTableRead(reader, reader->chunk_data,
+ next_chunk_size * sizeof(uint16));
+ ++reader->consumed_chunks;
+ reader->chunk_position = 0;
+ }
+
+ return blocks_found;
+}
+
+/*
+ * Release memory used while reading a block reference table from a file.
+ */
+void
+DestroyBlockRefTableReader(BlockRefTableReader *reader)
+{
+ if (reader->chunk_size != NULL)
+ {
+ pfree(reader->chunk_size);
+ reader->chunk_size = NULL;
+ }
+ pfree(reader);
+}
+
+/*
+ * Prepare to write a block reference table file incrementally.
+ *
+ * Caller must be able to supply BlockRefTableEntry objects sorted in the
+ * appropriate order.
+ */
+BlockRefTableWriter *
+CreateBlockRefTableWriter(io_callback_fn write_callback,
+ void *write_callback_arg)
+{
+ BlockRefTableWriter *writer;
+ uint32 magic = BLOCKREFTABLE_MAGIC;
+
+ /* Prepare buffer and CRC check and save callbacks. */
+ writer = palloc0(sizeof(BlockRefTableWriter));
+ writer->buffer.io_callback = write_callback;
+ writer->buffer.io_callback_arg = write_callback_arg;
+ INIT_CRC32C(writer->buffer.crc);
+
+ /* Write magic number. */
+ BlockRefTableWrite(&writer->buffer, &magic, sizeof(uint32));
+
+ return writer;
+}
+
+/*
+ * Append one entry to a block reference table file.
+ *
+ * Note that entries must be written in the proper order, that is, sorted by
+ * tablespace, then database, then relfilenumber, then fork number. Caller
+ * is responsible for supplying data in the correct order. If that seems hard,
+ * use an in-memory BlockRefTable instead.
+ */
+void
+BlockRefTableWriteEntry(BlockRefTableWriter *writer, BlockRefTableEntry *entry)
+{
+ BlockRefTableSerializedEntry sentry;
+ unsigned j;
+
+ /* Convert to serialized entry format. */
+ sentry.rlocator = entry->key.rlocator;
+ sentry.forknum = entry->key.forknum;
+ sentry.limit_block = entry->limit_block;
+ sentry.nchunks = entry->nchunks;
+
+ /* Trim trailing zero entries. */
+ while (sentry.nchunks > 0 && entry->chunk_usage[sentry.nchunks - 1] == 0)
+ sentry.nchunks--;
+
+ /* Write the serialized entry itself. */
+ BlockRefTableWrite(&writer->buffer, &sentry,
+ sizeof(BlockRefTableSerializedEntry));
+
+ /* Write the untruncated portion of the chunk length array. */
+ if (sentry.nchunks != 0)
+ BlockRefTableWrite(&writer->buffer, entry->chunk_usage,
+ sentry.nchunks * sizeof(uint16));
+
+ /* Write the contents of each chunk. */
+ for (j = 0; j < entry->nchunks; ++j)
+ {
+ if (entry->chunk_usage[j] == 0)
+ continue;
+ BlockRefTableWrite(&writer->buffer, entry->chunk_data[j],
+ entry->chunk_usage[j] * sizeof(uint16));
+ }
+}
+
+/*
+ * Finalize an incremental write of a block reference table file.
+ */
+void
+DestroyBlockRefTableWriter(BlockRefTableWriter *writer)
+{
+ BlockRefTableFileTerminate(&writer->buffer);
+ pfree(writer);
+}
+
+/*
+ * Allocate a standalone BlockRefTableEntry.
+ *
+ * When we're manipulating a full in-memory BlockRefTable, the entries are
+ * part of the hash table and are allocated by simplehash. This routine is
+ * used by callers that want to write out a BlockRefTable to a file without
+ * needing to store the whole thing in memory at once.
+ *
+ * Entries allocated by this function can be manipulated using the functions
+ * BlockRefTableEntrySetLimitBlock and BlockRefTableEntryMarkBlockModified
+ * and then written using BlockRefTableWriteEntry and freed using
+ * BlockRefTableFreeEntry.
+ */
+BlockRefTableEntry *
+CreateBlockRefTableEntry(RelFileLocator rlocator, ForkNumber forknum)
+{
+ BlockRefTableEntry *entry = palloc0(sizeof(BlockRefTableEntry));
+
+ memcpy(&entry->key.rlocator, &rlocator, sizeof(RelFileLocator));
+ entry->key.forknum = forknum;
+ entry->limit_block = InvalidBlockNumber;
+
+ return entry;
+}
+
+/*
+ * Update a BlockRefTableEntry with a new value for the "limit block" and
+ * forget any equal-or-higher-numbered modified blocks.
+ *
+ * The "limit block" is the shortest known length of the relation within the
+ * range of WAL records covered by this block reference table.
+ */
+void
+BlockRefTableEntrySetLimitBlock(BlockRefTableEntry *entry,
+ BlockNumber limit_block)
+{
+ unsigned chunkno;
+ unsigned limit_chunkno;
+ unsigned limit_chunkoffset;
+ BlockRefTableChunk limit_chunk;
+
+ /* If we already have an equal or lower limit block, do nothing. */
+ if (limit_block >= entry->limit_block)
+ return;
+
+ /* Record the new limit block value. */
+ entry->limit_block = limit_block;
+
+ /*
+ * Figure out which chunk would store the state of the new limit block,
+ * and which offset within that chunk.
+ */
+ limit_chunkno = limit_block / BLOCKS_PER_CHUNK;
+ limit_chunkoffset = limit_block % BLOCKS_PER_CHUNK;
+
+ /*
+ * If the number of chunks is not large enough for any blocks with equal
+ * or higher block numbers to exist, then there is nothing further to do.
+ */
+ if (limit_chunkno >= entry->nchunks)
+ return;
+
+ /* Discard entire contents of any higher-numbered chunks. */
+ for (chunkno = limit_chunkno + 1; chunkno < entry->nchunks; ++chunkno)
+ entry->chunk_usage[chunkno] = 0;
+
+ /*
+ * Next, we need to discard any offsets within the chunk that would
+ * contain the limit_block. We must handle this differenly depending on
+ * whether the chunk that would contain limit_block is a bitmap or an
+ * array of offsets.
+ */
+ limit_chunk = entry->chunk_data[limit_chunkno];
+ if (entry->chunk_usage[limit_chunkno] == MAX_ENTRIES_PER_CHUNK)
+ {
+ unsigned chunkoffset;
+
+ /* It's a bitmap. Unset bits. */
+ for (chunkoffset = limit_chunkoffset; chunkoffset < BLOCKS_PER_CHUNK;
+ ++chunkoffset)
+ limit_chunk[chunkoffset / BLOCKS_PER_ENTRY] &=
+ ~(1 << (chunkoffset % BLOCKS_PER_ENTRY));
+ }
+ else
+ {
+ unsigned i,
+ j = 0;
+
+ /* It's an offset array. Filter out large offsets. */
+ for (i = 0; i < entry->chunk_usage[limit_chunkno]; ++i)
+ {
+ Assert(j <= i);
+ if (limit_chunk[i] < limit_chunkoffset)
+ limit_chunk[j++] = limit_chunk[i];
+ }
+ Assert(j <= entry->chunk_usage[limit_chunkno]);
+ entry->chunk_usage[limit_chunkno] = j;
+ }
+}
+
+/*
+ * Mark a block in a given BlkRefTableEntry as known to have been modified.
+ */
+void
+BlockRefTableEntryMarkBlockModified(BlockRefTableEntry *entry,
+ ForkNumber forknum,
+ BlockNumber blknum)
+{
+ unsigned chunkno;
+ unsigned chunkoffset;
+ unsigned i;
+
+ /*
+ * Which chunk should store the state of this block? And what is the
+ * offset of this block relative to the start of that chunk?
+ */
+ chunkno = blknum / BLOCKS_PER_CHUNK;
+ chunkoffset = blknum % BLOCKS_PER_CHUNK;
+
+ /*
+ * If 'nchunks' isn't big enough for us to be able to represent the state
+ * of this block, we need to enlarge our arrays.
+ */
+ if (chunkno >= entry->nchunks)
+ {
+ unsigned max_chunks;
+ unsigned extra_chunks;
+
+ /*
+ * New array size is a power of 2, at least 16, big enough so that
+ * chunkno will be a valid array index.
+ */
+ max_chunks = Max(16, entry->nchunks);
+ while (max_chunks < chunkno + 1)
+ chunkno *= 2;
+ Assert(max_chunks > chunkno);
+ extra_chunks = max_chunks - entry->nchunks;
+
+ if (entry->nchunks == 0)
+ {
+ entry->chunk_size = palloc0(sizeof(uint16) * max_chunks);
+ entry->chunk_usage = palloc0(sizeof(uint16) * max_chunks);
+ entry->chunk_data =
+ palloc0(sizeof(BlockRefTableChunk) * max_chunks);
+ }
+ else
+ {
+ entry->chunk_size = repalloc(entry->chunk_size,
+ sizeof(uint16) * max_chunks);
+ memset(&entry->chunk_size[entry->nchunks], 0,
+ extra_chunks * sizeof(uint16));
+ entry->chunk_usage = repalloc(entry->chunk_usage,
+ sizeof(uint16) * max_chunks);
+ memset(&entry->chunk_usage[entry->nchunks], 0,
+ extra_chunks * sizeof(uint16));
+ entry->chunk_data = repalloc(entry->chunk_data,
+ sizeof(BlockRefTableChunk) * max_chunks);
+ memset(&entry->chunk_data[entry->nchunks], 0,
+ extra_chunks * sizeof(BlockRefTableChunk));
+ }
+ entry->nchunks = max_chunks;
+ }
+
+ /*
+ * If the chunk that covers this block number doesn't exist yet, create it
+ * as an array and add the appropriate offset to it. We make it pretty
+ * small initially, because there might only be 1 or a few block
+ * references in this chunk and we don't want to use up too much memory.
+ */
+ if (entry->chunk_size[chunkno] == 0)
+ {
+ entry->chunk_data[chunkno] =
+ palloc(sizeof(uint16) * INITIAL_ENTRIES_PER_CHUNK);
+ entry->chunk_size[chunkno] = INITIAL_ENTRIES_PER_CHUNK;
+ entry->chunk_data[chunkno][0] = chunkoffset;
+ entry->chunk_usage[chunkno] = 1;
+ return;
+ }
+
+ /*
+ * If the number of entries in this chunk is already maximum, it must be a
+ * bitmap. Just set the appropriate bit.
+ */
+ if (entry->chunk_usage[chunkno] == MAX_ENTRIES_PER_CHUNK)
+ {
+ BlockRefTableChunk chunk = entry->chunk_data[chunkno];
+
+ chunk[chunkoffset / BLOCKS_PER_ENTRY] |=
+ 1 << (chunkoffset % BLOCKS_PER_ENTRY);
+ return;
+ }
+
+ /*
+ * There is an existing chunk and it's in array format. Let's find out
+ * whether it already has an entry for this block. If so, we do not need
+ * to do anything.
+ */
+ for (i = 0; i < entry->chunk_usage[chunkno]; ++i)
+ {
+ if (entry->chunk_data[chunkno][i] == chunkoffset)
+ return;
+ }
+
+ /*
+ * If the number of entries currently used is one less than the maximum,
+ * it's time to convert to bitmap format.
+ */
+ if (entry->chunk_usage[chunkno] == MAX_ENTRIES_PER_CHUNK - 1)
+ {
+ BlockRefTableChunk newchunk;
+ unsigned j;
+
+ /* Allocate a new chunk. */
+ newchunk = palloc0(MAX_ENTRIES_PER_CHUNK * sizeof(uint16));
+
+ /* Set the bit for each existing entry. */
+ for (j = 0; j < entry->chunk_usage[chunkno]; ++j)
+ {
+ unsigned coff = entry->chunk_data[chunkno][j];
+
+ newchunk[coff / BLOCKS_PER_ENTRY] |=
+ 1 << (coff % BLOCKS_PER_ENTRY);
+ }
+
+ /* Set the bit for the new entry. */
+ newchunk[chunkoffset / BLOCKS_PER_ENTRY] |=
+ 1 << (chunkoffset % BLOCKS_PER_ENTRY);
+
+ /* Swap the new chunk into place and update metadata. */
+ pfree(entry->chunk_data[chunkno]);
+ entry->chunk_data[chunkno] = newchunk;
+ entry->chunk_size[chunkno] = MAX_ENTRIES_PER_CHUNK;
+ entry->chunk_usage[chunkno] = MAX_ENTRIES_PER_CHUNK;
+ return;
+ }
+
+ /*
+ * OK, we currently have an array, and we don't need to convert to a
+ * bitmap, but we do need to add a new element. If there's not enough
+ * room, we'll have to expand the array.
+ */
+ if (entry->chunk_usage[chunkno] == entry->chunk_size[chunkno])
+ {
+ unsigned newsize = entry->chunk_size[chunkno] * 2;
+
+ Assert(newsize <= MAX_ENTRIES_PER_CHUNK);
+ entry->chunk_data[chunkno] = repalloc(entry->chunk_data[chunkno],
+ newsize * sizeof(uint16));
+ entry->chunk_size[chunkno] = newsize;
+ }
+
+ /* Now we can add the new entry. */
+ entry->chunk_data[chunkno][entry->chunk_usage[chunkno]] =
+ chunkoffset;
+ entry->chunk_usage[chunkno]++;
+}
+
+/*
+ * Release memory for a BlockRefTablEntry that was created by
+ * CreateBlockRefTableEntry.
+ */
+void
+BlockRefTableFreeEntry(BlockRefTableEntry *entry)
+{
+ if (entry->chunk_size != NULL)
+ {
+ pfree(entry->chunk_size);
+ entry->chunk_size = NULL;
+ }
+
+ if (entry->chunk_usage != NULL)
+ {
+ pfree(entry->chunk_usage);
+ entry->chunk_usage = NULL;
+ }
+
+ if (entry->chunk_data != NULL)
+ {
+ pfree(entry->chunk_data);
+ entry->chunk_data = NULL;
+ }
+
+ pfree(entry);
+}
+
+/*
+ * Comparator for BlockRefTableSerializedEntry objects.
+ *
+ * We make the tablespace OID the first column of the sort key to match
+ * the on-disk tree structure.
+ */
+static int
+BlockRefTableComparator(const void *a, const void *b)
+{
+ const BlockRefTableSerializedEntry *sa = a;
+ const BlockRefTableSerializedEntry *sb = b;
+
+ if (sa->rlocator.spcOid > sb->rlocator.spcOid)
+ return 1;
+ if (sa->rlocator.spcOid < sb->rlocator.spcOid)
+ return -1;
+
+ if (sa->rlocator.dbOid > sb->rlocator.dbOid)
+ return 1;
+ if (sa->rlocator.dbOid < sb->rlocator.dbOid)
+ return -1;
+
+ if (sa->rlocator.relNumber > sb->rlocator.relNumber)
+ return 1;
+ if (sa->rlocator.relNumber < sb->rlocator.relNumber)
+ return -1;
+
+ if (sa->forknum > sb->forknum)
+ return 1;
+ if (sa->forknum < sb->forknum)
+ return -1;
+
+ return 0;
+}
+
+/*
+ * Flush any buffered data out of a BlockRefTableBuffer.
+ */
+static void
+BlockRefTableFlush(BlockRefTableBuffer *buffer)
+{
+ buffer->io_callback(buffer->io_callback_arg, buffer->data, buffer->used);
+ buffer->used = 0;
+}
+
+/*
+ * Read data from a BlockRefTableBuffer, and update the running CRC
+ * calculation for the returned data (but not any data that we may have
+ * buffered but not yet actually returned).
+ */
+static void
+BlockRefTableRead(BlockRefTableReader *reader, void *data, int length)
+{
+ BlockRefTableBuffer *buffer = &reader->buffer;
+
+ /* Loop until read is fully satisfied. */
+ while (length > 0)
+ {
+ if (buffer->cursor < buffer->used)
+ {
+ /*
+ * If any buffered data is available, use that to satisfy as much
+ * of the request as possible.
+ */
+ int bytes_to_copy = Min(length, buffer->used - buffer->cursor);
+
+ memcpy(data, &buffer->data[buffer->cursor], bytes_to_copy);
+ COMP_CRC32C(buffer->crc, &buffer->data[buffer->cursor],
+ bytes_to_copy);
+ buffer->cursor += bytes_to_copy;
+ data = ((char *) data) + bytes_to_copy;
+ length -= bytes_to_copy;
+ }
+ else if (length >= BUFSIZE)
+ {
+ /*
+ * If the request length is long, read directly into caller's
+ * buffer.
+ */
+ int bytes_read;
+
+ bytes_read = buffer->io_callback(buffer->io_callback_arg,
+ data, length);
+ COMP_CRC32C(buffer->crc, data, bytes_read);
+ data = ((char *) data) + bytes_read;
+ length -= bytes_read;
+
+ /* If we didn't get anything, that's bad. */
+ if (bytes_read == 0)
+ reader->error_callback(reader->error_callback_arg,
+ "file \"%s\" ends unexpectedly",
+ reader->error_filename);
+ }
+ else
+ {
+ /*
+ * Refill our buffer.
+ */
+ buffer->used = buffer->io_callback(buffer->io_callback_arg,
+ buffer->data, BUFSIZE);
+ buffer->cursor = 0;
+
+ /* If we didn't get anything, that's bad. */
+ if (buffer->used == 0)
+ reader->error_callback(reader->error_callback_arg,
+ "file \"%s\" ends unexpectedly",
+ reader->error_filename);
+ }
+ }
+}
+
+/*
+ * Supply data to a BlockRefTableBuffer for write to the underlying File,
+ * and update the running CRC calculation for that data.
+ */
+static void
+BlockRefTableWrite(BlockRefTableBuffer *buffer, void *data, int length)
+{
+ /* Update running CRC calculation. */
+ COMP_CRC32C(buffer->crc, data, length);
+
+ /* If the new data can't fit into the buffer, flush the buffer. */
+ if (buffer->used + length > BUFSIZE)
+ {
+ buffer->io_callback(buffer->io_callback_arg, buffer->data,
+ buffer->used);
+ buffer->used = 0;
+ }
+
+ /* If the new data would fill the buffer, or more, write it directly. */
+ if (length >= BUFSIZE)
+ {
+ buffer->io_callback(buffer->io_callback_arg, data, length);
+ return;
+ }
+
+ /* Otherwise, copy the new data into the buffer. */
+ memcpy(&buffer->data[buffer->used], data, length);
+ buffer->used += length;
+ Assert(buffer->used <= BUFSIZE);
+}
+
+/*
+ * Generate the sentinel and CRC required at the end of a block reference
+ * table file and flush them out of our internal buffer.
+ */
+static void
+BlockRefTableFileTerminate(BlockRefTableBuffer *buffer)
+{
+ BlockRefTableSerializedEntry zentry = {{0}};
+ pg_crc32c crc;
+
+ /* Write a sentinel indicating that there are no more entries. */
+ BlockRefTableWrite(buffer, &zentry,
+ sizeof(BlockRefTableSerializedEntry));
+
+ /*
+ * Writing the checksum will perturb the ongoing checksum calculation, so
+ * copy the state first and finalize the computation using the copy.
+ */
+ crc = buffer->crc;
+ FIN_CRC32C(crc);
+ BlockRefTableWrite(buffer, &crc, sizeof(pg_crc32c));
+
+ /* Flush any leftover data out of our buffer. */
+ BlockRefTableFlush(buffer);
+}
diff --git a/src/common/meson.build b/src/common/meson.build
index d52dd12bc9..7ad4270a3a 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -4,6 +4,7 @@ common_sources = files(
'archive.c',
'base64.c',
'binaryheap.c',
+ 'blkreftable.c',
'checksum_helper.c',
'compression.c',
'controldata_utils.c',
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index a14126d164..da71580364 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -209,6 +209,7 @@ extern int XLogFileOpen(XLogSegNo segno, TimeLineID tli);
extern void CheckXLogRemoved(XLogSegNo segno, TimeLineID tli);
extern XLogSegNo XLogGetLastRemovedSegno(void);
+extern XLogSegNo XLogGetOldestSegno(TimeLineID tli);
extern void XLogSetAsyncXactLSN(XLogRecPtr asyncXactLSN);
extern void XLogSetReplicationSlotMinimumLSN(XLogRecPtr lsn);
diff --git a/src/include/backup/walsummary.h b/src/include/backup/walsummary.h
new file mode 100644
index 0000000000..8e3dc7b837
--- /dev/null
+++ b/src/include/backup/walsummary.h
@@ -0,0 +1,49 @@
+/*-------------------------------------------------------------------------
+ *
+ * walsummary.h
+ * WAL summary management
+ *
+ * Portions Copyright (c) 2010-2022, PostgreSQL Global Development Group
+ *
+ * src/include/backup/walsummary.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef WALSUMMARY_H
+#define WALSUMMARY_H
+
+#include <time.h>
+
+#include "access/xlogdefs.h"
+#include "nodes/pg_list.h"
+#include "storage/fd.h"
+
+typedef struct WalSummaryIO
+{
+ File file;
+ off_t filepos;
+} WalSummaryIO;
+
+typedef struct WalSummaryFile
+{
+ XLogRecPtr start_lsn;
+ XLogRecPtr end_lsn;
+ TimeLineID tli;
+} WalSummaryFile;
+
+extern List *GetWalSummaries(TimeLineID tli, XLogRecPtr start_lsn,
+ XLogRecPtr end_lsn);
+extern List *FilterWalSummaries(List *wslist, TimeLineID tli,
+ XLogRecPtr start_lsn, XLogRecPtr end_lsn);
+extern bool WalSummariesAreComplete(List *wslist,
+ XLogRecPtr start_lsn, XLogRecPtr end_lsn,
+ XLogRecPtr *missing_lsn);
+extern File OpenWalSummaryFile(WalSummaryFile *ws, bool missing_ok);
+extern void RemoveWalSummaryIfOlderThan(WalSummaryFile *ws,
+ time_t cutoff_time);
+
+extern int ReadWalSummary(void *wal_summary_io, void *data, int length);
+extern int WriteWalSummary(void *wal_summary_io, void *data, int length);
+extern void ReportWalSummaryError(void *callback_arg, char *fmt,...) pg_attribute_printf(2, 3);
+
+#endif /* WALSUMMARY_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index f14aed422a..8c550560a7 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12086,4 +12086,23 @@
proname => 'any_value_transfn', prorettype => 'anyelement',
proargtypes => 'anyelement anyelement', prosrc => 'any_value_transfn' },
+{ oid => '8436',
+ descr => 'list of available WAL summary files',
+ proname => 'pg_available_wal_summaries', prorows => '100',
+ proretset => 't', provolatile => 'v', proparallel => 's',
+ prorettype => 'record', proargtypes => '',
+ proallargtypes => '{int8,pg_lsn,pg_lsn}',
+ proargmodes => '{o,o,o}',
+ proargnames => '{tli,start_lsn,end_lsn}',
+ prosrc => 'pg_available_wal_summaries' },
+{ oid => '8437',
+ descr => 'contents of a WAL sumamry file',
+ proname => 'pg_wal_summary_contents', prorows => '100',
+ proretset => 't', provolatile => 'v', proparallel => 's',
+ prorettype => 'record', proargtypes => 'int8 pg_lsn pg_lsn',
+ proallargtypes => '{int8,pg_lsn,pg_lsn,oid,oid,oid,int2,int8,bool}',
+ proargmodes => '{i,i,i,o,o,o,o,o,o}',
+ proargnames => '{tli,start_lsn,end_lsn,relfilenode,reltablespace,reldatabase,relforknumber,relblocknumber,is_limit_block}',
+ prosrc => 'pg_wal_summary_contents' },
+
]
diff --git a/src/include/common/blkreftable.h b/src/include/common/blkreftable.h
new file mode 100644
index 0000000000..70d6c072d7
--- /dev/null
+++ b/src/include/common/blkreftable.h
@@ -0,0 +1,120 @@
+/*-------------------------------------------------------------------------
+ *
+ * blkreftable.h
+ * Block reference tables.
+ *
+ * A block reference table is used to keep track of which blocks have
+ * been modified by WAL records within a certain LSN range.
+ *
+ * For each relation fork, there is a "limit block number". All existing
+ * blocks greater than or equal to the limit block number must be
+ * considered modified; for those less than the limit block number,
+ * we maintain a bitmap. When a relation fork is created or dropped,
+ * the limit block number should be set to 0. When it's truncated,
+ * the limit block number should be set to the length in blocks to
+ * which it was truncated.
+ *
+ * Portions Copyright (c) 2010-2022, PostgreSQL Global Development Group
+ *
+ * src/include/common/blkreftable.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef BLKREFTABLE_H
+#define BLKREFTABLE_H
+
+#include "storage/block.h"
+#include "storage/relfilelocator.h"
+
+/* Magic number for serialization file format. */
+#define BLOCKREFTABLE_MAGIC 0x652b137b
+
+struct BlockRefTable;
+struct BlockRefTableEntry;
+struct BlockRefTableReader;
+struct BlockRefTableWriter;
+typedef struct BlockRefTable BlockRefTable;
+typedef struct BlockRefTableEntry BlockRefTableEntry;
+typedef struct BlockRefTableReader BlockRefTableReader;
+typedef struct BlockRefTableWriter BlockRefTableWriter;
+
+/*
+ * The return value of io_callback_fn should be the number of bytes read
+ * or written. If an error occurs, the functions should report it and
+ * not return. When used as a write callback, short writes should be retried
+ * or treated as errors, so that if the callback returns, the return value
+ * is always the request length.
+ *
+ * report_error_fn should not return.
+ */
+typedef int (*io_callback_fn) (void *callback_arg, void *data, int length);
+typedef void (*report_error_fn) (void *calblack_arg, char *msg,...) pg_attribute_printf(2, 3);
+
+
+/*
+ * Functions for manipulating an entire in-memory block reference table.
+ */
+extern BlockRefTable *CreateEmptyBlockRefTable(void);
+extern void BlockRefTableSetLimitBlock(BlockRefTable *brtab,
+ const RelFileLocator *rlocator,
+ ForkNumber forknum,
+ BlockNumber limit_block);
+extern void BlockRefTableMarkBlockModified(BlockRefTable *brtab,
+ const RelFileLocator *rlocator,
+ ForkNumber forknum,
+ BlockNumber blknum);
+extern void WriteBlockRefTable(BlockRefTable *brtab,
+ io_callback_fn write_callback,
+ void *write_callback_arg);
+
+extern BlockRefTableEntry *BlockRefTableGetEntry(BlockRefTable *brtab,
+ const RelFileLocator *rlocator,
+ ForkNumber forknum,
+ BlockNumber *limit_block);
+extern int BlockRefTableEntryGetBlocks(BlockRefTableEntry *entry,
+ BlockNumber start_blkno,
+ BlockNumber stop_blkno,
+ BlockNumber *blocks,
+ int nblocks);
+
+/*
+ * Functions for reading a block reference table incrementally from disk.
+ */
+extern BlockRefTableReader *CreateBlockRefTableReader(io_callback_fn read_callback,
+ void *read_callback_arg,
+ char *error_filename,
+ report_error_fn error_callback,
+ void *error_callback_arg);
+extern bool BlockRefTableReaderNextRelation(BlockRefTableReader *reader,
+ RelFileLocator *rlocator,
+ ForkNumber *forknum,
+ BlockNumber *limit_block);
+extern unsigned BlockRefTableReaderGetBlocks(BlockRefTableReader *reader,
+ BlockNumber *blocks,
+ int nblocks);
+extern void DestroyBlockRefTableReader(BlockRefTableReader *reader);
+
+/*
+ * Functions for writing a block reference table incrementally to disk.
+ *
+ * Note that entries must be written in the proper order, that is, sorted by
+ * database, then tablespace, then relfilenumber, then fork number. Caller
+ * is responsible for supplying data in the correct order. If that seems hard,
+ * use an in-memory BlockRefTable instead.
+ */
+extern BlockRefTableWriter *CreateBlockRefTableWriter(io_callback_fn write_callback,
+ void *write_callback_arg);
+extern void BlockRefTableWriteEntry(BlockRefTableWriter *writer,
+ BlockRefTableEntry *entry);
+extern void DestroyBlockRefTableWriter(BlockRefTableWriter *writer);
+
+extern BlockRefTableEntry *CreateBlockRefTableEntry(RelFileLocator rlocator,
+ ForkNumber forknum);
+extern void BlockRefTableEntrySetLimitBlock(BlockRefTableEntry *entry,
+ BlockNumber limit_block);
+extern void BlockRefTableEntryMarkBlockModified(BlockRefTableEntry *entry,
+ ForkNumber forknum,
+ BlockNumber blknum);
+extern void BlockRefTableFreeEntry(BlockRefTableEntry *entry);
+
+#endif /* BLKREFTABLE_H */
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index f0cc651435..ab8f47379a 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -340,6 +340,7 @@ typedef enum BackendType
B_STARTUP,
B_WAL_RECEIVER,
B_WAL_SENDER,
+ B_WAL_SUMMARIZER,
B_WAL_WRITER,
} BackendType;
@@ -446,6 +447,7 @@ typedef enum
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
+ WalSummarizerProcess,
NUM_AUXPROCTYPES /* Must be last! */
} AuxProcType;
@@ -458,6 +460,7 @@ extern PGDLLIMPORT AuxProcType MyAuxProcType;
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
+#define AmWalSummarizerProcess() (MyAuxProcType == WalSummarizerProcess)
/*****************************************************************************
diff --git a/src/include/postmaster/walsummarizer.h b/src/include/postmaster/walsummarizer.h
new file mode 100644
index 0000000000..15db2377dd
--- /dev/null
+++ b/src/include/postmaster/walsummarizer.h
@@ -0,0 +1,31 @@
+/*-------------------------------------------------------------------------
+ *
+ * walsummarizer.h
+ *
+ * Header file for background WAL summarization process.
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/include/postmaster/walsummarizer.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef WALSUMMARIZER_H
+#define WALSUMMARIZER_H
+
+#include "access/xlogdefs.h"
+
+extern bool summarize_wal;
+extern int wal_summarize_keep_time;
+
+extern Size WalSummarizerShmemSize(void);
+extern void WalSummarizerShmemInit(void);
+extern void WalSummarizerMain(void) pg_attribute_noreturn();
+
+extern XLogRecPtr GetOldestUnsummarizedLSN(TimeLineID *tli,
+ bool *lsn_is_exact);
+extern void SetWalSummarizerLatch(void);
+extern XLogRecPtr WaitForWalSummarization(XLogRecPtr lsn, long timeout);
+
+#endif
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index ef74f32693..ee55008082 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -417,11 +417,12 @@ extern PGDLLIMPORT PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer, WAL writer and archiver run during normal
- * operation. Startup process and WAL receiver also consume 2 slots, but WAL
- * writer is launched only after startup has exited, so we only need 5 slots.
+ * Background writer, checkpointer, WAL writer, WAL summarizer, and archiver
+ * run during normal operation. Startup process and WAL receiver also consume
+ * 2 slots, but WAL writer is launched only after startup has exited, so we
+ * only need 6 slots.
*/
-#define NUM_AUXILIARY_PROCS 5
+#define NUM_AUXILIARY_PROCS 6
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/include/utils/guc_tables.h b/src/include/utils/guc_tables.h
index 0c38255961..eaa8c46dda 100644
--- a/src/include/utils/guc_tables.h
+++ b/src/include/utils/guc_tables.h
@@ -72,6 +72,7 @@ enum config_group
WAL_RECOVERY,
WAL_ARCHIVE_RECOVERY,
WAL_RECOVERY_TARGET,
+ WAL_SUMMARIZATION,
REPLICATION_SENDING,
REPLICATION_PRIMARY,
REPLICATION_STANDBY,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 87c1aee379..bf81b91e20 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3998,3 +3998,14 @@ yyscan_t
z_stream
z_streamp
zic_t
+BlockRefTable
+BlockRefTableBuffer
+BlockRefTableEntry
+BlockRefTableKey
+BlockRefTableReader
+BlockRefTableSerializedEntry
+BlockRefTableWriter
+SummarizerReadLocalXLogPrivate
+WalSummarizerData
+WalSummaryFile
+WalSummaryIO
--
2.37.1 (Apple Git-137.1)
[application/octet-stream] v8-0004-Add-support-for-incremental-backup.patch (216.4K, ../../CA+TgmoYiPn7GE_GFvGf+150XOt-Pkdm8h4rzy42jYCRrceUWhg@mail.gmail.com/6-v8-0004-Add-support-for-incremental-backup.patch)
download | inline diff:
From 811a0f412169277aceb89b0dd5cc5acd02d3e658 Mon Sep 17 00:00:00 2001
From: Robert Haas <[email protected]>
Date: Wed, 14 Jun 2023 11:31:29 -0400
Subject: [PATCH v8 4/6] Add support for incremental backup.
To take an incremental backup, you use the new replication command
UPLOAD_MANIFEST to upload the manifest for the prior backup. This
prior backup could either be a full backup or another incremental
backup. You then use BASE_BACKUP with the INCREMENTAL option to take
the backup. pg_basebackup now has an --incremental=PATH_TO_MANIFEST
option to trigger this behavior.
An incremental backup is like a regular full backup except that
some relation files are replaced with files with names like
INCREMENTAL.${ORIGINAL_NAME}, and the backup_label file contains
additional lines identifying it as an incremental backup. The new
pg_combinebackup tool can be used to reconstruct a data directory
from a full backup and a series of incremental backups.
XXX. Should we send the whole backup manifest to the server or, say,
just an LSN?
XXX. Should the timeout when waiting for WAL summaries be configurable?
If it is, then the maximum sleep time for the WAL summarizer needs
to vary accordingly.
XXX. It would be nice (but not essential) to do something about
incremental JSON parsing.
Patch by me. Thanks to Dilip Kumar and Andres Freund for some helpful
design discussions. Reviewed by Dilip Kumar and Jakub Wartak.
---
doc/src/sgml/backup.sgml | 89 +-
doc/src/sgml/config.sgml | 2 -
doc/src/sgml/ref/allfiles.sgml | 1 +
doc/src/sgml/ref/pg_basebackup.sgml | 37 +-
doc/src/sgml/ref/pg_combinebackup.sgml | 228 +++
doc/src/sgml/reference.sgml | 1 +
src/backend/access/transam/xlogbackup.c | 10 +
src/backend/access/transam/xlogrecovery.c | 6 +
src/backend/backup/Makefile | 1 +
src/backend/backup/basebackup.c | 334 ++++-
src/backend/backup/basebackup_incremental.c | 917 ++++++++++++
src/backend/backup/meson.build | 1 +
src/backend/replication/repl_gram.y | 14 +-
src/backend/replication/repl_scanner.l | 2 +
src/backend/replication/walsender.c | 162 ++-
src/backend/storage/ipc/ipci.c | 3 +
src/bin/Makefile | 1 +
src/bin/meson.build | 1 +
src/bin/pg_basebackup/bbstreamer_file.c | 1 +
src/bin/pg_basebackup/pg_basebackup.c | 110 +-
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 4 +-
src/bin/pg_combinebackup/.gitignore | 1 +
src/bin/pg_combinebackup/Makefile | 52 +
src/bin/pg_combinebackup/backup_label.c | 281 ++++
src/bin/pg_combinebackup/backup_label.h | 29 +
src/bin/pg_combinebackup/copy_file.c | 169 +++
src/bin/pg_combinebackup/copy_file.h | 19 +
src/bin/pg_combinebackup/load_manifest.c | 245 ++++
src/bin/pg_combinebackup/load_manifest.h | 67 +
src/bin/pg_combinebackup/meson.build | 38 +
src/bin/pg_combinebackup/pg_combinebackup.c | 1271 +++++++++++++++++
src/bin/pg_combinebackup/reconstruct.c | 681 +++++++++
src/bin/pg_combinebackup/reconstruct.h | 33 +
src/bin/pg_combinebackup/t/001_basic.pl | 23 +
.../pg_combinebackup/t/002_compare_backups.pl | 154 ++
src/bin/pg_combinebackup/t/003_timeline.pl | 90 ++
src/bin/pg_combinebackup/t/004_manifest.pl | 75 +
src/bin/pg_combinebackup/t/005_integrity.pl | 125 ++
src/bin/pg_combinebackup/write_manifest.c | 293 ++++
src/bin/pg_combinebackup/write_manifest.h | 33 +
src/bin/pg_resetwal/pg_resetwal.c | 36 +
src/include/access/xlogbackup.h | 2 +
src/include/backup/basebackup.h | 5 +-
src/include/backup/basebackup_incremental.h | 56 +
src/include/nodes/replnodes.h | 9 +
src/test/perl/PostgreSQL/Test/Cluster.pm | 21 +-
src/tools/pgindent/typedefs.list | 12 +
47 files changed, 5684 insertions(+), 61 deletions(-)
create mode 100644 doc/src/sgml/ref/pg_combinebackup.sgml
create mode 100644 src/backend/backup/basebackup_incremental.c
create mode 100644 src/bin/pg_combinebackup/.gitignore
create mode 100644 src/bin/pg_combinebackup/Makefile
create mode 100644 src/bin/pg_combinebackup/backup_label.c
create mode 100644 src/bin/pg_combinebackup/backup_label.h
create mode 100644 src/bin/pg_combinebackup/copy_file.c
create mode 100644 src/bin/pg_combinebackup/copy_file.h
create mode 100644 src/bin/pg_combinebackup/load_manifest.c
create mode 100644 src/bin/pg_combinebackup/load_manifest.h
create mode 100644 src/bin/pg_combinebackup/meson.build
create mode 100644 src/bin/pg_combinebackup/pg_combinebackup.c
create mode 100644 src/bin/pg_combinebackup/reconstruct.c
create mode 100644 src/bin/pg_combinebackup/reconstruct.h
create mode 100644 src/bin/pg_combinebackup/t/001_basic.pl
create mode 100644 src/bin/pg_combinebackup/t/002_compare_backups.pl
create mode 100644 src/bin/pg_combinebackup/t/003_timeline.pl
create mode 100644 src/bin/pg_combinebackup/t/004_manifest.pl
create mode 100644 src/bin/pg_combinebackup/t/005_integrity.pl
create mode 100644 src/bin/pg_combinebackup/write_manifest.c
create mode 100644 src/bin/pg_combinebackup/write_manifest.h
create mode 100644 src/include/backup/basebackup_incremental.h
diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml
index 8cb24d6ae5..b3468eea3c 100644
--- a/doc/src/sgml/backup.sgml
+++ b/doc/src/sgml/backup.sgml
@@ -857,12 +857,79 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/0
</para>
</sect2>
+ <sect2 id="backup-incremental-backup">
+ <title>Making an Incremental Backup</title>
+
+ <para>
+ You can use <xref linkend="app-pgbasebackup"/> to take an incremental
+ backup by specifying the <literal>--incremental</literal> option. You must
+ supply, as an argument to <literal>--incremental</literal>, the backup
+ manifest to an earlier backup from the same server. In the resulting
+ backup, non-relation files will be included in their entirety, but some
+ relation files may be replaced by smaller incremental files which contain
+ only the blocks which have been changed since the earlier backup and enough
+ metadata to reconstruct the current version of the file.
+ </para>
+
+ <para>
+ To figure out which blocks need to be backed up, the server uses WAL
+ summaries, which are stored in the data directory, inside the directory
+ <literal>pg_wal/summaries</literal>. If the required summary files are not
+ present, an attempt to take an incremental backup will fail. The summaries
+ present in this directory must cover all LSNs from the start LSN of the
+ prior backup to the start LSN of the current backup. Since the server looks
+ for WAL summaries just after establishing the start LSN of the current
+ backup, the necessary summary files probably won't be instantly present
+ on disk, but the server will wait for any missing files to show up.
+ This also helps if the WAL summarization process has fallen behind.
+ However, if the necessary files have already been removed, or if the WAL
+ summarizer doesn't catch up quickly enough, the incremental backup will
+ fail.
+ </para>
+
+ <para>
+ When restoring an incremental backup, it will be necessary to have not
+ only the incremental backup itself but also all earlier backups that
+ are required to supply the blocks omitted from the incremental backup.
+ See <xref linkend="app-pgcombinebackup"/> for further information about
+ this requirement.
+ </para>
+
+ <para>
+ Note that all of the requirements for making use of a full backup also
+ apply to an incremental backup. For instance, you still need all of the
+ WAL segment files generated during and after the file system backup, and
+ any relevant WAL history files. And you still need to create a
+ <literal>recovery.signal</literal> (or <literal>standby.signal</literal>)
+ and perform recovery, as described in
+ <xref linkend="backup-pitr-recovery" />. The requirement to have earlier
+ backups available at restore time and to use
+ <literal>pg_combinebackup</literal> is an additional requirement on top of
+ everything else. Keep in mind that <application>PostgreSQL</application>
+ has no built-in mechanism to figure out which backups are still needed as
+ a basis for restoring later incremental backups. You must keep track of
+ the relationships between your full and incremental backups on your own,
+ and be certain not to remove earlier backups if they might be needed when
+ restoring later incremental backups.
+ </para>
+
+ <para>
+ Incremental backups typically only make sense for relatively large
+ databases where a significant portion of the data does not change, or only
+ changes slowly. For a small database, it's simpler to ignore the existence
+ of incremental backups and simply take full backups, which are simpler
+ to manage. For a large database all of which is heavily modified,
+ incremental backups won't be much smaller than full backups.
+ </para>
+ </sect2>
+
<sect2 id="backup-lowlevel-base-backup">
<title>Making a Base Backup Using the Low Level API</title>
<para>
- The procedure for making a base backup using the low level
- APIs contains a few more steps than
- the <xref linkend="app-pgbasebackup"/> method, but is relatively
+ Instead of taking a full or incremental base backup using
+ <xref linkend="app-pgbasebackup"/>, you can take a base backup using the
+ low-level API. This procedure contains a few more steps than
+ the <application>pg_basebackup</application> method, but is relatively
simple. It is very important that these steps are executed in
sequence, and that the success of a step is verified before
proceeding to the next step.
@@ -1118,7 +1185,8 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true);
</listitem>
<listitem>
<para>
- Restore the database files from your file system backup. Be sure that they
+ If you're restoring a full backup, you can restore the database files
+ directly into the target directories. Be sure that they
are restored with the right ownership (the database system user, not
<literal>root</literal>!) and with the right permissions. If you are using
tablespaces,
@@ -1126,6 +1194,19 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true);
were correctly restored.
</para>
</listitem>
+ <listitem>
+ <para>
+ If you're restoring an incremental backup, you'll need to restore the
+ incremental backup and all earlier backups upon which it directly or
+ indirectly depends to the machine where you are performing the restore.
+ These backups will need to be placed in separate directories, not the
+ target directories where you want the running server to end up.
+ Once this is done, use <xref linkend="app-pgcombinebackup"/> to pull
+ data from the full backup and all of the subsequent incremental backups
+ and write out a synthetic full backup to the target directories. As above,
+ verify that permissions and tablespace links are correct.
+ </para>
+ </listitem>
<listitem>
<para>
Remove any files present in <filename>pg_wal/</filename>; these came from the
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 862a143f17..1e646f5978 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4137,13 +4137,11 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
<sect2 id="runtime-config-wal-summarization">
<title>WAL Summarization</title>
- <!--
<para>
These settings control WAL summarization, a feature which must be
enabled in order to perform an
<link linkend="backup-incremental-backup">incremental backup</link>.
</para>
- -->
<variablelist>
<varlistentry id="guc-summarize-wal" xreflabel="summarize_wal">
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index 54b5f22d6e..fda4690eab 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -202,6 +202,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY pgBasebackup SYSTEM "pg_basebackup.sgml">
<!ENTITY pgbench SYSTEM "pgbench.sgml">
<!ENTITY pgChecksums SYSTEM "pg_checksums.sgml">
+<!ENTITY pgCombinebackup SYSTEM "pg_combinebackup.sgml">
<!ENTITY pgConfig SYSTEM "pg_config-ref.sgml">
<!ENTITY pgControldata SYSTEM "pg_controldata.sgml">
<!ENTITY pgCtl SYSTEM "pg_ctl-ref.sgml">
diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index 0b87fd2d4d..7c183a5cfd 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -38,11 +38,25 @@ PostgreSQL documentation
</para>
<para>
- <application>pg_basebackup</application> makes an exact copy of the database
- cluster's files, while making sure the server is put into and
- out of backup mode automatically. Backups are always taken of the entire
- database cluster; it is not possible to back up individual databases or
- database objects. For selective backups, another tool such as
+ <application>pg_basebackup</application> can take a full or incremental
+ base backup of the database. When used to take a full backup, it makes an
+ exact copy of the database cluster's files. When used to take an incremental
+ backup, some files that would have been part of a full backup may be
+ replaced with incremental versions of the same files, containing only those
+ blocks that have been modified since the reference backup. An incremental
+ backup cannot be used directly; instead,
+ <xref linkend="app-pgcombinebackup"/> must first
+ be used to combine it with the previous backups upon which it depends.
+ See <xref linkend="backup-incremental-backup" /> for more information
+ about incremental backups, and <xref linkend="backup-pitr-recovery" />
+ for steps to recover from a backup.
+ </para>
+
+ <para>
+ In any mode, <application>pg_basebackup</application> makes sure the server
+ is put into and out of backup mode automatically. Backups are always taken of
+ the entire database cluster; it is not possible to back up individual
+ databases or database objects. For selective backups, another tool such as
<xref linkend="app-pgdump"/> must be used.
</para>
@@ -197,6 +211,19 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-i <replaceable class="parameter">old_manifest_file</replaceable></option></term>
+ <term><option>--incremental=<replaceable class="parameter">old_meanifest_file</replaceable></option></term>
+ <listitem>
+ <para>
+ Performs an <link linkend="backup-incremental-backup">incremental
+ backup</link>. The backup manifest for the reference
+ backup must be provided, and will be uploaded to the server, which will
+ respond by sending the requested incremental backup.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-R</option></term>
<term><option>--write-recovery-conf</option></term>
diff --git a/doc/src/sgml/ref/pg_combinebackup.sgml b/doc/src/sgml/ref/pg_combinebackup.sgml
new file mode 100644
index 0000000000..6cac73573f
--- /dev/null
+++ b/doc/src/sgml/ref/pg_combinebackup.sgml
@@ -0,0 +1,228 @@
+<!--
+doc/src/sgml/ref/pg_combinebackup.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="app-pgcombinebackup">
+ <indexterm zone="app-pgcombinebackup">
+ <primary>pg_combinebackup</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle><application>pg_combinebackup</application></refentrytitle>
+ <manvolnum>1</manvolnum>
+ <refmiscinfo>Application</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>pg_combinebackup</refname>
+ <refpurpose>reconstruct a full backup from an incremental backup and dependent backups</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_combinebackup</command>
+ <arg rep="repeat"><replaceable>option</replaceable></arg>
+ <arg rep="repeat"><replaceable>backup_directory</replaceable></arg>
+ </cmdsynopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+ <para>
+ <application>pg_combinebackup</application> is used to reconstruct a
+ synthetic full backup from an
+ <link linkend="backup-incremental-backup">incremental backup</link> and the
+ earlier backups upon which it depends.
+ </para>
+
+ <para>
+ Specify all of the required backups on the command line from oldest to newest.
+ That is, the first backup directory should be the path to the full backup, and
+ the last should be the path to the final incremental backup
+ that you wish to restore. The reconstructed backup will be written to the
+ output directory specified by the <option>-o</option> option.
+ </para>
+
+ <para>
+ Although <application>pg_combinebackup</application> will attempt to verify
+ that the backups you specify form a legal backup chain from which a correct
+ full backup can be reconstructed, it is not designed to help you keep track
+ of which backups depend on which other backups. If you remove the one or
+ more of the previous backups upon which your incremental
+ backup relies, you will not be able to restore it.
+ </para>
+
+ <para>
+ Since the output of <application>pg_combinebackup</application> is a
+ synthetic full backup, it can be used as an input to a future invocation of
+ <application>pg_combinebackup</application>. The synthetic full backup would
+ be specified on the command line in lieu of the chain of backups from which
+ it was reconstructed.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Options</title>
+
+ <para>
+ <variablelist>
+ <varlistentry>
+ <term><option>-d</option></term>
+ <term><option>--debug</option></term>
+ <listitem>
+ <para>
+ Print lots of debug logging output on <filename>stderr</filename>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-T <replaceable class="parameter">olddir</replaceable>=<replaceable class="parameter">newdir</replaceable></option></term>
+ <term><option>--tablespace-mapping=<replaceable class="parameter">olddir</replaceable>=<replaceable class="parameter">newdir</replaceable></option></term>
+ <listitem>
+ <para>
+ Relocates the tablespace in directory <replaceable>olddir</replaceable>
+ to <replaceable>newdir</replaceable> during the backup.
+ <replaceable>olddir</replaceable> is the absolute path of the tablespace
+ as it exists in the first backup specified on the command line,
+ and <replaceable>newdir</replaceable> is the absolute path to use for the
+ tablespace in the reconstructed backup. If either path needs to contain
+ an equal sign (<literal>=</literal>), precede that with a backslash.
+ This option can be specified multiple times for multiple tablespaces.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-N</option></term>
+ <term><option>--no-sync</option></term>
+ <listitem>
+ <para>
+ By default, <command>pg_combinebackup</command> will wait for all files
+ to be written safely to disk. This option causes
+ <command>pg_combinebackup</command> to return without waiting, which is
+ faster, but means that a subsequent operating system crash can leave
+ the output backup corrupt. Generally, this option is useful for testing
+ but should not be used when creating a production installation.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-o <replaceable class="parameter">outputdir</replaceable></option></term>
+ <term><option>--output=<replaceable class="parameter">outputdir</replaceable></option></term>
+ <listitem>
+ <para>
+ Specifies the output directory to which the synthetic full backup
+ should be written. Currently, this argument is required.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--sync-method</option></term>
+ <listitem>
+ <para>
+ When set to <literal>fsync</literal>, which is the default,
+ <command>pg_combinebackup</command> will recursively open and synchronize
+ all files in the backup directory. When the plain format is used, the
+ search for files will follow symbolic links for the WAL directory and
+ each configured tablespace.
+ </para>
+ <para>
+ On Linux, <literal>syncfs</literal> may be used instead to ask the
+ operating system to synchronize the whole file system that contains the
+ backup directory. When the plain format is used,
+ <command>pg_combinebackup</command> will also synchronize the file systems
+ that contain the WAL files and each tablespace. See
+ <xref linkend="syncfs"/> for more information about using
+ <function>syncfs()</function>.
+ </para>
+ <para>
+ This option has no effect when <option>--no-sync</option> is used.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--manifest-checksums=<replaceable class="parameter">algorithm</replaceable></option></term>
+ <listitem>
+ <para>
+ Like <xref linkend="app-pgbasebackup"/>,
+ <application>pg_combinebackup</application> writes a backup manifest
+ in the output directory. This option specifies the checksum algorithm
+ that should be applied to each file included in the backup manifest.
+ Currently, the available algorithms are <literal>NONE</literal>,
+ <literal>CRC32C</literal>, <literal>SHA224</literal>,
+ <literal>SHA256</literal>, <literal>SHA384</literal>,
+ and <literal>SHA512</literal>. The default is <literal>CRC32C</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--no-manifest</option></term>
+ <listitem>
+ <para>
+ Disables generation of a backup manifest. If this option is not
+ specified, a backup manifest for the reconstructed backup will be
+ written to the output directory.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-V</option></term>
+ <term><option>--version</option></term>
+ <listitem>
+ <para>
+ Prints the <application>pg_combinebackup</application> version and exits.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-?</option></term>
+ <term><option>--help</option></term>
+ <listitem>
+ <para>
+ Shows help about <application>pg_combinebackup</application> command line
+ arguments, and exits.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Environment</title>
+
+ <para>
+ This utility, like most other <productname>PostgreSQL</productname> utilities,
+ uses the environment variables supported by <application>libpq</application>
+ (see <xref linkend="libpq-envars"/>).
+ </para>
+
+ <para>
+ The environment variable <envar>PG_COLOR</envar> specifies whether to use
+ color in diagnostic messages. Possible values are
+ <literal>always</literal>, <literal>auto</literal> and
+ <literal>never</literal>.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="app-pgbasebackup"/></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index e11b4b6130..a07d2b5e01 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -250,6 +250,7 @@
&pgamcheck;
&pgBasebackup;
&pgbench;
+ &pgCombinebackup;
&pgConfig;
&pgDump;
&pgDumpall;
diff --git a/src/backend/access/transam/xlogbackup.c b/src/backend/access/transam/xlogbackup.c
index 21d68133ae..f51d4282bb 100644
--- a/src/backend/access/transam/xlogbackup.c
+++ b/src/backend/access/transam/xlogbackup.c
@@ -77,6 +77,16 @@ build_backup_content(BackupState *state, bool ishistoryfile)
appendStringInfo(result, "STOP TIMELINE: %u\n", state->stoptli);
}
+ /* either both istartpoint and istarttli should be set, or neither */
+ Assert(XLogRecPtrIsInvalid(state->istartpoint) == (state->istarttli == 0));
+ if (!XLogRecPtrIsInvalid(state->istartpoint))
+ {
+ appendStringInfo(result, "INCREMENTAL FROM LSN: %X/%X\n",
+ LSN_FORMAT_ARGS(state->istartpoint));
+ appendStringInfo(result, "INCREMENTAL FROM TLI: %u\n",
+ state->istarttli);
+ }
+
data = result->data;
pfree(result);
diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c
index c61566666a..7d2501274e 100644
--- a/src/backend/access/transam/xlogrecovery.c
+++ b/src/backend/access/transam/xlogrecovery.c
@@ -1295,6 +1295,12 @@ read_backup_label(XLogRecPtr *checkPointLoc, TimeLineID *backupLabelTLI,
tli_from_file, BACKUP_LABEL_FILE)));
}
+ if (fscanf(lfp, "INCREMENTAL FROM LSN: %X/%X\n", &hi, &lo) > 0)
+ ereport(FATAL,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("this is an incremental backup, not a data directory"),
+ errhint("Use pg_combinebackup to reconstruct a valid data directory.")));
+
if (ferror(lfp) || FreeFile(lfp))
ereport(FATAL,
(errcode_for_file_access(),
diff --git a/src/backend/backup/Makefile b/src/backend/backup/Makefile
index a67b3c58d4..751e6d3d5e 100644
--- a/src/backend/backup/Makefile
+++ b/src/backend/backup/Makefile
@@ -19,6 +19,7 @@ OBJS = \
basebackup.o \
basebackup_copy.o \
basebackup_gzip.o \
+ basebackup_incremental.o \
basebackup_lz4.o \
basebackup_zstd.o \
basebackup_progress.o \
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 4ba63ad8a6..8a70a9ae41 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -20,8 +20,10 @@
#include "access/xlogbackup.h"
#include "backup/backup_manifest.h"
#include "backup/basebackup.h"
+#include "backup/basebackup_incremental.h"
#include "backup/basebackup_sink.h"
#include "backup/basebackup_target.h"
+#include "catalog/pg_tablespace_d.h"
#include "commands/defrem.h"
#include "common/compression.h"
#include "common/file_perm.h"
@@ -64,6 +66,7 @@ typedef struct
bool fastcheckpoint;
bool nowait;
bool includewal;
+ bool incremental;
uint32 maxrate;
bool sendtblspcmapfile;
bool send_to_client;
@@ -76,21 +79,28 @@ typedef struct
} basebackup_options;
static int64 sendTablespace(bbsink *sink, char *path, Oid spcoid, bool sizeonly,
- struct backup_manifest_info *manifest);
+ struct backup_manifest_info *manifest,
+ IncrementalBackupInfo *ib);
static int64 sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
List *tablespaces, bool sendtblspclinks,
- backup_manifest_info *manifest, Oid spcoid);
+ backup_manifest_info *manifest, Oid spcoid,
+ IncrementalBackupInfo *ib);
static bool sendFile(bbsink *sink, const char *readfilename, const char *tarfilename,
struct stat *statbuf, bool missing_ok,
Oid dboid, Oid spcoid, RelFileNumber relfilenumber,
unsigned segno,
- backup_manifest_info *manifest);
+ backup_manifest_info *manifest,
+ unsigned num_incremental_blocks,
+ BlockNumber *incremental_blocks,
+ unsigned truncation_block_length);
static off_t read_file_data_into_buffer(bbsink *sink,
const char *readfilename, int fd,
off_t offset, size_t length,
BlockNumber blkno,
bool verify_checksum,
int *checksum_failures);
+static void push_to_sink(bbsink *sink, pg_checksum_context *checksum_ctx,
+ size_t *bytes_done, void *data, size_t length);
static bool verify_page_checksum(Page page, XLogRecPtr start_lsn,
BlockNumber blkno,
uint16 *expected_checksum);
@@ -102,7 +112,8 @@ static int64 _tarWriteHeader(bbsink *sink, const char *filename,
bool sizeonly);
static void _tarWritePadding(bbsink *sink, int len);
static void convert_link_to_directory(const char *pathbuf, struct stat *statbuf);
-static void perform_base_backup(basebackup_options *opt, bbsink *sink);
+static void perform_base_backup(basebackup_options *opt, bbsink *sink,
+ IncrementalBackupInfo *ib);
static void parse_basebackup_options(List *options, basebackup_options *opt);
static int compareWalFileNames(const ListCell *a, const ListCell *b);
static int basebackup_read_file(int fd, char *buf, size_t nbytes, off_t offset,
@@ -220,7 +231,8 @@ static const struct exclude_list_item excludeFiles[] =
* clobbered by longjmp" from stupider versions of gcc.
*/
static void
-perform_base_backup(basebackup_options *opt, bbsink *sink)
+perform_base_backup(basebackup_options *opt, bbsink *sink,
+ IncrementalBackupInfo *ib)
{
bbsink_state state;
XLogRecPtr endptr;
@@ -270,6 +282,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
ListCell *lc;
tablespaceinfo *newti;
+ /* If this is an incremental backup, execute preparatory steps. */
+ if (ib != NULL)
+ PrepareForIncrementalBackup(ib, backup_state);
+
/* Add a node for the base directory at the end */
newti = palloc0(sizeof(tablespaceinfo));
newti->size = -1;
@@ -289,10 +305,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
if (tmp->path == NULL)
tmp->size = sendDir(sink, ".", 1, true, state.tablespaces,
- true, NULL, InvalidOid);
+ true, NULL, InvalidOid, NULL);
else
tmp->size = sendTablespace(sink, tmp->path, tmp->oid, true,
- NULL);
+ NULL, NULL);
state.bytes_total += tmp->size;
}
state.bytes_total_is_valid = true;
@@ -330,7 +346,7 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
/* Then the bulk of the files... */
sendDir(sink, ".", 1, false, state.tablespaces,
- sendtblspclinks, &manifest, InvalidOid);
+ sendtblspclinks, &manifest, InvalidOid, ib);
/* ... and pg_control after everything else. */
if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0)
@@ -340,7 +356,7 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
XLOG_CONTROL_FILE)));
sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf,
false, InvalidOid, InvalidOid,
- InvalidRelFileNumber, 0, &manifest);
+ InvalidRelFileNumber, 0, &manifest, 0, NULL, 0);
}
else
{
@@ -348,7 +364,7 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_begin_archive(sink, archive_name);
- sendTablespace(sink, ti->path, ti->oid, false, &manifest);
+ sendTablespace(sink, ti->path, ti->oid, false, &manifest, ib);
}
/*
@@ -610,7 +626,7 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
sendFile(sink, pathbuf, pathbuf, &statbuf, false,
InvalidOid, InvalidOid, InvalidRelFileNumber, 0,
- &manifest);
+ &manifest, 0, NULL, 0);
/* unconditionally mark file as archived */
StatusFilePath(pathbuf, fname, ".done");
@@ -686,6 +702,7 @@ parse_basebackup_options(List *options, basebackup_options *opt)
bool o_checkpoint = false;
bool o_nowait = false;
bool o_wal = false;
+ bool o_incremental = false;
bool o_maxrate = false;
bool o_tablespace_map = false;
bool o_noverify_checksums = false;
@@ -764,6 +781,15 @@ parse_basebackup_options(List *options, basebackup_options *opt)
opt->includewal = defGetBoolean(defel);
o_wal = true;
}
+ else if (strcmp(defel->defname, "incremental") == 0)
+ {
+ if (o_incremental)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("duplicate option \"%s\"", defel->defname)));
+ opt->incremental = defGetBoolean(defel);
+ o_incremental = true;
+ }
else if (strcmp(defel->defname, "max_rate") == 0)
{
int64 maxrate;
@@ -956,7 +982,7 @@ parse_basebackup_options(List *options, basebackup_options *opt)
* the filesystem, bypassing the buffer cache.
*/
void
-SendBaseBackup(BaseBackupCmd *cmd)
+SendBaseBackup(BaseBackupCmd *cmd, IncrementalBackupInfo *ib)
{
basebackup_options opt;
bbsink *sink;
@@ -980,6 +1006,20 @@ SendBaseBackup(BaseBackupCmd *cmd)
set_ps_display(activitymsg);
}
+ /*
+ * If we're asked to perform an incremental backup and the user has not
+ * supplied a manifest, that's an ERROR.
+ *
+ * If we're asked to perform a full backup and the user did supply a
+ * manifest, just ignore it.
+ */
+ if (!opt.incremental)
+ ib = NULL;
+ else if (ib == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("must UPLOAD_MANIFEST before performing an incremental BASE_BACKUP")));
+
/*
* If the target is specifically 'client' then set up to stream the backup
* to the client; otherwise, it's being sent someplace else and should not
@@ -1011,7 +1051,7 @@ SendBaseBackup(BaseBackupCmd *cmd)
*/
PG_TRY();
{
- perform_base_backup(&opt, sink);
+ perform_base_backup(&opt, sink, ib);
}
PG_FINALLY();
{
@@ -1086,7 +1126,7 @@ sendFileWithContent(bbsink *sink, const char *filename, const char *content,
*/
static int64
sendTablespace(bbsink *sink, char *path, Oid spcoid, bool sizeonly,
- backup_manifest_info *manifest)
+ backup_manifest_info *manifest, IncrementalBackupInfo *ib)
{
int64 size;
char pathbuf[MAXPGPATH];
@@ -1120,7 +1160,7 @@ sendTablespace(bbsink *sink, char *path, Oid spcoid, bool sizeonly,
/* Send all the files in the tablespace version directory */
size += sendDir(sink, pathbuf, strlen(path), sizeonly, NIL, true, manifest,
- spcoid);
+ spcoid, ib);
return size;
}
@@ -1140,7 +1180,7 @@ sendTablespace(bbsink *sink, char *path, Oid spcoid, bool sizeonly,
static int64
sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
List *tablespaces, bool sendtblspclinks, backup_manifest_info *manifest,
- Oid spcoid)
+ Oid spcoid, IncrementalBackupInfo *ib)
{
DIR *dir;
struct dirent *de;
@@ -1149,7 +1189,16 @@ sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
int64 size = 0;
const char *lastDir; /* Split last dir from parent path. */
bool isRelationDir = false; /* Does directory contain relations? */
+ bool isGlobalDir = false;
Oid dboid = InvalidOid;
+ BlockNumber *relative_block_numbers = NULL;
+
+ /*
+ * Since this array is relatively large, avoid putting it on the stack.
+ * But we don't need it at all if this is not an incremental backup.
+ */
+ if (ib != NULL)
+ relative_block_numbers = palloc(sizeof(BlockNumber) * RELSEG_SIZE);
/*
* Determine if the current path is a database directory that can contain
@@ -1182,7 +1231,10 @@ sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
}
}
else if (strcmp(path, "./global") == 0)
+ {
isRelationDir = true;
+ isGlobalDir = true;
+ }
dir = AllocateDir(path);
while ((de = ReadDir(dir, path)) != NULL)
@@ -1331,11 +1383,13 @@ sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
&statbuf, sizeonly);
/*
- * Also send archive_status directory (by hackishly reusing
- * statbuf from above ...).
+ * Also send archive_status and summaries directories (by
+ * hackishly reusing statbuf from above ...).
*/
size += _tarWriteHeader(sink, "./pg_wal/archive_status", NULL,
&statbuf, sizeonly);
+ size += _tarWriteHeader(sink, "./pg_wal/summaries", NULL,
+ &statbuf, sizeonly);
continue; /* don't recurse into pg_wal */
}
@@ -1404,33 +1458,88 @@ sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
if (!skip_this_dir)
size += sendDir(sink, pathbuf, basepathlen, sizeonly, tablespaces,
- sendtblspclinks, manifest, spcoid);
+ sendtblspclinks, manifest, spcoid, ib);
}
else if (S_ISREG(statbuf.st_mode))
{
bool sent = false;
+ unsigned num_blocks_required = 0;
+ unsigned truncation_block_length = 0;
+ char tarfilenamebuf[MAXPGPATH * 2];
+ char *tarfilename = pathbuf + basepathlen + 1;
+ FileBackupMethod method = BACK_UP_FILE_FULLY;
- if (!sizeonly)
- sent = sendFile(sink, pathbuf, pathbuf + basepathlen + 1, &statbuf,
- true, dboid, spcoid,
- relfilenumber, segno, manifest);
+ if (ib != NULL && isRelationFile)
+ {
+ Oid relspcoid;
+ char *lookup_path;
- if (sent || sizeonly)
+ if (OidIsValid(spcoid))
+ {
+ relspcoid = spcoid;
+ lookup_path = psprintf("pg_tblspc/%u/%s", spcoid,
+ pathbuf + basepathlen + 1);
+ }
+ else
+ {
+ if (isGlobalDir)
+ relspcoid = GLOBALTABLESPACE_OID;
+ else
+ relspcoid = DEFAULTTABLESPACE_OID;
+ lookup_path = pstrdup(pathbuf + basepathlen + 1);
+ }
+
+ method = GetFileBackupMethod(ib, lookup_path, dboid, relspcoid,
+ relfilenumber, relForkNum,
+ segno, statbuf.st_size,
+ &num_blocks_required,
+ relative_block_numbers,
+ &truncation_block_length);
+ if (method == BACK_UP_FILE_INCREMENTALLY)
+ {
+ statbuf.st_size =
+ GetIncrementalFileSize(num_blocks_required);
+ snprintf(tarfilenamebuf, sizeof(tarfilenamebuf),
+ "%s/INCREMENTAL.%s",
+ path + basepathlen + 1,
+ de->d_name);
+ tarfilename = tarfilenamebuf;
+ }
+
+ pfree(lookup_path);
+ }
+
+ if (method != DO_NOT_BACK_UP_FILE)
{
- /* Add size. */
- size += statbuf.st_size;
+ if (!sizeonly)
+ sent = sendFile(sink, pathbuf, tarfilename, &statbuf,
+ true, dboid, spcoid,
+ relfilenumber, segno, manifest,
+ num_blocks_required,
+ method == BACK_UP_FILE_INCREMENTALLY ? relative_block_numbers : NULL,
+ truncation_block_length);
+
+ if (sent || sizeonly)
+ {
+ /* Add size. */
+ size += statbuf.st_size;
- /* Pad to a multiple of the tar block size. */
- size += tarPaddingBytesRequired(statbuf.st_size);
+ /* Pad to a multiple of the tar block size. */
+ size += tarPaddingBytesRequired(statbuf.st_size);
- /* Size of the header for the file. */
- size += TAR_BLOCK_SIZE;
+ /* Size of the header for the file. */
+ size += TAR_BLOCK_SIZE;
+ }
}
}
else
ereport(WARNING,
(errmsg("skipping special file \"%s\"", pathbuf)));
}
+
+ if (relative_block_numbers != NULL)
+ pfree(relative_block_numbers);
+
FreeDir(dir);
return size;
}
@@ -1443,6 +1552,12 @@ sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
* If dboid is anything other than InvalidOid then any checksum failures
* detected will get reported to the cumulative stats system.
*
+ * If the file is to be sent incrementally, then num_incremental_blocks
+ * should be the number of blocks to be sent, and incremental_blocks
+ * an array of block numbers relative to the start of the current segment.
+ * If the whole file is to be sent, then incremental_blocks should be NULL,
+ * and num_incremental_blocks can have any value, as it will be ignored.
+ *
* Returns true if the file was successfully sent, false if 'missing_ok',
* and the file did not exist.
*/
@@ -1450,7 +1565,8 @@ static bool
sendFile(bbsink *sink, const char *readfilename, const char *tarfilename,
struct stat *statbuf, bool missing_ok, Oid dboid, Oid spcoid,
RelFileNumber relfilenumber, unsigned segno,
- backup_manifest_info *manifest)
+ backup_manifest_info *manifest, unsigned num_incremental_blocks,
+ BlockNumber *incremental_blocks, unsigned truncation_block_length)
{
int fd;
BlockNumber blkno = 0;
@@ -1459,6 +1575,7 @@ sendFile(bbsink *sink, const char *readfilename, const char *tarfilename,
pgoff_t bytes_done = 0;
bool verify_checksum = false;
pg_checksum_context checksum_ctx;
+ int ibindex = 0;
if (pg_checksum_init(&checksum_ctx, manifest->checksum_type) < 0)
elog(ERROR, "could not initialize checksum of file \"%s\"",
@@ -1491,22 +1608,111 @@ sendFile(bbsink *sink, const char *readfilename, const char *tarfilename,
RelFileNumberIsValid(relfilenumber))
verify_checksum = true;
+ /*
+ * If we're sending an incremental file, write the file header.
+ */
+ if (incremental_blocks != NULL)
+ {
+ unsigned magic = INCREMENTAL_MAGIC;
+ size_t header_bytes_done = 0;
+
+ /* Emit header data. */
+ push_to_sink(sink, &checksum_ctx, &header_bytes_done,
+ &magic, sizeof(magic));
+ push_to_sink(sink, &checksum_ctx, &header_bytes_done,
+ &num_incremental_blocks, sizeof(num_incremental_blocks));
+ push_to_sink(sink, &checksum_ctx, &header_bytes_done,
+ &truncation_block_length, sizeof(truncation_block_length));
+ push_to_sink(sink, &checksum_ctx, &header_bytes_done,
+ incremental_blocks,
+ sizeof(BlockNumber) * num_incremental_blocks);
+
+ /* Flush out any data still in the buffer so it's again empty. */
+ if (header_bytes_done > 0)
+ {
+ bbsink_archive_contents(sink, header_bytes_done);
+ if (pg_checksum_update(&checksum_ctx,
+ (uint8 *) sink->bbs_buffer,
+ header_bytes_done) < 0)
+ elog(ERROR, "could not update checksum of base backup");
+ }
+
+ /* Update our notion of file position. */
+ bytes_done += sizeof(magic);
+ bytes_done += sizeof(num_incremental_blocks);
+ bytes_done += sizeof(truncation_block_length);
+ bytes_done += sizeof(BlockNumber) * num_incremental_blocks;
+ }
+
/*
* Loop until we read the amount of data the caller told us to expect. The
* file could be longer, if it was extended while we were sending it, but
* for a base backup we can ignore such extended data. It will be restored
* from WAL.
*/
- while (bytes_done < statbuf->st_size)
+ while (1)
{
- size_t remaining = statbuf->st_size - bytes_done;
+ /*
+ * Determine whether we've read all the data that we need, and if not,
+ * read some more.
+ */
+ if (incremental_blocks == NULL)
+ {
+ size_t remaining = statbuf->st_size - bytes_done;
+
+ /*
+ * If we've read the required number of bytes, then it's time to
+ * stop.
+ */
+ if (bytes_done >= statbuf->st_size)
+ break;
+
+ /*
+ * Read as many bytes as will fit in the buffer, or however many
+ * are left to read, whichever is less.
+ */
+ cnt = read_file_data_into_buffer(sink, readfilename, fd,
+ bytes_done, remaining,
+ blkno + segno * RELSEG_SIZE,
+ verify_checksum,
+ &checksum_failures);
+ }
+ else
+ {
+ BlockNumber relative_blkno;
- /* Try to read some more data. */
- cnt = read_file_data_into_buffer(sink, readfilename, fd, bytes_done,
- remaining,
- blkno + segno * RELSEG_SIZE,
- verify_checksum,
- &checksum_failures);
+ /*
+ * If we've read all the blocks, then it's time to stop.
+ */
+ if (ibindex >= num_incremental_blocks)
+ break;
+
+ /*
+ * Read just one block, whichever one is the next that we're
+ * supposed to include.
+ */
+ relative_blkno = incremental_blocks[ibindex++];
+ cnt = read_file_data_into_buffer(sink, readfilename, fd,
+ relative_blkno * BLCKSZ,
+ BLCKSZ,
+ relative_blkno + segno * RELSEG_SIZE,
+ verify_checksum,
+ &checksum_failures);
+
+ /*
+ * If we get a partial read, that must mean that the relation is
+ * being truncated. Ultimately, it should be truncated to a
+ * multiple of BLCKSZ, since this path should only be reached for
+ * relation files, but we might transiently observe an
+ * intermediate value.
+ *
+ * It should be fine to treat this just as if the entire block had
+ * been truncated away - i.e. fill this and all later blocks with
+ * zeroes. WAL replay will fix things up.
+ */
+ if (cnt < BLCKSZ)
+ break;
+ }
/*
* If the amount of data we were able to read was not a multiple of
@@ -1689,6 +1895,56 @@ read_file_data_into_buffer(bbsink *sink, const char *readfilename, int fd,
return cnt;
}
+/*
+ * Push data into a bbsink.
+ *
+ * It's better, when possible, to read data directly into the bbsink's buffer,
+ * rather than using this function to copy it into the buffer; this function is
+ * for cases where that approach is not practical.
+ *
+ * bytes_done should point to a count of the number of bytes that are
+ * currently used in the bbsink's buffer. Upon return, the bytes identified by
+ * data and length will have been copied into the bbsink's buffer, flushing
+ * as required, and *bytes_done will have been updated accordingly. If the
+ * buffer was flushed, the previous contents will also have been fed to
+ * checksum_ctx.
+ *
+ * Note that after one or more calls to this function it is the caller's
+ * responsibility to perform any required final flush.
+ */
+static void
+push_to_sink(bbsink *sink, pg_checksum_context *checksum_ctx,
+ size_t *bytes_done, void *data, size_t length)
+{
+ while (length > 0)
+ {
+ size_t bytes_to_copy;
+
+ /*
+ * We use < here rather than <= so that if the data exactly fills the
+ * remaining buffer space, we trigger a flush now.
+ */
+ if (length < sink->bbs_buffer_length - *bytes_done)
+ {
+ /* Append remaining data to buffer. */
+ memcpy(sink->bbs_buffer + *bytes_done, data, length);
+ *bytes_done += length;
+ return;
+ }
+
+ /* Copy until buffer is full and flush it. */
+ bytes_to_copy = sink->bbs_buffer_length - *bytes_done;
+ memcpy(sink->bbs_buffer + *bytes_done, data, bytes_to_copy);
+ data = ((char *) data) + bytes_to_copy;
+ length -= bytes_to_copy;
+ bbsink_archive_contents(sink, sink->bbs_buffer_length);
+ if (pg_checksum_update(checksum_ctx, (uint8 *) sink->bbs_buffer,
+ sink->bbs_buffer_length) < 0)
+ elog(ERROR, "could not update checksum");
+ *bytes_done = 0;
+ }
+}
+
/*
* Try to verify the checksum for the provided page, if it seems appropriate
* to do so.
diff --git a/src/backend/backup/basebackup_incremental.c b/src/backend/backup/basebackup_incremental.c
new file mode 100644
index 0000000000..12699b5984
--- /dev/null
+++ b/src/backend/backup/basebackup_incremental.c
@@ -0,0 +1,917 @@
+/*-------------------------------------------------------------------------
+ *
+ * basebackup_incremental.c
+ * code for incremental backup support
+ *
+ * This code isn't actually in charge of taking an incremental backup;
+ * the actual construction of the incremental backup happens in
+ * basebackup.c. Here, we're concerned with providing the necessary
+ * supports for that operation. In particular, we need to parse the
+ * backup manifest supplied by the user taking the incremental backup
+ * and extract the required information from it.
+ *
+ * Portions Copyright (c) 2010-2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/backup/basebackup_incremental.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/timeline.h"
+#include "access/xlog.h"
+#include "access/xlogrecovery.h"
+#include "backup/basebackup_incremental.h"
+#include "backup/walsummary.h"
+#include "common/blkreftable.h"
+#include "common/parse_manifest.h"
+#include "common/hashfn.h"
+#include "postmaster/walsummarizer.h"
+
+#define BLOCKS_PER_READ 512
+
+/*
+ * Details extracted from the WAL ranges present in the supplied backup manifest.
+ */
+typedef struct
+{
+ TimeLineID tli;
+ XLogRecPtr start_lsn;
+ XLogRecPtr end_lsn;
+} backup_wal_range;
+
+/*
+ * Details extracted from the file list present in the supplied backup manifest.
+ */
+typedef struct
+{
+ uint32 status;
+ const char *path;
+ size_t size;
+} backup_file_entry;
+
+static uint32 hash_string_pointer(const char *s);
+#define SH_PREFIX backup_file
+#define SH_ELEMENT_TYPE backup_file_entry
+#define SH_KEY_TYPE const char *
+#define SH_KEY path
+#define SH_HASH_KEY(tb, key) hash_string_pointer(key)
+#define SH_EQUAL(tb, a, b) (strcmp(a, b) == 0)
+#define SH_SCOPE static inline
+#define SH_DECLARE
+#define SH_DEFINE
+#include "lib/simplehash.h"
+
+struct IncrementalBackupInfo
+{
+ /* Memory context for this object and its subsidiary objects. */
+ MemoryContext mcxt;
+
+ /* Temporary buffer for storing the manifest while parsing it. */
+ StringInfoData buf;
+
+ /* WAL ranges extracted from the backup manifest. */
+ List *manifest_wal_ranges;
+
+ /*
+ * Files extracted from the backup manifest.
+ *
+ * We don't really need this information, because we use WAL summaries to
+ * figure what's changed. It would be unsafe to just rely on the list of
+ * files that existed before, because it's possible for a file to be
+ * removed and a new one created with the same name and different
+ * contents. In such cases, the whole file must still be sent. We can tell
+ * from the WAL summaries whether that happened, but not from the file
+ * list.
+ *
+ * Nonetheless, this data is useful for sanity checking. If a file that we
+ * think we shouldn't need to send is not present in the manifest for the
+ * prior backup, something has gone terribly wrong. We retain the file
+ * names and sizes, but not the checksums or last modified times, for
+ * which we have no use.
+ *
+ * One significant downside of storing this data is that it consumes
+ * memory. If that turns out to be a problem, we might have to decide not
+ * to retain this information, or to make it optional.
+ */
+ backup_file_hash *manifest_files;
+
+ /*
+ * Block-reference table for the incremental backup.
+ *
+ * It's possible that storing the entire block-reference table in memory
+ * will be a problem for some users. The in-memory format that we're using
+ * here is pretty efficient, converging to little more than 1 bit per
+ * block for relation forks with large numbers of modified blocks. It's
+ * possible, however, that if you try to perform an incremental backup of
+ * a database with a sufficiently large number of relations on a
+ * sufficiently small machine, you could run out of memory here. If that
+ * turns out to be a problem in practice, we'll need to be more clever.
+ */
+ BlockRefTable *brtab;
+};
+
+static void manifest_process_file(JsonManifestParseContext *,
+ char *pathname,
+ size_t size,
+ pg_checksum_type checksum_type,
+ int checksum_length,
+ uint8 *checksum_payload);
+static void manifest_process_wal_range(JsonManifestParseContext *,
+ TimeLineID tli,
+ XLogRecPtr start_lsn,
+ XLogRecPtr end_lsn);
+static void manifest_report_error(JsonManifestParseContext *ib,
+ const char *fmt,...)
+ pg_attribute_printf(2, 3) pg_attribute_noreturn();
+static int compare_block_numbers(const void *a, const void *b);
+
+/*
+ * Create a new object for storing information extracted from the manifest
+ * supplied when creating an incremental backup.
+ */
+IncrementalBackupInfo *
+CreateIncrementalBackupInfo(MemoryContext mcxt)
+{
+ IncrementalBackupInfo *ib;
+ MemoryContext oldcontext;
+
+ oldcontext = MemoryContextSwitchTo(mcxt);
+
+ ib = palloc0(sizeof(IncrementalBackupInfo));
+ ib->mcxt = mcxt;
+ initStringInfo(&ib->buf);
+
+ /*
+ * It's hard to guess how many files a "typical" installation will have in
+ * the data directory, but a fresh initdb creates almost 1000 files as of
+ * this writing, so it seems to make sense for our estimate to
+ * substantially higher.
+ */
+ ib->manifest_files = backup_file_create(mcxt, 10000, NULL);
+
+ MemoryContextSwitchTo(oldcontext);
+
+ return ib;
+}
+
+/*
+ * Before taking an incremental backup, the caller must supply the backup
+ * manifest from a prior backup. Each chunk of manifest data recieved
+ * from the client should be passed to this function.
+ */
+void
+AppendIncrementalManifestData(IncrementalBackupInfo *ib, const char *data,
+ int len)
+{
+ MemoryContext oldcontext;
+
+ /* Switch to our memory context. */
+ oldcontext = MemoryContextSwitchTo(ib->mcxt);
+
+ /*
+ * XXX. Our json parser is at present incapable of parsing json blobs
+ * incrementally, so we have to accumulate the entire backup manifest
+ * before we can do anything with it. This should really be fixed, since
+ * some users might have very large numbers of files in the data
+ * directory.
+ */
+ appendBinaryStringInfo(&ib->buf, data, len);
+
+ /* Switch back to previous memory context. */
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Finalize an IncrementalBackupInfo object after all manifest data has
+ * been supplied via calls to AppendIncrementalManifestData.
+ */
+void
+FinalizeIncrementalManifest(IncrementalBackupInfo *ib)
+{
+ JsonManifestParseContext context;
+ MemoryContext oldcontext;
+
+ /* Switch to our memory context. */
+ oldcontext = MemoryContextSwitchTo(ib->mcxt);
+
+ /* Parse the manifest. */
+ context.private_data = ib;
+ context.perfile_cb = manifest_process_file;
+ context.perwalrange_cb = manifest_process_wal_range;
+ context.error_cb = manifest_report_error;
+ json_parse_manifest(&context, ib->buf.data, ib->buf.len);
+
+ /* Done with the buffer, so release memory. */
+ pfree(ib->buf.data);
+ ib->buf.data = NULL;
+
+ /* Switch back to previous memory context. */
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Prepare to take an incremental backup.
+ *
+ * Before this function is called, AppendIncrementalManifestData and
+ * FinalizeIncrementalManifest should have already been called to pass all
+ * the manifest data to this object.
+ *
+ * This function performs sanity checks on the data extracted from the
+ * manifest and figures out for which WAL ranges we need summaries, and
+ * whether those summaries are available. Then, it reads and combines the
+ * data from those summary files. It also updates the backup_state with the
+ * reference TLI and LSN for the prior backup.
+ */
+void
+PrepareForIncrementalBackup(IncrementalBackupInfo *ib,
+ BackupState *backup_state)
+{
+ MemoryContext oldcontext;
+ List *expectedTLEs;
+ List *all_wslist,
+ *required_wslist = NIL;
+ ListCell *lc;
+ TimeLineHistoryEntry **tlep;
+ int num_wal_ranges;
+ int i;
+ bool found_backup_start_tli = false;
+ TimeLineID earliest_wal_range_tli = 0;
+ XLogRecPtr earliest_wal_range_start_lsn = InvalidXLogRecPtr;
+ TimeLineID latest_wal_range_tli = 0;
+ XLogRecPtr summarized_lsn;
+
+ Assert(ib->buf.data == NULL);
+
+ /* Switch to our memory context. */
+ oldcontext = MemoryContextSwitchTo(ib->mcxt);
+
+ /*
+ * A valid backup manifest must always contain at least one WAL range
+ * (usually exactly one, unless the backup spanned a timeline switch).
+ */
+ num_wal_ranges = list_length(ib->manifest_wal_ranges);
+ if (num_wal_ranges == 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("manifest contains no required WAL ranges")));
+
+ /*
+ * Match up the TLIs that appear in the WAL ranges of the backup manifest
+ * with those that appear in this server's timeline history. We expect
+ * every backup_wal_range to match to a TimeLineHistoryEntry; if it does
+ * not, that's an error.
+ *
+ * This loop also decides which of the WAL ranges is the manifest is most
+ * ancient and which one is the newest, according to the timeline history
+ * of this server, and stores TLIs of those WAL ranges into
+ * earliest_wal_range_tli and latest_wal_range_tli. It also updates
+ * earliest_wal_range_start_lsn to the start LSN of the WAL range for
+ * earliest_wal_range_tli.
+ *
+ * Note that the return value of readTimeLineHistory puts the latest
+ * timeline at the beginning of the list, not the end. Hence, the earliest
+ * TLI is the one that occurs nearest the end of the list returned by
+ * readTimeLineHistory, and the latest TLI is the one that occurs closest
+ * to the beginning.
+ */
+ expectedTLEs = readTimeLineHistory(backup_state->starttli);
+ tlep = palloc0(num_wal_ranges * sizeof(TimeLineHistoryEntry *));
+ for (i = 0; i < num_wal_ranges; ++i)
+ {
+ backup_wal_range *range = list_nth(ib->manifest_wal_ranges, i);
+ bool saw_earliest_wal_range_tli = false;
+ bool saw_latest_wal_range_tli = false;
+
+ /* Search this server's history for this WAL range's TLI. */
+ foreach(lc, expectedTLEs)
+ {
+ TimeLineHistoryEntry *tle = lfirst(lc);
+
+ if (tle->tli == range->tli)
+ {
+ tlep[i] = tle;
+ break;
+ }
+
+ if (tle->tli == earliest_wal_range_tli)
+ saw_earliest_wal_range_tli = true;
+ if (tle->tli == latest_wal_range_tli)
+ saw_latest_wal_range_tli = true;
+ }
+
+ /*
+ * An incremental backup can only be taken relative to a backup that
+ * represents a previous state of this server. If the backup requires
+ * WAL from a timeline that's not in our history, that definitely
+ * isn't the case.
+ */
+ if (tlep[i] == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("timeline %u found in manifest, but not in this server's history",
+ range->tli)));
+
+ /*
+ * If we found this TLI in the server's history before encountering
+ * the latest TLI seen so far in the server's history, then this TLI
+ * is the latest one seen so far.
+ *
+ * If on the other hand we saw the earliest TLI seen so far before
+ * finding this TLI, this TLI is earlier than the earliest one seen so
+ * far. And if this is the first TLI for which we've searched, it's
+ * also the earliest one seen so far.
+ *
+ * On the first loop iteration, both things should necessarily be
+ * true.
+ */
+ if (!saw_latest_wal_range_tli)
+ latest_wal_range_tli = range->tli;
+ if (earliest_wal_range_tli == 0 || saw_earliest_wal_range_tli)
+ {
+ earliest_wal_range_tli = range->tli;
+ earliest_wal_range_start_lsn = range->start_lsn;
+ }
+ }
+
+ /*
+ * Propagate information about the prior backup into the backup_label that
+ * will be generated for this backup.
+ */
+ backup_state->istartpoint = earliest_wal_range_start_lsn;
+ backup_state->istarttli = earliest_wal_range_tli;
+
+ /*
+ * Sanity check start and end LSNs for the WAL ranges in the manifest.
+ *
+ * Commonly, there won't be any timeline switches during the prior backup
+ * at all, but if there are, they should happen at the same LSNs that this
+ * server switched timelines.
+ *
+ * Whether there are any timeline switches during the prior backup or not,
+ * the prior backup shouldn't require any WAL from a timeline prior to the
+ * start of that timeline. It also shouldn't require any WAL from later
+ * than the start of this backup.
+ *
+ * If any of these sanity checks fail, one possible explanation is that
+ * the user has generated WAL on the same timeline with the same LSNs more
+ * than once. For instance, if two standbys running on timeline 1 were
+ * both promoted and (due to a broken archiving setup) both selected new
+ * timeline ID 2, then it's possible that one of these checks might trip.
+ *
+ * Note that there are lots of ways for the user to do something very bad
+ * without tripping any of these checks, and they are not intended to be
+ * comprehensive. It's pretty hard to see how we could be certain of
+ * anything here. However, if there's a problem staring us right in the
+ * face, it's best to report it, so we do.
+ */
+ for (i = 0; i < num_wal_ranges; ++i)
+ {
+ backup_wal_range *range = list_nth(ib->manifest_wal_ranges, i);
+
+ if (range->tli == earliest_wal_range_tli)
+ {
+ if (range->start_lsn < tlep[i]->begin)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("manifest requires WAL from initial timeline %u starting at %X/%X, but that timeline begins at %X/%X",
+ range->tli,
+ LSN_FORMAT_ARGS(range->start_lsn),
+ LSN_FORMAT_ARGS(tlep[i]->begin))));
+ }
+ else
+ {
+ if (range->start_lsn != tlep[i]->begin)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("manifest requires WAL from continuation timeline %u starting at %X/%X, but that timeline begins at %X/%X",
+ range->tli,
+ LSN_FORMAT_ARGS(range->start_lsn),
+ LSN_FORMAT_ARGS(tlep[i]->begin))));
+ }
+
+ if (range->tli == latest_wal_range_tli)
+ {
+ if (range->end_lsn > backup_state->startpoint)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("manifest requires WAL from final timeline %u ending at %X/%X, but this backup starts at %X/%X",
+ range->tli,
+ LSN_FORMAT_ARGS(range->end_lsn),
+ LSN_FORMAT_ARGS(backup_state->startpoint))));
+ }
+ else
+ {
+ if (range->end_lsn != tlep[i]->end)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("manifest requires WAL from non-final timeline %u ending at %X/%X, but this server switched timelines at %X/%X",
+ range->tli,
+ LSN_FORMAT_ARGS(range->end_lsn),
+ LSN_FORMAT_ARGS(tlep[i]->end))));
+ }
+
+ }
+
+ /*
+ * Wait for WAL summarization to catch up to the backup start LSN (but
+ * time out if it doesn't do so quickly enough).
+ */
+ /* XXX make timeout configurable */
+ summarized_lsn = WaitForWalSummarization(backup_state->startpoint, 60000);
+ if (summarized_lsn < backup_state->startpoint)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("timeout waiting for WAL summarization"),
+ errdetail("This backup requires WAL to be summarized up to %X/%X, but summarizer has only reached %X/%X.",
+ LSN_FORMAT_ARGS(backup_state->startpoint),
+ LSN_FORMAT_ARGS(summarized_lsn))));
+
+ /*
+ * Retrieve a list of all WAL summaries on any timeline that overlap with
+ * the LSN range of interest. We could instead call GetWalSummaries() once
+ * per timeline in the loop that follows, but that would involve reading
+ * the directory multiple times. It should be mildly faster - and perhaps
+ * a bit safer - to do it just once.
+ */
+ all_wslist = GetWalSummaries(0, earliest_wal_range_start_lsn,
+ backup_state->startpoint);
+
+ /*
+ * We need WAL summaries for everything that happened during the prior
+ * backup and everything that happened afterward up until the point where
+ * the current backup started.
+ */
+ foreach(lc, expectedTLEs)
+ {
+ TimeLineHistoryEntry *tle = lfirst(lc);
+ XLogRecPtr tli_start_lsn = tle->begin;
+ XLogRecPtr tli_end_lsn = tle->end;
+ XLogRecPtr tli_missing_lsn = InvalidXLogRecPtr;
+ List *tli_wslist;
+
+ /*
+ * Working through the history of this server from the current
+ * timeline backwards, we skip everything until we find the timeline
+ * where this backup started. Most of the time, this means we won't
+ * skip anything at all, as it's unlikely that the timeline has
+ * changed since the beginning of the backup moments ago.
+ */
+ if (tle->tli == backup_state->starttli)
+ {
+ found_backup_start_tli = true;
+ tli_end_lsn = backup_state->startpoint;
+ }
+ else if (!found_backup_start_tli)
+ continue;
+
+ /*
+ * Find the summaries that overlap the LSN range of interest for this
+ * timeline. If this is the earliest timeline involved, the range of
+ * interest begins with the start LSN of the prior backup; otherwise,
+ * it begins at the LSN at which this timeline came into existence. If
+ * this is the latest TLI involved, the range of interest ends at the
+ * start LSN of the current backup; otherwise, it ends at the point
+ * where we switched from this timeline to the next one.
+ */
+ if (tle->tli == earliest_wal_range_tli)
+ tli_start_lsn = earliest_wal_range_start_lsn;
+ tli_wslist = FilterWalSummaries(all_wslist, tle->tli,
+ tli_start_lsn, tli_end_lsn);
+
+ /*
+ * There is no guarantee that the WAL summaries we found cover the
+ * entire range of LSNs for which summaries are required, or indeed
+ * that we found any WAL summaries at all. Check whether we have a
+ * problem of that sort.
+ */
+ if (!WalSummariesAreComplete(tli_wslist, tli_start_lsn, tli_end_lsn,
+ &tli_missing_lsn))
+ {
+ if (XLogRecPtrIsInvalid(tli_missing_lsn))
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("WAL summaries are required on timeline %u from %X/%X to %X/%X, but no summaries for that timeline and LSN range exist",
+ tle->tli,
+ LSN_FORMAT_ARGS(tli_start_lsn),
+ LSN_FORMAT_ARGS(tli_end_lsn))));
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("WAL summaries are required on timeline %u from %X/%X to %X/%X, but the summaries for that timeline and LSN range are incomplete",
+ tle->tli,
+ LSN_FORMAT_ARGS(tli_start_lsn),
+ LSN_FORMAT_ARGS(tli_end_lsn)),
+ errdetail("The first unsummarized LSN is this range is %X/%X.",
+ LSN_FORMAT_ARGS(tli_missing_lsn))));
+ }
+
+ /*
+ * Remember that we need to read these summaries.
+ *
+ * Technically, it's possible that this could read more files than
+ * required, since tli_wslist in theory could contain redundant
+ * summaries. For instance, if we have a summary from 0/10000000 to
+ * 0/20000000 and also one from 0/00000000 to 0/30000000, then the
+ * latter subsumes the former and the former could be ignored.
+ *
+ * We ignore this possibility because the WAL summarizer only tries to
+ * generate summaries that do not overlap. If somehow they exist,
+ * we'll do a bit of extra work but the results should still be
+ * correct.
+ */
+ required_wslist = list_concat(required_wslist, tli_wslist);
+
+ /*
+ * Timelines earlier than the one in which the prior backup began are
+ * not relevant.
+ */
+ if (tle->tli == earliest_wal_range_tli)
+ break;
+ }
+
+ /*
+ * Read all of the required block reference table files and merge all of
+ * the data into a single in-memory block reference table.
+ *
+ * See the comments for struct IncrementalBackupInfo for some thoughts on
+ * memory usage.
+ */
+ ib->brtab = CreateEmptyBlockRefTable();
+ foreach(lc, required_wslist)
+ {
+ WalSummaryFile *ws = lfirst(lc);
+ WalSummaryIO wsio;
+ BlockRefTableReader *reader;
+ RelFileLocator rlocator;
+ ForkNumber forknum;
+ BlockNumber limit_block;
+ BlockNumber blocks[BLOCKS_PER_READ];
+
+ wsio.file = OpenWalSummaryFile(ws, false);
+ wsio.filepos = 0;
+ ereport(DEBUG1,
+ (errmsg_internal("reading WAL summary file \"%s\"",
+ FilePathName(wsio.file))));
+ reader = CreateBlockRefTableReader(ReadWalSummary, &wsio,
+ FilePathName(wsio.file),
+ ReportWalSummaryError, NULL);
+ while (BlockRefTableReaderNextRelation(reader, &rlocator, &forknum,
+ &limit_block))
+ {
+ BlockRefTableSetLimitBlock(ib->brtab, &rlocator,
+ forknum, limit_block);
+
+ while (1)
+ {
+ unsigned nblocks;
+ unsigned i;
+
+ nblocks = BlockRefTableReaderGetBlocks(reader, blocks,
+ BLOCKS_PER_READ);
+ if (nblocks == 0)
+ break;
+
+ for (i = 0; i < nblocks; ++i)
+ BlockRefTableMarkBlockModified(ib->brtab, &rlocator,
+ forknum, blocks[i]);
+ }
+ }
+ DestroyBlockRefTableReader(reader);
+ FileClose(wsio.file);
+ }
+
+ /* Switch back to previous memory context. */
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Get the pathname that should be used when a file is sent incrementally.
+ *
+ * The result is a palloc'd string.
+ */
+char *
+GetIncrementalFilePath(Oid dboid, Oid spcoid, RelFileNumber relfilenumber,
+ ForkNumber forknum, unsigned segno)
+{
+ char *path;
+ char *lastslash;
+ char *ipath;
+
+ path = GetRelationPath(dboid, spcoid, relfilenumber, InvalidBackendId,
+ forknum);
+
+ lastslash = strrchr(path, '/');
+ Assert(lastslash != NULL);
+ *lastslash = '\0';
+
+ if (segno > 0)
+ ipath = psprintf("%s/INCREMENTAL.%s.%u", path, lastslash + 1, segno);
+ else
+ ipath = psprintf("%s/INCREMENTAL.%s", path, lastslash + 1);
+
+ pfree(path);
+
+ return ipath;
+}
+
+/*
+ * How should we back up a particular file as part of an incremental backup?
+ *
+ * If the return value is BACK_UP_FILE_FULLY, caller should back up the whole
+ * file just as if this were not an incremental backup.
+ *
+ * If the return value is BACK_UP_FILE_INCREMENTALLY, caller should include
+ * an incremental file in the backup instead of the entire file. On return,
+ * *num_blocks_required will be set to the number of blocks that need to be
+ * sent, and the actual block numbers will have been stored in
+ * relative_block_numbers, which should be an array of at least RELSEG_SIZE.
+ * In addition, *truncation_block_length will be set to the value that should
+ * be included in the incremental file.
+ *
+ * If the return value is DO_NOT_BACK_UP_FILE, the caller should not include
+ * the file in the backup at all.
+ */
+FileBackupMethod
+GetFileBackupMethod(IncrementalBackupInfo *ib, char *path,
+ Oid dboid, Oid spcoid,
+ RelFileNumber relfilenumber, ForkNumber forknum,
+ unsigned segno, size_t size,
+ unsigned *num_blocks_required,
+ BlockNumber *relative_block_numbers,
+ unsigned *truncation_block_length)
+{
+ BlockNumber absolute_block_numbers[RELSEG_SIZE];
+ BlockNumber limit_block;
+ BlockNumber start_blkno;
+ BlockNumber stop_blkno;
+ RelFileLocator rlocator;
+ BlockRefTableEntry *brtentry;
+ unsigned i;
+ unsigned nblocks;
+
+ /* Should only be called after PrepareForIncrementalBackup. */
+ Assert(ib->buf.data == NULL);
+
+ /*
+ * dboid could be InvalidOid if shared rel, but spcoid and relfilenumber
+ * should have legal values.
+ */
+ Assert(OidIsValid(spcoid));
+ Assert(RelFileNumberIsValid(relfilenumber));
+
+ /*
+ * If the file size is too large or not a multiple of BLCKSZ, then
+ * something weird is happening, so give up and send the whole file.
+ */
+ if ((size % BLCKSZ) != 0 || size / BLCKSZ > RELSEG_SIZE)
+ return BACK_UP_FILE_FULLY;
+
+ /*
+ * The free-space map fork is not properly WAL-logged, so we need to
+ * backup the entire file every time.
+ */
+ if (forknum == FSM_FORKNUM)
+ return BACK_UP_FILE_FULLY;
+
+ /*
+ * Check whether this file is part of the prior backup. If it isn't, back
+ * up the whole file.
+ */
+ if (backup_file_lookup(ib->manifest_files, path) == NULL)
+ {
+ char *ipath;
+
+ ipath = GetIncrementalFilePath(dboid, spcoid, relfilenumber,
+ forknum, segno);
+ if (backup_file_lookup(ib->manifest_files, ipath) == NULL)
+ return BACK_UP_FILE_FULLY;
+ }
+
+ /* Look up the block reference table entry. */
+ rlocator.spcOid = spcoid;
+ rlocator.dbOid = dboid;
+ rlocator.relNumber = relfilenumber;
+ brtentry = BlockRefTableGetEntry(ib->brtab, &rlocator, forknum,
+ &limit_block);
+
+ /*
+ * If there is no entry, then there have been no WAL-logged changes to the
+ * relation since the predecessor backup was taken, so we can back it up
+ * incrementally and need not include any modified blocks.
+ *
+ * However, if the file is zero-length, we should do a full backup,
+ * because an incremental file is always more than zero length, and it's
+ * silly to take an incremental backup when a full backup would be
+ * smaller.
+ */
+ if (brtentry == NULL)
+ {
+ *num_blocks_required = 0;
+ *truncation_block_length = size / BLCKSZ;
+ if (size == 0)
+ return BACK_UP_FILE_FULLY;
+ return BACK_UP_FILE_INCREMENTALLY;
+ }
+
+ /*
+ * If the limit_block is less than or equal to the point where this
+ * segment starts, send the whole file.
+ */
+ if (limit_block <= segno * RELSEG_SIZE)
+ return BACK_UP_FILE_FULLY;
+
+ /*
+ * Get relevant entries from the block reference table entry.
+ *
+ * We shouldn't overflow computing the start or stop block numbers, but if
+ * it manages to happen somehow, detect it and throw an error.
+ */
+ start_blkno = segno * RELSEG_SIZE;
+ stop_blkno = start_blkno + (size / BLCKSZ);
+ if (start_blkno / RELSEG_SIZE != segno || stop_blkno < start_blkno)
+ ereport(ERROR,
+ errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg_internal("overflow computing block number bounds for segment %u with size %lu",
+ segno, size));
+ nblocks = BlockRefTableEntryGetBlocks(brtentry, start_blkno, stop_blkno,
+ absolute_block_numbers, RELSEG_SIZE);
+ Assert(nblocks <= RELSEG_SIZE);
+
+ /*
+ * Sort the returned block numbers. If the block reference table was using
+ * the bitmap representation for a given chunk, the block numbers in that
+ * chunk will already be sorted, but when the array-of-offsets
+ * representation is used, we can receive block numbers here out of order.
+ */
+ qsort(absolute_block_numbers, nblocks, sizeof(BlockNumber),
+ compare_block_numbers);
+
+ /*
+ * If we're going to have to send nearly all of the blocks, then just send
+ * the whole file, because that won't require much extra storage or
+ * transfer and will speed up and simplify backup restoration. It's not
+ * clear what threshold is most appropriate here and perhaps it ought to
+ * be configurable, but for now we're just going to say that if we'd need
+ * to send 90% of the blocks anyway, give up and send the whole file.
+ *
+ * NB: If you change the threshold here, at least make sure to back up the
+ * file fully when every single block must be sent, because there's
+ * nothing good about sending an incremental file in that case.
+ */
+ if (nblocks * BLCKSZ > size * 0.9)
+ return BACK_UP_FILE_FULLY;
+
+ /*
+ * Looks like we can send an incremental file so transpose absolute block
+ * numbers to relative block numbers.
+ */
+ for (i = 0; i < nblocks; ++i)
+ relative_block_numbers[i] = absolute_block_numbers[i] - start_blkno;
+ *num_blocks_required = nblocks;
+
+ /*
+ * The truncation block length is the minimum length of the reconstructed
+ * file. Any block numbers below this threshold that are not present in
+ * the backup need to be fetched from the prior backup. At or above this
+ * threshold, blocks should only be included in the result if they are
+ * present in the backup. (This may require inserting zero blocks if the
+ * blocks included in the backup are non-consecutive.)
+ */
+ *truncation_block_length = size / BLCKSZ;
+ if (BlockNumberIsValid(limit_block))
+ {
+ unsigned relative_limit = limit_block - segno * RELSEG_SIZE;
+
+ if (*truncation_block_length < relative_limit)
+ *truncation_block_length = relative_limit;
+ }
+
+ /* Send it incrementally. */
+ return BACK_UP_FILE_INCREMENTALLY;
+}
+
+/*
+ * Compute the size for an incremental file containing a given number of blocks.
+ */
+extern size_t
+GetIncrementalFileSize(unsigned num_blocks_required)
+{
+ size_t result;
+
+ /* Make sure we're not going to overflow. */
+ Assert(num_blocks_required <= RELSEG_SIZE);
+
+ /*
+ * Three four byte quantities (magic number, truncation block length,
+ * block count) followed by block numbers followed by block contents.
+ */
+ result = 3 * sizeof(uint32);
+ result += (BLCKSZ + sizeof(BlockNumber)) * num_blocks_required;
+
+ return result;
+}
+
+/*
+ * Helper function for filemap hash table.
+ */
+static uint32
+hash_string_pointer(const char *s)
+{
+ unsigned char *ss = (unsigned char *) s;
+
+ return hash_bytes(ss, strlen(s));
+}
+
+/*
+ * This callback is invoked for each file mentioned in the backup manifest.
+ *
+ * We store the path to each file and the size of each file for sanity-checking
+ * purposes. For further details, see comments for IncrementalBackupInfo.
+ */
+static void
+manifest_process_file(JsonManifestParseContext *context,
+ char *pathname, size_t size,
+ pg_checksum_type checksum_type,
+ int checksum_length,
+ uint8 *checksum_payload)
+{
+ IncrementalBackupInfo *ib = context->private_data;
+ backup_file_entry *entry;
+ bool found;
+
+ entry = backup_file_insert(ib->manifest_files, pathname, &found);
+ if (!found)
+ {
+ entry->path = MemoryContextStrdup(ib->manifest_files->ctx,
+ pathname);
+ entry->size = size;
+ }
+}
+
+/*
+ * This callback is invoked for each WAL range mentioned in the backup
+ * manifest.
+ *
+ * We're just interested in learning the oldest LSN and the corresponding TLI
+ * that appear in any WAL range.
+ */
+static void
+manifest_process_wal_range(JsonManifestParseContext *context,
+ TimeLineID tli, XLogRecPtr start_lsn,
+ XLogRecPtr end_lsn)
+{
+ IncrementalBackupInfo *ib = context->private_data;
+ backup_wal_range *range = palloc(sizeof(backup_wal_range));
+
+ range->tli = tli;
+ range->start_lsn = start_lsn;
+ range->end_lsn = end_lsn;
+ ib->manifest_wal_ranges = lappend(ib->manifest_wal_ranges, range);
+}
+
+/*
+ * This callback is invoked if an error occurs while parsing the backup
+ * manifest.
+ */
+static void
+manifest_report_error(JsonManifestParseContext *context, const char *fmt,...)
+{
+ StringInfoData errbuf;
+
+ initStringInfo(&errbuf);
+
+ for (;;)
+ {
+ va_list ap;
+ int needed;
+
+ va_start(ap, fmt);
+ needed = appendStringInfoVA(&errbuf, fmt, ap);
+ va_end(ap);
+ if (needed == 0)
+ break;
+ enlargeStringInfo(&errbuf, needed);
+ }
+
+ ereport(ERROR,
+ errmsg_internal("%s", errbuf.data));
+}
+
+/*
+ * Quicksort comparator for block numbers.
+ */
+static int
+compare_block_numbers(const void *a, const void *b)
+{
+ BlockNumber aa = *(BlockNumber *) a;
+ BlockNumber bb = *(BlockNumber *) b;
+
+ if (aa > bb)
+ return 1;
+ else if (aa == bb)
+ return 0;
+ else
+ return -1;
+}
diff --git a/src/backend/backup/meson.build b/src/backend/backup/meson.build
index 0e2de91e9f..19c355ceca 100644
--- a/src/backend/backup/meson.build
+++ b/src/backend/backup/meson.build
@@ -5,6 +5,7 @@ backend_sources += files(
'basebackup.c',
'basebackup_copy.c',
'basebackup_gzip.c',
+ 'basebackup_incremental.c',
'basebackup_lz4.c',
'basebackup_progress.c',
'basebackup_server.c',
diff --git a/src/backend/replication/repl_gram.y b/src/backend/replication/repl_gram.y
index 0c874e33cf..a5d118ed68 100644
--- a/src/backend/replication/repl_gram.y
+++ b/src/backend/replication/repl_gram.y
@@ -76,11 +76,12 @@ Node *replication_parse_result;
%token K_EXPORT_SNAPSHOT
%token K_NOEXPORT_SNAPSHOT
%token K_USE_SNAPSHOT
+%token K_UPLOAD_MANIFEST
%type <node> command
%type <node> base_backup start_replication start_logical_replication
create_replication_slot drop_replication_slot identify_system
- read_replication_slot timeline_history show
+ read_replication_slot timeline_history show upload_manifest
%type <list> generic_option_list
%type <defelt> generic_option
%type <uintval> opt_timeline
@@ -114,6 +115,7 @@ command:
| read_replication_slot
| timeline_history
| show
+ | upload_manifest
;
/*
@@ -307,6 +309,15 @@ timeline_history:
}
;
+/* UPLOAD_MANIFEST doesn't currently accept any arguments */
+upload_manifest:
+ K_UPLOAD_MANIFEST
+ {
+ UploadManifestCmd *cmd = makeNode(UploadManifestCmd);
+
+ $$ = (Node *) cmd;
+ }
+
opt_physical:
K_PHYSICAL
| /* EMPTY */
@@ -411,6 +422,7 @@ ident_or_keyword:
| K_EXPORT_SNAPSHOT { $$ = "export_snapshot"; }
| K_NOEXPORT_SNAPSHOT { $$ = "noexport_snapshot"; }
| K_USE_SNAPSHOT { $$ = "use_snapshot"; }
+ | K_UPLOAD_MANIFEST { $$ = "upload_manifest"; }
;
%%
diff --git a/src/backend/replication/repl_scanner.l b/src/backend/replication/repl_scanner.l
index 1cc7fb858c..4805da08ee 100644
--- a/src/backend/replication/repl_scanner.l
+++ b/src/backend/replication/repl_scanner.l
@@ -136,6 +136,7 @@ EXPORT_SNAPSHOT { return K_EXPORT_SNAPSHOT; }
NOEXPORT_SNAPSHOT { return K_NOEXPORT_SNAPSHOT; }
USE_SNAPSHOT { return K_USE_SNAPSHOT; }
WAIT { return K_WAIT; }
+UPLOAD_MANIFEST { return K_UPLOAD_MANIFEST; }
{space}+ { /* do nothing */ }
@@ -303,6 +304,7 @@ replication_scanner_is_replication_command(void)
case K_DROP_REPLICATION_SLOT:
case K_READ_REPLICATION_SLOT:
case K_TIMELINE_HISTORY:
+ case K_UPLOAD_MANIFEST:
case K_SHOW:
/* Yes; push back the first token so we can parse later. */
repl_pushed_back_token = first_token;
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index e250b0567e..b33b86671b 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -58,6 +58,7 @@
#include "access/xlogrecovery.h"
#include "access/xlogutils.h"
#include "backup/basebackup.h"
+#include "backup/basebackup_incremental.h"
#include "catalog/pg_authid.h"
#include "catalog/pg_type.h"
#include "commands/dbcommands.h"
@@ -137,6 +138,17 @@ bool wake_wal_senders = false;
*/
static XLogReaderState *xlogreader = NULL;
+/*
+ * If the UPLOAD_MANIFEST command is used to provide a backup manifest in
+ * preparation for an incremental backup, uploaded_manifest will be point
+ * to an object containing information about its contexts, and
+ * uploaded_manifest_mcxt will point to the memory context that contains
+ * that object and all of its subordinate data. Otherwise, both values will
+ * be NULL.
+ */
+static IncrementalBackupInfo *uploaded_manifest = NULL;
+static MemoryContext uploaded_manifest_mcxt = NULL;
+
/*
* These variables keep track of the state of the timeline we're currently
* sending. sendTimeLine identifies the timeline. If sendTimeLineIsHistoric,
@@ -233,6 +245,9 @@ static void XLogSendLogical(void);
static void WalSndDone(WalSndSendDataCallback send_data);
static XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli);
static void IdentifySystem(void);
+static void UploadManifest(void);
+static bool HandleUploadManifestPacket(StringInfo buf, off_t *offset,
+ IncrementalBackupInfo *ib);
static void ReadReplicationSlot(ReadReplicationSlotCmd *cmd);
static void CreateReplicationSlot(CreateReplicationSlotCmd *cmd);
static void DropReplicationSlot(DropReplicationSlotCmd *cmd);
@@ -660,6 +675,143 @@ SendTimeLineHistory(TimeLineHistoryCmd *cmd)
pq_endmessage(&buf);
}
+/*
+ * Handle UPLOAD_MANIFEST command.
+ */
+static void
+UploadManifest(void)
+{
+ MemoryContext mcxt;
+ IncrementalBackupInfo *ib;
+ off_t offset = 0;
+ StringInfoData buf;
+
+ /*
+ * parsing the manifest will use the cryptohash stuff, which requires a
+ * resource owner
+ */
+ Assert(CurrentResourceOwner == NULL);
+ CurrentResourceOwner = ResourceOwnerCreate(NULL, "base backup");
+
+ /* Prepare to read manifest data into a temporary context. */
+ mcxt = AllocSetContextCreate(CurrentMemoryContext,
+ "incremental backup information",
+ ALLOCSET_DEFAULT_SIZES);
+ ib = CreateIncrementalBackupInfo(mcxt);
+
+ /* Send a CopyInResponse message */
+ pq_beginmessage(&buf, 'G');
+ pq_sendbyte(&buf, 0);
+ pq_sendint16(&buf, 0);
+ pq_endmessage_reuse(&buf);
+ pq_flush();
+
+ /* Recieve packets from client until done. */
+ while (HandleUploadManifestPacket(&buf, &offset, ib))
+ ;
+
+ /* Finish up manifest processing. */
+ FinalizeIncrementalManifest(ib);
+
+ /*
+ * Discard any old manifest information and arrange to preserve the new
+ * information we just got.
+ *
+ * We assume that MemoryContextDelete and MemoryContextSetParent won't
+ * fail, and thus we shouldn't end up bailing out of here in such a way as
+ * to leave dangling pointrs.
+ */
+ if (uploaded_manifest_mcxt != NULL)
+ MemoryContextDelete(uploaded_manifest_mcxt);
+ MemoryContextSetParent(mcxt, CacheMemoryContext);
+ uploaded_manifest = ib;
+ uploaded_manifest_mcxt = mcxt;
+
+ /* clean up the resource owner we created */
+ WalSndResourceCleanup(true);
+}
+
+/*
+ * Process one packet received during the handling of an UPLOAD_MANIFEST
+ * operation.
+ *
+ * 'buf' is scratch space. This function expects it to be initialized, doesn't
+ * care what the current contents are, and may override them with completely
+ * new contents.
+ *
+ * The return value is true if the caller should continue processing
+ * additional packets and false if the UPLOAD_MANIFEST operation is complete.
+ */
+static bool
+HandleUploadManifestPacket(StringInfo buf, off_t *offset,
+ IncrementalBackupInfo *ib)
+{
+ int mtype;
+ int maxmsglen;
+
+ HOLD_CANCEL_INTERRUPTS();
+
+ pq_startmsgread();
+ mtype = pq_getbyte();
+ if (mtype == EOF)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("unexpected EOF on client connection with an open transaction")));
+
+ switch (mtype)
+ {
+ case 'd': /* CopyData */
+ maxmsglen = PQ_LARGE_MESSAGE_LIMIT;
+ break;
+ case 'c': /* CopyDone */
+ case 'f': /* CopyFail */
+ case 'H': /* Flush */
+ case 'S': /* Sync */
+ maxmsglen = PQ_SMALL_MESSAGE_LIMIT;
+ break;
+ default:
+ ereport(ERROR,
+ (errcode(ERRCODE_PROTOCOL_VIOLATION),
+ errmsg("unexpected message type 0x%02X during COPY from stdin",
+ mtype)));
+ maxmsglen = 0; /* keep compiler quiet */
+ break;
+ }
+
+ /* Now collect the message body */
+ if (pq_getmessage(buf, maxmsglen))
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("unexpected EOF on client connection with an open transaction")));
+ RESUME_CANCEL_INTERRUPTS();
+
+ /* Process the message */
+ switch (mtype)
+ {
+ case 'd': /* CopyData */
+ AppendIncrementalManifestData(ib, buf->data, buf->len);
+ return true;
+
+ case 'c': /* CopyDone */
+ return false;
+
+ case 'H': /* Sync */
+ case 'S': /* Flush */
+ /* Ignore these while in CopyOut mode as we do elsewhere. */
+ return true;
+
+ case 'f':
+ ereport(ERROR,
+ (errcode(ERRCODE_QUERY_CANCELED),
+ errmsg("COPY from stdin failed: %s",
+ pq_getmsgstring(buf))));
+ }
+
+ /* Not reached. */
+ Assert(false);
+ return false;
+}
+
/*
* Handle START_REPLICATION command.
*
@@ -1801,7 +1953,7 @@ exec_replication_command(const char *cmd_string)
cmdtag = "BASE_BACKUP";
set_ps_display(cmdtag);
PreventInTransactionBlock(true, cmdtag);
- SendBaseBackup((BaseBackupCmd *) cmd_node);
+ SendBaseBackup((BaseBackupCmd *) cmd_node, uploaded_manifest);
EndReplicationCommand(cmdtag);
break;
@@ -1863,6 +2015,14 @@ exec_replication_command(const char *cmd_string)
}
break;
+ case T_UploadManifestCmd:
+ cmdtag = "UPLOAD_MANIFEST";
+ set_ps_display(cmdtag);
+ PreventInTransactionBlock(true, cmdtag);
+ UploadManifest();
+ EndReplicationCommand(cmdtag);
+ break;
+
default:
elog(ERROR, "unrecognized replication command node tag: %u",
cmd_node->type);
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index a3d8eacb8d..3a6729003a 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -31,6 +31,7 @@
#include "postmaster/bgworker_internals.h"
#include "postmaster/bgwriter.h"
#include "postmaster/postmaster.h"
+#include "postmaster/walsummarizer.h"
#include "replication/logicallauncher.h"
#include "replication/origin.h"
#include "replication/slot.h"
@@ -136,6 +137,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, ReplicationOriginShmemSize());
size = add_size(size, WalSndShmemSize());
size = add_size(size, WalRcvShmemSize());
+ size = add_size(size, WalSummarizerShmemSize());
size = add_size(size, PgArchShmemSize());
size = add_size(size, ApplyLauncherShmemSize());
size = add_size(size, BTreeShmemSize());
@@ -291,6 +293,7 @@ CreateSharedMemoryAndSemaphores(void)
ReplicationOriginShmemInit();
WalSndShmemInit();
WalRcvShmemInit();
+ WalSummarizerShmemInit();
PgArchShmemInit();
ApplyLauncherShmemInit();
diff --git a/src/bin/Makefile b/src/bin/Makefile
index 373077bf52..aa2210925e 100644
--- a/src/bin/Makefile
+++ b/src/bin/Makefile
@@ -19,6 +19,7 @@ SUBDIRS = \
pg_archivecleanup \
pg_basebackup \
pg_checksums \
+ pg_combinebackup \
pg_config \
pg_controldata \
pg_ctl \
diff --git a/src/bin/meson.build b/src/bin/meson.build
index 67cb50630c..4cb6fd59bb 100644
--- a/src/bin/meson.build
+++ b/src/bin/meson.build
@@ -5,6 +5,7 @@ subdir('pg_amcheck')
subdir('pg_archivecleanup')
subdir('pg_basebackup')
subdir('pg_checksums')
+subdir('pg_combinebackup')
subdir('pg_config')
subdir('pg_controldata')
subdir('pg_ctl')
diff --git a/src/bin/pg_basebackup/bbstreamer_file.c b/src/bin/pg_basebackup/bbstreamer_file.c
index 45f32974ff..6b78ee283d 100644
--- a/src/bin/pg_basebackup/bbstreamer_file.c
+++ b/src/bin/pg_basebackup/bbstreamer_file.c
@@ -296,6 +296,7 @@ should_allow_existing_directory(const char *pathname)
if (strcmp(filename, "pg_wal") == 0 ||
strcmp(filename, "pg_xlog") == 0 ||
strcmp(filename, "archive_status") == 0 ||
+ strcmp(filename, "summaries") == 0 ||
strcmp(filename, "pg_tblspc") == 0)
return true;
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index f32684a8f2..26fd9ad0bc 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -101,6 +101,11 @@ typedef void (*WriteDataCallback) (size_t nbytes, char *buf,
*/
#define MINIMUM_VERSION_FOR_TERMINATED_TARFILE 150000
+/*
+ * pg_wal/summaries exists beginning with version 17.
+ */
+#define MINIMUM_VERSION_FOR_WAL_SUMMARIES 170000
+
/*
* Different ways to include WAL
*/
@@ -217,7 +222,8 @@ static void ReceiveBackupManifestInMemoryChunk(size_t r, char *copybuf,
void *callback_data);
static void BaseBackup(char *compression_algorithm, char *compression_detail,
CompressionLocation compressloc,
- pg_compress_specification *client_compress);
+ pg_compress_specification *client_compress,
+ char *incremental_manifest);
static bool reached_end_position(XLogRecPtr segendpos, uint32 timeline,
bool segment_finished);
@@ -390,6 +396,8 @@ usage(void)
printf(_("\nOptions controlling the output:\n"));
printf(_(" -D, --pgdata=DIRECTORY receive base backup into directory\n"));
printf(_(" -F, --format=p|t output format (plain (default), tar)\n"));
+ printf(_(" -i, --incremental=OLDMANIFEST\n"));
+ printf(_(" take incremental or differential backup\n"));
printf(_(" -r, --max-rate=RATE maximum transfer rate to transfer data directory\n"
" (in kB/s, or use suffix \"k\" or \"M\")\n"));
printf(_(" -R, --write-recovery-conf\n"
@@ -688,6 +696,23 @@ StartLogStreamer(char *startpos, uint32 timeline, char *sysidentifier,
if (pg_mkdir_p(statusdir, pg_dir_create_mode) != 0 && errno != EEXIST)
pg_fatal("could not create directory \"%s\": %m", statusdir);
+
+ /*
+ * For newer server versions, likewise create pg_wal/summaries
+ */
+ if (PQserverVersion(conn) < MINIMUM_VERSION_FOR_WAL_SUMMARIES)
+ {
+ char summarydir[MAXPGPATH];
+
+ snprintf(summarydir, sizeof(summarydir), "%s/%s/summaries",
+ basedir,
+ PQserverVersion(conn) < MINIMUM_VERSION_FOR_PG_WAL ?
+ "pg_xlog" : "pg_wal");
+
+ if (pg_mkdir_p(statusdir, pg_dir_create_mode) != 0 &&
+ errno != EEXIST)
+ pg_fatal("could not create directory \"%s\": %m", summarydir);
+ }
}
/*
@@ -1728,7 +1753,9 @@ ReceiveBackupManifestInMemoryChunk(size_t r, char *copybuf,
static void
BaseBackup(char *compression_algorithm, char *compression_detail,
- CompressionLocation compressloc, pg_compress_specification *client_compress)
+ CompressionLocation compressloc,
+ pg_compress_specification *client_compress,
+ char *incremental_manifest)
{
PGresult *res;
char *sysidentifier;
@@ -1794,7 +1821,74 @@ BaseBackup(char *compression_algorithm, char *compression_detail,
exit(1);
/*
- * Start the actual backup
+ * If the user wants an incremental backup, we must upload the manifest
+ * for the previous backup upon which it is to be based.
+ */
+ if (incremental_manifest != NULL)
+ {
+ int fd;
+ char mbuf[65536];
+ int nbytes;
+
+ /* XXX add a server version check here */
+
+ /* Open the file. */
+ fd = open(incremental_manifest, O_RDONLY | PG_BINARY, 0);
+ if (fd < 0)
+ pg_fatal("could not open file \"%s\": %m", incremental_manifest);
+
+ /* Tell the server what we want to do. */
+ if (PQsendQuery(conn, "UPLOAD_MANIFEST") == 0)
+ pg_fatal("could not send replication command \"%s\": %s",
+ "UPLOAD_MANIFEST", PQerrorMessage(conn));
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ if (PQresultStatus(res) == PGRES_FATAL_ERROR)
+ pg_fatal("could not upload manifest: %s",
+ PQerrorMessage(conn));
+ else
+ pg_fatal("could not upload manifest: unexpected status %s",
+ PQresStatus(PQresultStatus(res)));
+ }
+
+ /* Loop, reading from the file and sending the data to the server. */
+ while ((nbytes = read(fd, mbuf, sizeof mbuf)) > 0)
+ {
+ if (PQputCopyData(conn, mbuf, nbytes) < 0)
+ pg_fatal("could not send COPY data: %s",
+ PQerrorMessage(conn));
+ }
+
+ /* Bail out if we exited the loop due to an error. */
+ if (nbytes < 0)
+ pg_fatal("could not read file \"%s\": %m", incremental_manifest);
+
+ /* End the COPY operation. */
+ if (PQputCopyEnd(conn, NULL) < 0)
+ pg_fatal("could not send end-of-COPY: %s",
+ PQerrorMessage(conn));
+
+ /* See whether the server is happy with what we sent. */
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) == PGRES_FATAL_ERROR)
+ pg_fatal("could not upload manifest: %s",
+ PQerrorMessage(conn));
+ else if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pg_fatal("could not upload manifest: unexpected status %s",
+ PQresStatus(PQresultStatus(res)));
+
+ /* Consume ReadyForQuery message from server. */
+ res = PQgetResult(conn);
+ if (res != NULL)
+ pg_fatal("unexpected extra result while sending manifest");
+
+ /* Add INCREMENTAL option to BASE_BACKUP command. */
+ AppendPlainCommandOption(&buf, use_new_option_syntax, "INCREMENTAL");
+ }
+
+ /*
+ * Continue building up the options list for the BASE_BACKUP command.
*/
AppendStringCommandOption(&buf, use_new_option_syntax, "LABEL", label);
if (estimatesize)
@@ -1901,6 +1995,7 @@ BaseBackup(char *compression_algorithm, char *compression_detail,
else
basebkp = psprintf("BASE_BACKUP %s", buf.data);
+ /* OK, try to start the backup. */
if (PQsendQuery(conn, basebkp) == 0)
pg_fatal("could not send replication command \"%s\": %s",
"BASE_BACKUP", PQerrorMessage(conn));
@@ -2256,6 +2351,7 @@ main(int argc, char **argv)
{"version", no_argument, NULL, 'V'},
{"pgdata", required_argument, NULL, 'D'},
{"format", required_argument, NULL, 'F'},
+ {"incremental", required_argument, NULL, 'i'},
{"checkpoint", required_argument, NULL, 'c'},
{"create-slot", no_argument, NULL, 'C'},
{"max-rate", required_argument, NULL, 'r'},
@@ -2293,6 +2389,7 @@ main(int argc, char **argv)
int option_index;
char *compression_algorithm = "none";
char *compression_detail = NULL;
+ char *incremental_manifest = NULL;
CompressionLocation compressloc = COMPRESS_LOCATION_UNSPECIFIED;
pg_compress_specification client_compress;
@@ -2317,7 +2414,7 @@ main(int argc, char **argv)
atexit(cleanup_directories_atexit);
- while ((c = getopt_long(argc, argv, "c:Cd:D:F:h:l:nNp:Pr:Rs:S:t:T:U:vwWX:zZ:",
+ while ((c = getopt_long(argc, argv, "c:Cd:D:F:h:i:l:nNp:Pr:Rs:S:t:T:U:vwWX:zZ:",
long_options, &option_index)) != -1)
{
switch (c)
@@ -2352,6 +2449,9 @@ main(int argc, char **argv)
case 'h':
dbhost = pg_strdup(optarg);
break;
+ case 'i':
+ incremental_manifest = pg_strdup(optarg);
+ break;
case 'l':
label = pg_strdup(optarg);
break;
@@ -2765,7 +2865,7 @@ main(int argc, char **argv)
}
BaseBackup(compression_algorithm, compression_detail, compressloc,
- &client_compress);
+ &client_compress, incremental_manifest);
success = true;
return 0;
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index b9f5e1266b..bf765291e7 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -223,10 +223,10 @@ SKIP:
"check backup dir permissions");
}
-# Only archive_status directory should be copied in pg_wal/.
+# Only archive_status and summaries directories should be copied in pg_wal/.
is_deeply(
[ sort(slurp_dir("$tempdir/backup/pg_wal/")) ],
- [ sort qw(. .. archive_status) ],
+ [ sort qw(. .. archive_status summaries) ],
'no WAL files copied');
# Contents of these directories should not be copied.
diff --git a/src/bin/pg_combinebackup/.gitignore b/src/bin/pg_combinebackup/.gitignore
new file mode 100644
index 0000000000..d7e617438c
--- /dev/null
+++ b/src/bin/pg_combinebackup/.gitignore
@@ -0,0 +1 @@
+pg_combinebackup
diff --git a/src/bin/pg_combinebackup/Makefile b/src/bin/pg_combinebackup/Makefile
new file mode 100644
index 0000000000..78ba05e624
--- /dev/null
+++ b/src/bin/pg_combinebackup/Makefile
@@ -0,0 +1,52 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/bin/pg_combinebackup
+#
+# Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/bin/pg_combinebackup/Makefile
+#
+#-------------------------------------------------------------------------
+
+PGFILEDESC = "pg_combinebackup - combine incremental backups"
+PGAPPICON=win32
+
+subdir = src/bin/pg_combinebackup
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
+LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils
+
+OBJS = \
+ $(WIN32RES) \
+ pg_combinebackup.o \
+ backup_label.o \
+ copy_file.o \
+ load_manifest.o \
+ reconstruct.o \
+ write_manifest.o
+
+all: pg_combinebackup
+
+pg_combinebackup: $(OBJS) | submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
+install: all installdirs
+ $(INSTALL_PROGRAM) pg_combinebackup$(X) '$(DESTDIR)$(bindir)/pg_combinebackup$(X)'
+
+installdirs:
+ $(MKDIR_P) '$(DESTDIR)$(bindir)'
+
+uninstall:
+ rm -f '$(DESTDIR)$(bindir)/pg_combinebackup$(X)'
+
+clean distclean maintainer-clean:
+ rm -f pg_combinebackup$(X) $(OBJS)
+
+check:
+ $(prove_check)
+
+installcheck:
+ $(prove_installcheck)
diff --git a/src/bin/pg_combinebackup/backup_label.c b/src/bin/pg_combinebackup/backup_label.c
new file mode 100644
index 0000000000..2a62aa6fad
--- /dev/null
+++ b/src/bin/pg_combinebackup/backup_label.c
@@ -0,0 +1,281 @@
+/*-------------------------------------------------------------------------
+ *
+ * Read and manipulate backup label files
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_combinebackup/backup_label.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include <unistd.h>
+
+#include "access/xlogdefs.h"
+#include "backup_label.h"
+#include "common/logging.h"
+#include "common/file_perm.h"
+#include "write_manifest.h"
+
+static int get_eol_offset(StringInfo buf);
+static bool line_starts_with(char *s, char *e, char *match, char **sout);
+static bool parse_lsn(char *s, char *e, XLogRecPtr *lsn, char **c);
+static bool parse_tli(char *s, char *e, TimeLineID *tli);
+
+/*
+ * Parse a backup label file, starting at buf->cursor.
+ *
+ * We expect to find a START WAL LOCATION line, followed by a LSN, followed
+ * by a space; the resulting LSN is stored into *start_lsn.
+ *
+ * We expect to find a START TIMELINE line, followed by a TLI, followed by
+ * a newline; the resulting TLI is stored into *start_tli.
+ *
+ * We expect to find either both INCREMENTAL FROM LSN and INCREMENTAL FROM TLI
+ * or neither. If these are found, they should be followed by an LSN or TLI
+ * respectively and then by a newline, and the values will be stored into
+ * *previous_lsn and *previous_tli, respectively.
+ *
+ * Other lines in the provided backup_label data are ignored. filename is used
+ * for error reporting; errors are fatal.
+ */
+void
+parse_backup_label(char *filename, StringInfo buf,
+ TimeLineID *start_tli, XLogRecPtr *start_lsn,
+ TimeLineID *previous_tli, XLogRecPtr *previous_lsn)
+{
+ int found = 0;
+
+ *start_tli = 0;
+ *start_lsn = InvalidXLogRecPtr;
+ *previous_tli = 0;
+ *previous_lsn = InvalidXLogRecPtr;
+
+ while (buf->cursor < buf->len)
+ {
+ char *s = &buf->data[buf->cursor];
+ int eo = get_eol_offset(buf);
+ char *e = &buf->data[eo];
+ char *c;
+
+ if (line_starts_with(s, e, "START WAL LOCATION: ", &s))
+ {
+ if (!parse_lsn(s, e, start_lsn, &c))
+ pg_fatal("%s: could not parse START WAL LOCATION",
+ filename);
+ if (c >= e || *c != ' ')
+ pg_fatal("%s: improper terminator for START WAL LOCATION",
+ filename);
+ found |= 1;
+ }
+ else if (line_starts_with(s, e, "START TIMELINE: ", &s))
+ {
+ if (!parse_tli(s, e, start_tli))
+ pg_fatal("%s: could not parse TLI for START TIMELINE",
+ filename);
+ if (*start_tli == 0)
+ pg_fatal("%s: invalid TLI", filename);
+ found |= 2;
+ }
+ else if (line_starts_with(s, e, "INCREMENTAL FROM LSN: ", &s))
+ {
+ if (!parse_lsn(s, e, previous_lsn, &c))
+ pg_fatal("%s: could not parse INCREMENTAL FROM LSN",
+ filename);
+ if (c >= e || *c != '\n')
+ pg_fatal("%s: improper terminator for INCREMENTAL FROM LSN",
+ filename);
+ found |= 4;
+ }
+ else if (line_starts_with(s, e, "INCREMENTAL FROM TLI: ", &s))
+ {
+ if (!parse_tli(s, e, previous_tli))
+ pg_fatal("%s: could not parse INCREMENTAL FROM TLI",
+ filename);
+ if (*previous_tli == 0)
+ pg_fatal("%s: invalid TLI", filename);
+ found |= 8;
+ }
+
+ buf->cursor = eo;
+ }
+
+ if ((found & 1) == 0)
+ pg_fatal("%s: could not find START WAL LOCATION", filename);
+ if ((found & 2) == 0)
+ pg_fatal("%s: could not find START TIMELINE", filename);
+ if ((found & 4) != 0 && (found & 8) == 0)
+ pg_fatal("%s: INCREMENTAL FROM LSN requires INCREMENTAL FROM TLI", filename);
+ if ((found & 8) != 0 && (found & 4) == 0)
+ pg_fatal("%s: INCREMENTAL FROM TLI requires INCREMENTAL FROM LSN", filename);
+}
+
+/*
+ * Write a backup label file to the output directory.
+ *
+ * This will be identical to the provided backup_label file, except that the
+ * INCREMENTAL FROM LSN and INCREMENTAL FROM TLI lines will be omitted.
+ *
+ * The new file will be checksummed using the specified algorithm. If
+ * mwriter != NULL, it will be added to the manifest.
+ */
+void
+write_backup_label(char *output_directory, StringInfo buf,
+ pg_checksum_type checksum_type, manifest_writer *mwriter)
+{
+ char output_filename[MAXPGPATH];
+ int output_fd;
+ pg_checksum_context checksum_ctx;
+ uint8 checksum_payload[PG_CHECKSUM_MAX_LENGTH];
+ int checksum_length;
+
+ pg_checksum_init(&checksum_ctx, checksum_type);
+
+ snprintf(output_filename, MAXPGPATH, "%s/backup_label", output_directory);
+
+ if ((output_fd = open(output_filename,
+ O_WRONLY | O_CREAT | O_EXCL | PG_BINARY,
+ pg_file_create_mode)) < 0)
+ pg_fatal("could not open file \"%s\": %m", output_filename);
+
+ while (buf->cursor < buf->len)
+ {
+ char *s = &buf->data[buf->cursor];
+ int eo = get_eol_offset(buf);
+ char *e = &buf->data[eo];
+
+ if (!line_starts_with(s, e, "INCREMENTAL FROM LSN: ", NULL) &&
+ !line_starts_with(s, e, "INCREMENTAL FROM TLI: ", NULL))
+ {
+ ssize_t wb;
+
+ wb = write(output_fd, s, e - s);
+ if (wb != e - s)
+ {
+ if (wb < 0)
+ pg_fatal("could not write file \"%s\": %m", output_filename);
+ else
+ pg_fatal("could not write file \"%s\": wrote only %d of %d bytes",
+ output_filename, (int) wb, (int) (e - s));
+ }
+ if (pg_checksum_update(&checksum_ctx, (uint8 *) s, e - s) < 0)
+ pg_fatal("could not update checksum of file \"%s\"",
+ output_filename);
+ }
+
+ buf->cursor = eo;
+ }
+
+ if (close(output_fd) != 0)
+ pg_fatal("could not close \"%s\": %m", output_filename);
+
+ checksum_length = pg_checksum_final(&checksum_ctx, checksum_payload);
+
+ if (mwriter != NULL)
+ {
+ struct stat sb;
+
+ /*
+ * We could track the length ourselves, but must stat() to get the
+ * mtime.
+ */
+ if (stat(output_filename, &sb) < 0)
+ pg_fatal("could not stat file \"%s\": %m", output_filename);
+ add_file_to_manifest(mwriter, "backup_label", sb.st_size,
+ sb.st_mtime, checksum_type,
+ checksum_length, checksum_payload);
+ }
+}
+
+/*
+ * Return the offset at which the next line in the buffer starts, or there
+ * is none, the offset at which the buffer ends.
+ *
+ * The search begins at buf->cursor.
+ */
+static int
+get_eol_offset(StringInfo buf)
+{
+ int eo = buf->cursor;
+
+ while (eo < buf->len)
+ {
+ if (buf->data[eo] == '\n')
+ return eo + 1;
+ ++eo;
+ }
+
+ return eo;
+}
+
+/*
+ * Test whether the line that runs from s to e (inclusive of *s, but not
+ * inclusive of *e) starts with the match string provided, and return true
+ * or false according to whether or not this is the case.
+ *
+ * If the function returns true and if *sout != NULL, stores a pointer to the
+ * byte following the match into *sout.
+ */
+static bool
+line_starts_with(char *s, char *e, char *match, char **sout)
+{
+ while (s < e && *match != '\0' && *s == *match)
+ ++s, ++match;
+
+ if (*match == '\0' && sout != NULL)
+ *sout = s;
+
+ return (*match == '\0');
+}
+
+/*
+ * Parse an LSN starting at s and not stopping at or before e. The return value
+ * is true on success and otherwise false. On success, stores the result into
+ * *lsn and sets *c to the first character that is not part of the LSN.
+ */
+static bool
+parse_lsn(char *s, char *e, XLogRecPtr *lsn, char **c)
+{
+ char save = *e;
+ int nchars;
+ bool success;
+ unsigned hi;
+ unsigned lo;
+
+ *e = '\0';
+ success = (sscanf(s, "%X/%X%n", &hi, &lo, &nchars) == 2);
+ *e = save;
+
+ if (success)
+ {
+ *lsn = ((XLogRecPtr) hi) << 32 | (XLogRecPtr) lo;
+ *c = s + nchars;
+ }
+
+ return success;
+}
+
+/*
+ * Parse a TLI starting at s and stopping at or before e. The return value is
+ * true on success and otherwise false. On success, stores the result into
+ * *tli. If the first character that is not part of the TLI is anything other
+ * than a newline, that is deemed a failure.
+ */
+static bool
+parse_tli(char *s, char *e, TimeLineID *tli)
+{
+ char save = *e;
+ int nchars;
+ bool success;
+
+ *e = '\0';
+ success = (sscanf(s, "%u%n", tli, &nchars) == 1);
+ *e = save;
+
+ if (success && s[nchars] != '\n')
+ success = false;
+
+ return success;
+}
diff --git a/src/bin/pg_combinebackup/backup_label.h b/src/bin/pg_combinebackup/backup_label.h
new file mode 100644
index 0000000000..08d6ed67a9
--- /dev/null
+++ b/src/bin/pg_combinebackup/backup_label.h
@@ -0,0 +1,29 @@
+/*-------------------------------------------------------------------------
+ *
+ * Read and manipulate backup label files
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_combinebackup/backup_label.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef BACKUP_LABEL_H
+#define BACKUP_LABEL_H
+
+#include "common/checksum_helper.h"
+#include "lib/stringinfo.h"
+
+struct manifest_writer;
+
+extern void parse_backup_label(char *filename, StringInfo buf,
+ TimeLineID *start_tli,
+ XLogRecPtr *start_lsn,
+ TimeLineID *previous_tli,
+ XLogRecPtr *previous_lsn);
+extern void write_backup_label(char *output_directory, StringInfo buf,
+ pg_checksum_type checksum_type,
+ struct manifest_writer *mwriter);
+
+#endif /* BACKUP_LABEL_H */
diff --git a/src/bin/pg_combinebackup/copy_file.c b/src/bin/pg_combinebackup/copy_file.c
new file mode 100644
index 0000000000..8ba6cc09e4
--- /dev/null
+++ b/src/bin/pg_combinebackup/copy_file.c
@@ -0,0 +1,169 @@
+/*
+ * Copy entire files.
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_combinebackup/copy_file.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#ifdef HAVE_COPYFILE_H
+#include <copyfile.h>
+#endif
+#include <fcntl.h>
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include "common/file_perm.h"
+#include "common/logging.h"
+#include "copy_file.h"
+
+static void copy_file_blocks(const char *src, const char *dst,
+ pg_checksum_context *checksum_ctx);
+
+#ifdef WIN32
+static void copy_file_copyfile(const char *src, const char *dst);
+#endif
+
+/*
+ * Copy a regular file, optionally computing a checksum, and emitting
+ * appropriate debug messages. But if we're in dry-run mode, then just emit
+ * the messages and don't copy anything.
+ */
+void
+copy_file(const char *src, const char *dst,
+ pg_checksum_context *checksum_ctx, bool dry_run)
+{
+ /*
+ * In dry-run mode, we don't actually copy anything, nor do we read any
+ * data from the source file, but we do verify that we can open it.
+ */
+ if (dry_run)
+ {
+ int fd;
+
+ if ((fd = open(src, O_RDONLY | PG_BINARY)) < 0)
+ pg_fatal("could not open \"%s\": %m", src);
+ if (close(fd) < 0)
+ pg_fatal("could not close \"%s\": %m", src);
+ }
+
+ /*
+ * If we don't need to compute a checksum, then we can use any special
+ * operating system primitives that we know about to copy the file; this
+ * may be quicker than a naive block copy.
+ */
+ if (checksum_ctx->type != CHECKSUM_TYPE_NONE)
+ {
+ char *strategy_name = NULL;
+ void (*strategy_implementation) (const char *, const char *) = NULL;
+
+#ifdef WIN32
+ strategy_name = "CopyFile";
+ strategy_implementation = copy_file_copyfile;
+#endif
+
+ if (strategy_name != NULL)
+ {
+ if (dry_run)
+ pg_log_debug("would copy \"%s\" to \"%s\" using strategy %s",
+ src, dst, strategy_name);
+ else
+ {
+ pg_log_debug("copying \"%s\" to \"%s\" using strategy %s",
+ src, dst, strategy_name);
+ (*strategy_implementation) (src, dst);
+ }
+ return;
+ }
+ }
+
+ /*
+ * Fall back to the simple approach of reading and writing all the blocks,
+ * feeding them into the checksum context as we go.
+ */
+ if (dry_run)
+ {
+ if (checksum_ctx->type == CHECKSUM_TYPE_NONE)
+ pg_log_debug("would copy \"%s\" to \"%s\"",
+ src, dst);
+ else
+ pg_log_debug("would copy \"%s\" to \"%s\" and checksum with %s",
+ src, dst, pg_checksum_type_name(checksum_ctx->type));
+ }
+ else
+ {
+ if (checksum_ctx->type == CHECKSUM_TYPE_NONE)
+ pg_log_debug("copying \"%s\" to \"%s\"",
+ src, dst);
+ else
+ pg_log_debug("copying \"%s\" to \"%s\" and checksumming with %s",
+ src, dst, pg_checksum_type_name(checksum_ctx->type));
+ copy_file_blocks(src, dst, checksum_ctx);
+ }
+}
+
+/*
+ * Copy a file block by block, and optionally compute a checksum as we go.
+ */
+static void
+copy_file_blocks(const char *src, const char *dst,
+ pg_checksum_context *checksum_ctx)
+{
+ int src_fd;
+ int dest_fd;
+ uint8 *buffer;
+ const int buffer_size = 50 * BLCKSZ;
+ ssize_t rb;
+ unsigned offset = 0;
+
+ if ((src_fd = open(src, O_RDONLY | PG_BINARY, 0)) < 0)
+ pg_fatal("could not open file \"%s\": %m", src);
+
+ if ((dest_fd = open(dst, O_WRONLY | O_CREAT | O_EXCL | PG_BINARY,
+ pg_file_create_mode)) < 0)
+ pg_fatal("could not open file \"%s\": %m", dst);
+
+ buffer = pg_malloc(buffer_size);
+
+ while ((rb = read(src_fd, buffer, buffer_size)) > 0)
+ {
+ ssize_t wb;
+
+ if ((wb = write(dest_fd, buffer, rb)) != rb)
+ {
+ if (wb < 0)
+ pg_fatal("could not write file \"%s\": %m", dst);
+ else
+ pg_fatal("could not write file \"%s\": wrote only %d of %d bytes at offset %u",
+ dst, (int) wb, (int) rb, offset);
+ }
+
+ if (pg_checksum_update(checksum_ctx, buffer, rb) < 0)
+ pg_fatal("could not update checksum of file \"%s\"", dst);
+
+ offset += rb;
+ }
+
+ if (rb < 0)
+ pg_fatal("could not read file \"%s\": %m", dst);
+
+ pg_free(buffer);
+ close(src_fd);
+ close(dest_fd);
+}
+
+#ifdef WIN32
+static void
+copy_file_copyfile(const char *src, const char *dst)
+{
+ if (CopyFile(src, dst, true) == 0)
+ {
+ _dosmaperr(GetLastError());
+ pg_fatal("could not copy \"%s\" to \"%s\": %m", src, dst);
+ }
+}
+#endif /* WIN32 */
diff --git a/src/bin/pg_combinebackup/copy_file.h b/src/bin/pg_combinebackup/copy_file.h
new file mode 100644
index 0000000000..031030bacb
--- /dev/null
+++ b/src/bin/pg_combinebackup/copy_file.h
@@ -0,0 +1,19 @@
+/*
+ * Copy entire files.
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_combinebackup/copy_file.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef COPY_FILE_H
+#define COPY_FILE_H
+
+#include "common/checksum_helper.h"
+
+extern void copy_file(const char *src, const char *dst,
+ pg_checksum_context *checksum_ctx, bool dry_run);
+
+#endif /* COPY_FILE_H */
diff --git a/src/bin/pg_combinebackup/load_manifest.c b/src/bin/pg_combinebackup/load_manifest.c
new file mode 100644
index 0000000000..d0b8de7912
--- /dev/null
+++ b/src/bin/pg_combinebackup/load_manifest.c
@@ -0,0 +1,245 @@
+/*-------------------------------------------------------------------------
+ *
+ * Load data from a backup manifest into memory.
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_combinebackup/load_manifest.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include "common/hashfn.h"
+#include "common/logging.h"
+#include "common/parse_manifest.h"
+#include "load_manifest.h"
+
+/*
+ * For efficiency, we'd like our hash table containing information about the
+ * manifest to start out with approximately the correct number of entries.
+ * There's no way to know the exact number of entries without reading the whole
+ * file, but we can get an estimate by dividing the file size by the estimated
+ * number of bytes per line.
+ *
+ * This could be off by about a factor of two in either direction, because the
+ * checksum algorithm has a big impact on the line lengths; e.g. a SHA512
+ * checksum is 128 hex bytes, whereas a CRC-32C value is only 8, and there
+ * might be no checksum at all.
+ */
+#define ESTIMATED_BYTES_PER_MANIFEST_LINE 100
+
+/*
+ * Define a hash table which we can use to store information about the files
+ * mentioned in the backup manifest.
+ */
+static uint32 hash_string_pointer(char *s);
+#define SH_PREFIX manifest_files
+#define SH_ELEMENT_TYPE manifest_file
+#define SH_KEY_TYPE char *
+#define SH_KEY pathname
+#define SH_HASH_KEY(tb, key) hash_string_pointer(key)
+#define SH_EQUAL(tb, a, b) (strcmp(a, b) == 0)
+#define SH_SCOPE extern
+#define SH_RAW_ALLOCATOR pg_malloc0
+#define SH_DEFINE
+#include "lib/simplehash.h"
+
+static void record_manifest_details_for_file(JsonManifestParseContext *context,
+ char *pathname, size_t size,
+ pg_checksum_type checksum_type,
+ int checksum_length,
+ uint8 *checksum_payload);
+static void record_manifest_details_for_wal_range(JsonManifestParseContext *context,
+ TimeLineID tli,
+ XLogRecPtr start_lsn,
+ XLogRecPtr end_lsn);
+static void report_manifest_error(JsonManifestParseContext *context,
+ const char *fmt,...)
+ pg_attribute_printf(2, 3) pg_attribute_noreturn();
+
+/*
+ * Load backup_manifest files from an array of backups and produces an array
+ * of manifest_data objects.
+ *
+ * NB: Since load_backup_manifest() can return NULL, the resulting array could
+ * contain NULL entries.
+ */
+manifest_data **
+load_backup_manifests(int n_backups, char **backup_directories)
+{
+ manifest_data **result;
+ int i;
+
+ result = pg_malloc(sizeof(manifest_data *) * n_backups);
+ for (i = 0; i < n_backups; ++i)
+ result[i] = load_backup_manifest(backup_directories[i]);
+
+ return result;
+}
+
+/*
+ * Parse the backup_manifest file in the named backup directory. Construct a
+ * hash table with information about all the files it mentions, and a linked
+ * list of all the WAL ranges it mentions.
+ *
+ * If the backup_manifest file simply doesn't exist, logs a warning and returns
+ * NULL. Any other error, or any error parsing the contents of the file, is
+ * fatal.
+ */
+manifest_data *
+load_backup_manifest(char *backup_directory)
+{
+ char pathname[MAXPGPATH];
+ int fd;
+ struct stat statbuf;
+ off_t estimate;
+ uint32 initial_size;
+ manifest_files_hash *ht;
+ char *buffer;
+ int rc;
+ JsonManifestParseContext context;
+ manifest_data *result;
+
+ /* Open the manifest file. */
+ snprintf(pathname, MAXPGPATH, "%s/backup_manifest", backup_directory);
+ if ((fd = open(pathname, O_RDONLY | PG_BINARY, 0)) < 0)
+ {
+ if (errno == EEXIST)
+ {
+ pg_log_warning("\"%s\" does not exist", pathname);
+ return NULL;
+ }
+ pg_fatal("could not open file \"%s\": %m", pathname);
+ }
+
+ /* Figure out how big the manifest is. */
+ if (fstat(fd, &statbuf) != 0)
+ pg_fatal("could not stat file \"%s\": %m", pathname);
+
+ /* Guess how large to make the hash table based on the manifest size. */
+ estimate = statbuf.st_size / ESTIMATED_BYTES_PER_MANIFEST_LINE;
+ initial_size = Min(PG_UINT32_MAX, Max(estimate, 256));
+
+ /* Create the hash table. */
+ ht = manifest_files_create(initial_size, NULL);
+
+ /*
+ * Slurp in the whole file.
+ *
+ * This is not ideal, but there's currently no way to get pg_parse_json()
+ * to perform incremental parsing.
+ */
+ buffer = pg_malloc(statbuf.st_size);
+ rc = read(fd, buffer, statbuf.st_size);
+ if (rc != statbuf.st_size)
+ {
+ if (rc < 0)
+ pg_fatal("could not read file \"%s\": %m", pathname);
+ else
+ pg_fatal("could not read file \"%s\": read %d of %lld",
+ pathname, rc, (long long int) statbuf.st_size);
+ }
+
+ /* Close the manifest file. */
+ close(fd);
+
+ /* Parse the manifest. */
+ result = pg_malloc0(sizeof(manifest_data));
+ result->files = ht;
+ context.private_data = result;
+ context.perfile_cb = record_manifest_details_for_file;
+ context.perwalrange_cb = record_manifest_details_for_wal_range;
+ context.error_cb = report_manifest_error;
+ json_parse_manifest(&context, buffer, statbuf.st_size);
+
+ /* All done. */
+ pfree(buffer);
+ return result;
+}
+
+/*
+ * Report an error while parsing the manifest.
+ *
+ * We consider all such errors to be fatal errors. The manifest parser
+ * expects this function not to return.
+ */
+static void
+report_manifest_error(JsonManifestParseContext *context, const char *fmt,...)
+{
+ va_list ap;
+
+ va_start(ap, fmt);
+ pg_log_generic_v(PG_LOG_ERROR, PG_LOG_PRIMARY, gettext(fmt), ap);
+ va_end(ap);
+
+ exit(1);
+}
+
+/*
+ * Record details extracted from the backup manifest for one file.
+ */
+static void
+record_manifest_details_for_file(JsonManifestParseContext *context,
+ char *pathname, size_t size,
+ pg_checksum_type checksum_type,
+ int checksum_length, uint8 *checksum_payload)
+{
+ manifest_data *manifest = context->private_data;
+ manifest_file *m;
+ bool found;
+
+ /* Make a new entry in the hash table for this file. */
+ m = manifest_files_insert(manifest->files, pathname, &found);
+ if (found)
+ pg_fatal("duplicate path name in backup manifest: \"%s\"", pathname);
+
+ /* Initialize the entry. */
+ m->size = size;
+ m->checksum_type = checksum_type;
+ m->checksum_length = checksum_length;
+ m->checksum_payload = checksum_payload;
+}
+
+/*
+ * Record details extracted from the backup manifest for one WAL range.
+ */
+static void
+record_manifest_details_for_wal_range(JsonManifestParseContext *context,
+ TimeLineID tli,
+ XLogRecPtr start_lsn, XLogRecPtr end_lsn)
+{
+ manifest_data *manifest = context->private_data;
+ manifest_wal_range *range;
+
+ /* Allocate and initialize a struct describing this WAL range. */
+ range = palloc(sizeof(manifest_wal_range));
+ range->tli = tli;
+ range->start_lsn = start_lsn;
+ range->end_lsn = end_lsn;
+ range->prev = manifest->last_wal_range;
+ range->next = NULL;
+
+ /* Add it to the end of the list. */
+ if (manifest->first_wal_range == NULL)
+ manifest->first_wal_range = range;
+ else
+ manifest->last_wal_range->next = range;
+ manifest->last_wal_range = range;
+}
+
+/*
+ * Helper function for manifest_files hash table.
+ */
+static uint32
+hash_string_pointer(char *s)
+{
+ unsigned char *ss = (unsigned char *) s;
+
+ return hash_bytes(ss, strlen(s));
+}
diff --git a/src/bin/pg_combinebackup/load_manifest.h b/src/bin/pg_combinebackup/load_manifest.h
new file mode 100644
index 0000000000..2bfeeff156
--- /dev/null
+++ b/src/bin/pg_combinebackup/load_manifest.h
@@ -0,0 +1,67 @@
+/*-------------------------------------------------------------------------
+ *
+ * Load data from a backup manifest into memory.
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_combinebackup/load_manifest.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef LOAD_MANIFEST_H
+#define LOAD_MANIFEST_H
+
+#include "access/xlogdefs.h"
+#include "common/checksum_helper.h"
+
+/*
+ * Each file described by the manifest file is parsed to produce an object
+ * like this.
+ */
+typedef struct manifest_file
+{
+ uint32 status; /* hash status */
+ char *pathname;
+ size_t size;
+ pg_checksum_type checksum_type;
+ int checksum_length;
+ uint8 *checksum_payload;
+} manifest_file;
+
+#define SH_PREFIX manifest_files
+#define SH_ELEMENT_TYPE manifest_file
+#define SH_KEY_TYPE char *
+#define SH_SCOPE extern
+#define SH_RAW_ALLOCATOR pg_malloc0
+#define SH_DECLARE
+#include "lib/simplehash.h"
+
+/*
+ * Each WAL range described by the manifest file is parsed to produce an
+ * object like this.
+ */
+typedef struct manifest_wal_range
+{
+ TimeLineID tli;
+ XLogRecPtr start_lsn;
+ XLogRecPtr end_lsn;
+ struct manifest_wal_range *next;
+ struct manifest_wal_range *prev;
+} manifest_wal_range;
+
+/*
+ * All the data parsed from a backup_manifest file.
+ */
+typedef struct manifest_data
+{
+ manifest_files_hash *files;
+ manifest_wal_range *first_wal_range;
+ manifest_wal_range *last_wal_range;
+} manifest_data;
+
+extern manifest_data *load_backup_manifest(char *backup_directory);
+extern manifest_data **load_backup_manifests(int n_backups,
+ char **backup_directories);
+
+#endif /* LOAD_MANIFEST_H */
diff --git a/src/bin/pg_combinebackup/meson.build b/src/bin/pg_combinebackup/meson.build
new file mode 100644
index 0000000000..e402d6f50e
--- /dev/null
+++ b/src/bin/pg_combinebackup/meson.build
@@ -0,0 +1,38 @@
+# Copyright (c) 2022-2023, PostgreSQL Global Development Group
+
+pg_combinebackup_sources = files(
+ 'pg_combinebackup.c',
+ 'backup_label.c',
+ 'copy_file.c',
+ 'load_manifest.c',
+ 'reconstruct.c',
+ 'write_manifest.c',
+)
+
+if host_system == 'windows'
+ pg_combinebackup_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'pg_combinebackup',
+ '--FILEDESC', 'pg_combinebackup - combine incremental backups',])
+endif
+
+pg_combinebackup = executable('pg_combinebackup',
+ pg_combinebackup_sources,
+ dependencies: [frontend_code],
+ kwargs: default_bin_args,
+)
+bin_targets += pg_combinebackup
+
+tests += {
+ 'name': 'pg_combinebackup',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'tap': {
+ 'tests': [
+ 't/001_basic.pl',
+ 't/002_compare_backups.pl',
+ 't/003_timeline.pl',
+ 't/004_manifest.pl',
+ 't/005_integrity.pl',
+ ],
+ }
+}
diff --git a/src/bin/pg_combinebackup/pg_combinebackup.c b/src/bin/pg_combinebackup/pg_combinebackup.c
new file mode 100644
index 0000000000..e607f35edb
--- /dev/null
+++ b/src/bin/pg_combinebackup/pg_combinebackup.c
@@ -0,0 +1,1271 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_combinebackup.c
+ * Combine incremental backups with prior backups.
+ *
+ * Copyright (c) 2017-2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_combinebackup/pg_combinebackup.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include <dirent.h>
+#include <fcntl.h>
+#include <limits.h>
+
+#include "access/xlogdefs.h"
+#include "backup_label.h"
+#include "common/blkreftable.h"
+#include "common/checksum_helper.h"
+#include "common/controldata_utils.h"
+#include "common/file_perm.h"
+#include "common/file_utils.h"
+#include "common/logging.h"
+#include "copy_file.h"
+#include "fe_utils/option_utils.h"
+#include "lib/stringinfo.h"
+#include "load_manifest.h"
+#include "getopt_long.h"
+#include "reconstruct.h"
+#include "write_manifest.h"
+
+/* Incremental file naming convention. */
+#define INCREMENTAL_PREFIX "INCREMENTAL."
+#define INCREMENTAL_PREFIX_LENGTH 12
+
+/*
+ * Tracking for directories that need to be removed, or have their contents
+ * removed, if the operation fails.
+ */
+typedef struct cb_cleanup_dir
+{
+ char *target_path;
+ bool rmtopdir;
+ struct cb_cleanup_dir *next;
+} cb_cleanup_dir;
+
+/*
+ * Stores a tablespace mapping provided using -T, --tablespace-mapping.
+ */
+typedef struct cb_tablespace_mapping
+{
+ char old_dir[MAXPGPATH];
+ char new_dir[MAXPGPATH];
+ struct cb_tablespace_mapping *next;
+} cb_tablespace_mapping;
+
+/*
+ * Stores data parsed from all command-line options.
+ */
+typedef struct cb_options
+{
+ bool debug;
+ char *output;
+ bool dry_run;
+ bool no_sync;
+ cb_tablespace_mapping *tsmappings;
+ pg_checksum_type manifest_checksums;
+ bool no_manifest;
+ DataDirSyncMethod sync_method;
+} cb_options;
+
+/*
+ * Data about a tablespace.
+ *
+ * Every normal tablespace needs a tablespace mapping, but in-place tablespaces
+ * don't, so the list of tablespaces can contain more entries than the list of
+ * tablespace mappings.
+ */
+typedef struct cb_tablespace
+{
+ Oid oid;
+ bool in_place;
+ char old_dir[MAXPGPATH];
+ char new_dir[MAXPGPATH];
+ struct cb_tablespace *next;
+} cb_tablespace;
+
+/* Directories to be removed if we exit uncleanly. */
+cb_cleanup_dir *cleanup_dir_list = NULL;
+
+static void add_tablespace_mapping(cb_options *opt, char *arg);
+static StringInfo check_backup_label_files(int n_backups, char **backup_dirs);
+static void check_control_files(int n_backups, char **backup_dirs);
+static void check_input_dir_permissions(char *dir);
+static void cleanup_directories_atexit(void);
+static void create_output_directory(char *dirname, cb_options *opt);
+static void help(const char *progname);
+static bool parse_oid(char *s, Oid *result);
+static void process_directory_recursively(Oid tsoid,
+ char *input_directory,
+ char *output_directory,
+ char *relative_path,
+ int n_prior_backups,
+ char **prior_backup_dirs,
+ manifest_data **manifests,
+ manifest_writer *mwriter,
+ cb_options *opt);
+static int read_pg_version_file(char *directory);
+static void remember_to_cleanup_directory(char *target_path, bool rmtopdir);
+static void reset_directory_cleanup_list(void);
+static cb_tablespace *scan_for_existing_tablespaces(char *pathname,
+ cb_options *opt);
+static void slurp_file(int fd, char *filename, StringInfo buf, int maxlen);
+
+/*
+ * Main program.
+ */
+int
+main(int argc, char *argv[])
+{
+ static struct option long_options[] = {
+ {"debug", no_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"no-sync", no_argument, NULL, 'N'},
+ {"output", required_argument, NULL, 'o'},
+ {"tablespace-mapping", no_argument, NULL, 'T'},
+ {"manifest-checksums", required_argument, NULL, 1},
+ {"no-manifest", no_argument, NULL, 2},
+ {"sync-method", required_argument, NULL, 3},
+ {NULL, 0, NULL, 0}
+ };
+
+ const char *progname;
+ char *last_input_dir;
+ int optindex;
+ int c;
+ int n_backups;
+ int n_prior_backups;
+ int version;
+ char **prior_backup_dirs;
+ cb_options opt;
+ cb_tablespace *tablespaces;
+ cb_tablespace *ts;
+ StringInfo last_backup_label;
+ manifest_data **manifests;
+ manifest_writer *mwriter;
+
+ pg_logging_init(argv[0]);
+ progname = get_progname(argv[0]);
+ handle_help_version_opts(argc, argv, progname, help);
+
+ memset(&opt, 0, sizeof(opt));
+ opt.manifest_checksums = CHECKSUM_TYPE_CRC32C;
+ opt.sync_method = DATA_DIR_SYNC_METHOD_FSYNC;
+
+ /* process command-line options */
+ while ((c = getopt_long(argc, argv, "do:nNPT:",
+ long_options, &optindex)) != -1)
+ {
+ switch (c)
+ {
+ case 'd':
+ opt.debug = true;
+ pg_logging_increase_verbosity();
+ break;
+ case 'o':
+ opt.output = optarg;
+ break;
+ case 'n':
+ opt.dry_run = true;
+ break;
+ case 'N':
+ opt.no_sync = true;
+ break;
+ case 'T':
+ add_tablespace_mapping(&opt, optarg);
+ break;
+ case 1:
+ if (!pg_checksum_parse_type(optarg,
+ &opt.manifest_checksums))
+ pg_fatal("unrecognized checksum algorithm: \"%s\"",
+ optarg);
+ break;
+ case 2:
+ opt.no_manifest = true;
+ break;
+ case 3:
+ if (!parse_sync_method(optarg, &opt.sync_method))
+ exit(1);
+ break;
+ default:
+ /* getopt_long already emitted a complaint */
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ if (optind >= argc)
+ {
+ pg_log_error("%s: no input directories specified", progname);
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ if (opt.output == NULL)
+ pg_fatal("no output directory specified");
+
+ /* If no manifest is needed, no checksums are needed, either. */
+ if (opt.no_manifest)
+ opt.manifest_checksums = CHECKSUM_TYPE_NONE;
+
+ /* Read the server version from the final backup. */
+ version = read_pg_version_file(argv[argc - 1]);
+
+ /* Sanity-check control files. */
+ n_backups = argc - optind;
+ check_control_files(n_backups, argv + optind);
+
+ /* Sanity-check backup_label files, and get the contents of the last one. */
+ last_backup_label = check_backup_label_files(n_backups, argv + optind);
+
+ /* Load backup manifests. */
+ manifests = load_backup_manifests(n_backups, argv + optind);
+
+ /* Figure out which tablespaces are going to be included in the output. */
+ last_input_dir = argv[argc - 1];
+ check_input_dir_permissions(last_input_dir);
+ tablespaces = scan_for_existing_tablespaces(last_input_dir, &opt);
+
+ /*
+ * Create output directories.
+ *
+ * We create one output directory for the main data directory plus one for
+ * each non-in-place tablespace. create_output_directory() will arrange
+ * for those directories to be cleaned up on failure. In-place tablespaces
+ * aren't handled at this stage because they're located beneath the main
+ * output directory, and thus the cleanup of that directory will get rid
+ * of them. Plus, the pg_tblspc directory that needs to contain them
+ * doesn't exist yet.
+ */
+ atexit(cleanup_directories_atexit);
+ create_output_directory(opt.output, &opt);
+ for (ts = tablespaces; ts != NULL; ts = ts->next)
+ if (!ts->in_place)
+ create_output_directory(ts->new_dir, &opt);
+
+ /* If we need to write a backup_manifest, prepare to do so. */
+ if (!opt.dry_run && !opt.no_manifest)
+ mwriter = create_manifest_writer(opt.output);
+ else
+ mwriter = NULL;
+
+ /* Write backup label into output directory. */
+ if (opt.dry_run)
+ pg_log_debug("would generate \"%s/backup_label\"", opt.output);
+ else
+ {
+ pg_log_debug("generating \"%s/backup_label\"", opt.output);
+ last_backup_label->cursor = 0;
+ write_backup_label(opt.output, last_backup_label,
+ opt.manifest_checksums, mwriter);
+ }
+
+ /*
+ * We'll need the pathnames to the prior backups. By "prior" we mean all
+ * but the last one listed on the command line.
+ */
+ n_prior_backups = argc - optind - 1;
+ prior_backup_dirs = argv + optind;
+
+ /* Process everything that's not part of a user-defined tablespace. */
+ pg_log_debug("processing backup directory \"%s\"", last_input_dir);
+ process_directory_recursively(InvalidOid, last_input_dir, opt.output,
+ NULL, n_prior_backups, prior_backup_dirs,
+ manifests, mwriter, &opt);
+
+ /* Process user-defined tablespaces. */
+ for (ts = tablespaces; ts != NULL; ts = ts->next)
+ {
+ pg_log_debug("processing tablespace directory \"%s\"", ts->old_dir);
+
+ /*
+ * If it's a normal tablespace, we need to set up a symbolic link from
+ * pg_tblspc/${OID} to the target directory; if it's an in-place
+ * tablespace, we need to create a directory at pg_tblspc/${OID}.
+ */
+ if (!ts->in_place)
+ {
+ char linkpath[MAXPGPATH];
+
+ snprintf(linkpath, MAXPGPATH, "%s/pg_tblspc/%u", opt.output,
+ ts->oid);
+
+ if (opt.dry_run)
+ pg_log_debug("would create symbolic link from \"%s\" to \"%s\"",
+ linkpath, ts->new_dir);
+ else
+ {
+ pg_log_debug("creating symbolic link from \"%s\" to \"%s\"",
+ linkpath, ts->new_dir);
+ if (symlink(ts->new_dir, linkpath) != 0)
+ pg_fatal("could not create symbolic link from \"%s\" to \"%s\": %m",
+ linkpath, ts->new_dir);
+ }
+ }
+ else
+ {
+ if (opt.dry_run)
+ pg_log_debug("would create directory \"%s\"", ts->new_dir);
+ else
+ {
+ pg_log_debug("creating directory \"%s\"", ts->new_dir);
+ if (pg_mkdir_p(ts->new_dir, pg_dir_create_mode) == -1)
+ pg_fatal("could not create directory \"%s\": %m",
+ ts->new_dir);
+ }
+ }
+
+ /* OK, now handle the directory contents. */
+ process_directory_recursively(ts->oid, ts->old_dir, ts->new_dir,
+ NULL, n_prior_backups, prior_backup_dirs,
+ manifests, mwriter, &opt);
+ }
+
+ /* Finalize the backup_manifest, if we're generating one. */
+ if (mwriter != NULL)
+ finalize_manifest(mwriter,
+ manifests[n_prior_backups]->first_wal_range);
+
+ /* fsync that output directory unless we've been told not to do so */
+ if (!opt.no_sync)
+ {
+ if (opt.dry_run)
+ pg_log_debug("would recursively fsync \"%s\"", opt.output);
+ else
+ {
+ pg_log_debug("recursively fsyncing \"%s\"", opt.output);
+ sync_pgdata(opt.output, version * 10000, opt.sync_method);
+ }
+ }
+
+ /* It's a success, so don't remove the output directories. */
+ reset_directory_cleanup_list();
+ exit(0);
+}
+
+/*
+ * Process the option argument for the -T, --tablespace-mapping switch.
+ */
+static void
+add_tablespace_mapping(cb_options *opt, char *arg)
+{
+ cb_tablespace_mapping *tsmap = pg_malloc0(sizeof(cb_tablespace_mapping));
+ char *dst;
+ char *dst_ptr;
+ char *arg_ptr;
+
+ /*
+ * Basically, we just want to copy everything before the equals sign to
+ * tsmap->old_dir and everything afterwards to tsmap->new_dir, but if
+ * there's more or less than one equals sign, that's an error, and if
+ * there's an equals sign preceded by a backslash, don't treat it as a
+ * field separator but instead copy a literal equals sign.
+ */
+ dst_ptr = dst = tsmap->old_dir;
+ for (arg_ptr = arg; *arg_ptr != '\0'; arg_ptr++)
+ {
+ if (dst_ptr - dst >= MAXPGPATH)
+ pg_fatal("directory name too long");
+
+ if (*arg_ptr == '\\' && *(arg_ptr + 1) == '=')
+ ; /* skip backslash escaping = */
+ else if (*arg_ptr == '=' && (arg_ptr == arg || *(arg_ptr - 1) != '\\'))
+ {
+ if (tsmap->new_dir[0] != '\0')
+ pg_fatal("multiple \"=\" signs in tablespace mapping");
+ else
+ dst = dst_ptr = tsmap->new_dir;
+ }
+ else
+ *dst_ptr++ = *arg_ptr;
+ }
+ if (!tsmap->old_dir[0] || !tsmap->new_dir[0])
+ pg_fatal("invalid tablespace mapping format \"%s\", must be \"OLDDIR=NEWDIR\"", arg);
+
+ /*
+ * All tablespaces are created with absolute directories, so specifying a
+ * non-absolute path here would never match, possibly confusing users.
+ *
+ * In contrast to pg_basebackup, both the old and new directories are on
+ * the local machine, so the local machine's definition of an absolute
+ * path is the only relevant one.
+ */
+ if (!is_absolute_path(tsmap->old_dir))
+ pg_fatal("old directory is not an absolute path in tablespace mapping: %s",
+ tsmap->old_dir);
+
+ if (!is_absolute_path(tsmap->new_dir))
+ pg_fatal("old directory is not an absolute path in tablespace mapping: %s",
+ tsmap->new_dir);
+
+ /* Canonicalize paths to avoid spurious failures when comparing. */
+ canonicalize_path(tsmap->old_dir);
+ canonicalize_path(tsmap->new_dir);
+
+ /* Add it to the list. */
+ tsmap->next = opt->tsmappings;
+ opt->tsmappings = tsmap;
+}
+
+/*
+ * Check that the backup_label files form a coherent backup chain, and return
+ * the contents of the backup_label file from the latest backup.
+ */
+static StringInfo
+check_backup_label_files(int n_backups, char **backup_dirs)
+{
+ StringInfo buf = makeStringInfo();
+ StringInfo lastbuf = buf;
+ int i;
+ TimeLineID check_tli = 0;
+ XLogRecPtr check_lsn = InvalidXLogRecPtr;
+
+ /* Try to read each backup_label file in turn, last to first. */
+ for (i = n_backups - 1; i >= 0; --i)
+ {
+ char pathbuf[MAXPGPATH];
+ int fd;
+ TimeLineID start_tli;
+ TimeLineID previous_tli;
+ XLogRecPtr start_lsn;
+ XLogRecPtr previous_lsn;
+
+ /* Open the backup_label file. */
+ snprintf(pathbuf, MAXPGPATH, "%s/backup_label", backup_dirs[i]);
+ pg_log_debug("reading \"%s\"", pathbuf);
+ if ((fd = open(pathbuf, O_RDONLY, 0)) < 0)
+ pg_fatal("could not open file \"%s\": %m", pathbuf);
+
+ /*
+ * Slurp the whole file into memory.
+ *
+ * The exact size limit that we impose here doesn't really matter --
+ * most of what's supposed to be in the file is fixed size and quite
+ * short. However, the length of the backup_label is limited (at least
+ * by some parts of the code) to MAXGPATH, so include that value in
+ * the maximum length that we tolerate.
+ */
+ slurp_file(fd, pathbuf, buf, 10000 + MAXPGPATH);
+
+ /* Close the file. */
+ if (close(fd) != 0)
+ pg_fatal("could not close \"%s\": %m", pathbuf);
+
+ /* Parse the file contents. */
+ parse_backup_label(pathbuf, buf, &start_tli, &start_lsn,
+ &previous_tli, &previous_lsn);
+
+ /*
+ * Sanity checks.
+ *
+ * XXX. It's actually not required that start_lsn == check_lsn. It
+ * would be OK if start_lsn > check_lsn provided that start_lsn is
+ * less than or equal to the relevant switchpoint. But at the moment
+ * we don't have that information.
+ */
+ if (i > 0 && previous_tli == 0)
+ pg_fatal("backup at \"%s\" is a full backup, but only the first backup should be a full backup",
+ backup_dirs[i]);
+ if (i == 0 && previous_tli != 0)
+ pg_fatal("backup at \"%s\" is an incremental backup, but the first backup should be a full backup",
+ backup_dirs[i]);
+ if (i < n_backups - 1 && start_tli != check_tli)
+ pg_fatal("backup at \"%s\" starts on timeline %u, but expected %u",
+ backup_dirs[i], start_tli, check_tli);
+ if (i < n_backups - 1 && start_lsn != check_lsn)
+ pg_fatal("backup at \"%s\" starts at LSN %X/%X, but expected %X/%X",
+ backup_dirs[i],
+ LSN_FORMAT_ARGS(start_lsn),
+ LSN_FORMAT_ARGS(check_lsn));
+ check_tli = previous_tli;
+ check_lsn = previous_lsn;
+
+ /*
+ * The last backup label in the chain needs to be saved for later use,
+ * while the others are only needed within this loop.
+ */
+ if (lastbuf == buf)
+ buf = makeStringInfo();
+ else
+ resetStringInfo(buf);
+ }
+
+ /* Free memory that we don't need any more. */
+ if (lastbuf != buf)
+ {
+ pfree(buf->data);
+ pfree(buf);
+ }
+
+ /*
+ * Return the data from the first backup_info that we read (which is the
+ * backup_label from the last directory specified on the command line).
+ */
+ return lastbuf;
+}
+
+/*
+ * Sanity check control files.
+ */
+static void
+check_control_files(int n_backups, char **backup_dirs)
+{
+ int i;
+ uint64 system_identifier = 0; /* placate compiler */
+
+ /* Try to read each control file in turn, last to first. */
+ for (i = n_backups - 1; i >= 0; --i)
+ {
+ ControlFileData *control_file;
+ bool crc_ok;
+
+ pg_log_debug("reading \"%s/global/pg_control\"", backup_dirs[i]);
+ control_file = get_controlfile(backup_dirs[i], &crc_ok);
+
+ /* Control file contents not meaningful if CRC is bad. */
+ if (!crc_ok)
+ pg_fatal("%s/global/pg_control: crc is incorrect", backup_dirs[i]);
+
+ /* Can't interpret control file if not current version. */
+ if (control_file->pg_control_version != PG_CONTROL_VERSION)
+ pg_fatal("%s/global/pg_control: unexpected control file version",
+ backup_dirs[i]);
+
+ /* System identifiers should all match. */
+ if (i == n_backups - 1)
+ system_identifier = control_file->system_identifier;
+ else if (system_identifier != control_file->system_identifier)
+ pg_fatal("%s/global/pg_control: expected system identifier %llu, but found %llu",
+ backup_dirs[i], (unsigned long long) system_identifier,
+ (unsigned long long) control_file->system_identifier);
+
+ /* Release memory. */
+ pfree(control_file);
+ }
+
+ /*
+ * If debug output is enabled, make a note of the system identifier that
+ * we found in all of the relevant control files.
+ */
+ pg_log_debug("system identifier is %llu",
+ (unsigned long long) system_identifier);
+}
+
+/*
+ * Set default permissions for new files and directories based on the
+ * permissions of the given directory. The intent here is that the output
+ * directory should use the same permissions scheme as the final input
+ * directory.
+ */
+static void
+check_input_dir_permissions(char *dir)
+{
+ struct stat st;
+
+ if (stat(dir, &st) != 0)
+ pg_fatal("could not stat \"%s\": %m", dir);
+
+ SetDataDirectoryCreatePerm(st.st_mode);
+}
+
+/*
+ * Clean up output directories before exiting.
+ */
+static void
+cleanup_directories_atexit(void)
+{
+ while (cleanup_dir_list != NULL)
+ {
+ cb_cleanup_dir *dir = cleanup_dir_list;
+
+ if (dir->rmtopdir)
+ {
+ pg_log_info("removing output directory \"%s\"", dir->target_path);
+ if (!rmtree(dir->target_path, dir->rmtopdir))
+ pg_log_error("failed to remove output directory");
+ }
+ else
+ {
+ pg_log_info("removing contents of output directory \"%s\"",
+ dir->target_path);
+ if (!rmtree(dir->target_path, dir->rmtopdir))
+ pg_log_error("failed to remove contents of output directory");
+ }
+
+ cleanup_dir_list = cleanup_dir_list->next;
+ pfree(dir);
+ }
+}
+
+/*
+ * Create the named output directory, unless it already exists or we're in
+ * dry-run mode. If it already exists but is not empty, that's a fatal error.
+ *
+ * Adds the created directory to the list of directories to be cleaned up
+ * at process exit.
+ */
+static void
+create_output_directory(char *dirname, cb_options *opt)
+{
+ switch (pg_check_dir(dirname))
+ {
+ case 0:
+ if (opt->dry_run)
+ {
+ pg_log_debug("would create directory \"%s\"", dirname);
+ return;
+ }
+ pg_log_debug("creating directory \"%s\"", dirname);
+ if (pg_mkdir_p(dirname, pg_dir_create_mode) == -1)
+ pg_fatal("could not create directory \"%s\": %m", dirname);
+ remember_to_cleanup_directory(dirname, true);
+ break;
+
+ case 1:
+ pg_log_debug("using existing directory \"%s\"", dirname);
+ remember_to_cleanup_directory(dirname, false);
+ break;
+
+ case 2:
+ case 3:
+ case 4:
+ pg_fatal("directory \"%s\" exists but is not empty", dirname);
+
+ case -1:
+ pg_fatal("could not access directory \"%s\": %m", dirname);
+ }
+}
+
+/*
+ * help
+ *
+ * Prints help page for the program
+ *
+ * progname: the name of the executed program, such as "pg_combinebackup"
+ */
+static void
+help(const char *progname)
+{
+ printf(_("%s reconstructs full backups from incrementals.\n\n"), progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION]... DIRECTORY...\n"), progname);
+ printf(_("\nOptions:\n"));
+ printf(_(" -d, --debug generate lots of debugging output\n"));
+ printf(_(" -n, --dry-run don't actually do anything\n"));
+ printf(_(" -N, --no-sync do not wait for changes to be written safely to disk\n"));
+ printf(_(" -o, --output output directory\n"));
+ printf(_(" -T, --tablespace-mapping=OLDDIR=NEWDIR\n"));
+ printf(_(" relocate tablespace in OLDDIR to NEWDIR\n"));
+ printf(_(" --manifest-checksums=SHA{224,256,384,512}|CRC32C|NONE\n"
+ " use algorithm for manifest checksums\n"));
+ printf(_(" --no-manifest suppress generation of backup manifest\n"));
+ printf(_(" --sync-method=METHOD set method for syncing files to disk\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
+
+ printf(_("\nReport bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+ printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
+
+/*
+ * Try to parse a string as a non-zero OID without leading zeroes.
+ *
+ * If it works, return true and set *result to the answer, else return false.
+ */
+static bool
+parse_oid(char *s, Oid *result)
+{
+ Oid oid;
+ char *ep;
+
+ errno = 0;
+ oid = strtoul(s, &ep, 10);
+ if (errno != 0 || *ep != '\0' || oid < 1 || oid > PG_UINT32_MAX)
+ return false;
+
+ *result = oid;
+ return true;
+}
+
+/*
+ * Copy files from the input directory to the output directory, reconstructing
+ * full files from incremental files as required.
+ *
+ * If processing is a user-defined tablespace, the tsoid should be the OID
+ * of that tablespace and input_directory and output_directory should be the
+ * toplevel input and output directories for that tablespace. Otherwise,
+ * tsoid should be InvalidOid and input_directory and output_directory should
+ * be the main input and output directories.
+ *
+ * relative_path is the path beneath the given input and output directories
+ * that we are currently processing. If NULL, it indicates that we're
+ * processing the input and output directories themselves.
+ *
+ * n_prior_backups is the number of prior backups that we have available.
+ * This doesn't count the very last backup, which is referenced by
+ * output_directory, just the older ones. prior_backup_dirs is an array of
+ * the locations of those previous backups.
+ */
+static void
+process_directory_recursively(Oid tsoid,
+ char *input_directory,
+ char *output_directory,
+ char *relative_path,
+ int n_prior_backups,
+ char **prior_backup_dirs,
+ manifest_data **manifests,
+ manifest_writer *mwriter,
+ cb_options *opt)
+{
+ char ifulldir[MAXPGPATH];
+ char ofulldir[MAXPGPATH];
+ char manifest_prefix[MAXPGPATH];
+ DIR *dir;
+ struct dirent *de;
+ bool is_pg_tblspc;
+ bool is_pg_wal;
+ manifest_data *latest_manifest = manifests[n_prior_backups];
+ pg_checksum_type checksum_type;
+
+ StaticAssertStmt(strlen(INCREMENTAL_PREFIX) == INCREMENTAL_PREFIX_LENGTH,
+ "INCREMENTAL_PREFIX_LENGTH is incorrect");
+
+ /*
+ * pg_tblspc and pg_wal are special cases, so detect those here.
+ *
+ * pg_tblspc is only special at the top level, but subdirectories of
+ * pg_wal are just as special as the top level directory.
+ *
+ * Since incremental backup does not exist in pre-v10 versions, we don't
+ * have to worry about the old pg_xlog naming.
+ */
+ is_pg_tblspc = !OidIsValid(tsoid) && relative_path != NULL &&
+ strcmp(relative_path, "pg_tblspc") == 0;
+ is_pg_wal = !OidIsValid(tsoid) && relative_path != NULL &&
+ (strcmp(relative_path, "pg_wal") == 0 ||
+ strncmp(relative_path, "pg_wal/", 7) == 0);
+
+ /*
+ * If we're under pg_wal, then we don't need checksums, because these
+ * files aren't included in the backup manifest. Otherwise use whatever
+ * type of checksum is configured.
+ */
+ if (!is_pg_wal)
+ checksum_type = opt->manifest_checksums;
+ else
+ checksum_type = CHECKSUM_TYPE_NONE;
+
+ /*
+ * Append the relative path to the input and output directories, and
+ * figure out the appropriate prefix to add to files in this directory
+ * when looking them up in a backup manifest.
+ */
+ if (relative_path == NULL)
+ {
+ strncpy(ifulldir, input_directory, MAXPGPATH);
+ strncpy(ofulldir, output_directory, MAXPGPATH);
+ if (OidIsValid(tsoid))
+ snprintf(manifest_prefix, MAXPGPATH, "pg_tblspc/%u/", tsoid);
+ else
+ manifest_prefix[0] = '\0';
+ }
+ else
+ {
+ snprintf(ifulldir, MAXPGPATH, "%s/%s", input_directory,
+ relative_path);
+ snprintf(ofulldir, MAXPGPATH, "%s/%s", output_directory,
+ relative_path);
+ if (OidIsValid(tsoid))
+ snprintf(manifest_prefix, MAXPGPATH, "pg_tblspc/%u/%s/",
+ tsoid, relative_path);
+ else
+ snprintf(manifest_prefix, MAXPGPATH, "%s/", relative_path);
+ }
+
+ /*
+ * Toplevel output directories have already been created by the time this
+ * function is called, but any subdirectories are our responsibility.
+ */
+ if (relative_path != NULL)
+ {
+ if (opt->dry_run)
+ pg_log_debug("would create directory \"%s\"", ofulldir);
+ else
+ {
+ pg_log_debug("creating directory \"%s\"", ofulldir);
+ if (mkdir(ofulldir, pg_dir_create_mode) == -1)
+ pg_fatal("could not create directory \"%s\": %m", ofulldir);
+ }
+ }
+
+ /* It's time to scan the directory. */
+ if ((dir = opendir(ifulldir)) == NULL)
+ pg_fatal("could not open directory \"%s\": %m", ifulldir);
+ while (errno = 0, (de = readdir(dir)) != NULL)
+ {
+ PGFileType type;
+ char ifullpath[MAXPGPATH];
+ char ofullpath[MAXPGPATH];
+ char manifest_path[MAXPGPATH];
+ Oid oid = InvalidOid;
+ int checksum_length = 0;
+ uint8 *checksum_payload = NULL;
+ pg_checksum_context checksum_ctx;
+
+ /* Ignore "." and ".." entries. */
+ if (strcmp(de->d_name, ".") == 0 ||
+ strcmp(de->d_name, "..") == 0)
+ continue;
+
+ /* Construct input path. */
+ snprintf(ifullpath, MAXPGPATH, "%s/%s", ifulldir, de->d_name);
+
+ /* Figure out what kind of directory entry this is. */
+ type = get_dirent_type(ifullpath, de, false, PG_LOG_ERROR);
+ if (type == PGFILETYPE_ERROR)
+ exit(1);
+
+ /*
+ * If we're processing pg_tblspc, then check whether the filename
+ * looks like it could be a tablespace OID. If so, and if the
+ * directory entry is a symbolic link or a directory, skip it.
+ *
+ * Our goal here is to ignore anything that would have been considered
+ * by scan_for_existing_tablespaces to be a tablespace.
+ */
+ if (is_pg_tblspc && parse_oid(de->d_name, &oid) &&
+ (type == PGFILETYPE_LNK || type == PGFILETYPE_DIR))
+ continue;
+
+ /* If it's a directory, recurse. */
+ if (type == PGFILETYPE_DIR)
+ {
+ char new_relative_path[MAXPGPATH];
+
+ /* Append new pathname component to relative path. */
+ if (relative_path == NULL)
+ strncpy(new_relative_path, de->d_name, MAXPGPATH);
+ else
+ snprintf(new_relative_path, MAXPGPATH, "%s/%s", relative_path,
+ de->d_name);
+
+ /* And recurse. */
+ process_directory_recursively(tsoid,
+ input_directory, output_directory,
+ new_relative_path,
+ n_prior_backups, prior_backup_dirs,
+ manifests, mwriter, opt);
+ continue;
+ }
+
+ /* Skip anything that's not a regular file. */
+ if (type != PGFILETYPE_REG)
+ {
+ if (type == PGFILETYPE_LNK)
+ pg_log_warning("skipping symbolic link \"%s\"", ifullpath);
+ else
+ pg_log_warning("skipping special file \"%s\"", ifullpath);
+ continue;
+ }
+
+ /*
+ * Skip the backup_label and backup_manifest files; they require
+ * special handling and are handled elsewhere.
+ */
+ if (relative_path == NULL &&
+ (strcmp(de->d_name, "backup_label") == 0 ||
+ strcmp(de->d_name, "backup_manifest") == 0))
+ continue;
+
+ /*
+ * If it's an incremental file, hand it off to the reconstruction
+ * code, which will figure out what to do.
+ */
+ if (strncmp(de->d_name, INCREMENTAL_PREFIX,
+ INCREMENTAL_PREFIX_LENGTH) == 0)
+ {
+ /* Output path should not include "INCREMENTAL." prefix. */
+ snprintf(ofullpath, MAXPGPATH, "%s/%s", ofulldir,
+ de->d_name + INCREMENTAL_PREFIX_LENGTH);
+
+
+ /* Manifest path likewise omits incremental prefix. */
+ snprintf(manifest_path, MAXPGPATH, "%s%s", manifest_prefix,
+ de->d_name + INCREMENTAL_PREFIX_LENGTH);
+
+ /* Reconstruction logic will do the rest. */
+ reconstruct_from_incremental_file(ifullpath, ofullpath,
+ relative_path,
+ de->d_name + INCREMENTAL_PREFIX_LENGTH,
+ n_prior_backups,
+ prior_backup_dirs,
+ manifests,
+ manifest_path,
+ checksum_type,
+ &checksum_length,
+ &checksum_payload,
+ opt->debug,
+ opt->dry_run);
+ }
+ else
+ {
+ /* Construct the path that the backup_manifest will use. */
+ snprintf(manifest_path, MAXPGPATH, "%s%s", manifest_prefix,
+ de->d_name);
+
+ /*
+ * It's not an incremental file, so we need to copy the entire
+ * file to the output directory.
+ *
+ * If a checksum of the required type already exists in the
+ * backup_manifest for the final input directory, we can save some
+ * work by reusing that checksum instead of computing a new one.
+ */
+ if (checksum_type != CHECKSUM_TYPE_NONE &&
+ latest_manifest != NULL)
+ {
+ manifest_file *mfile;
+
+ mfile = manifest_files_lookup(latest_manifest->files,
+ manifest_path);
+ if (mfile == NULL)
+ {
+ /*
+ * The directory is out of sync with the backup_manifest,
+ * so emit a warning.
+ */
+ pg_log_warning("\"%s/backup_manifest\" contains no entry for \"%s\"",
+ input_directory, manifest_path);
+ }
+ else if (mfile->checksum_type == checksum_type)
+ {
+ checksum_length = mfile->checksum_length;
+ checksum_payload = mfile->checksum_payload;
+ }
+ }
+
+ /*
+ * If we're reusing a checksum, then we don't need copy_file() to
+ * compute one for us, but otherwise, it needs to compute whatever
+ * type of checksum we need.
+ */
+ if (checksum_length != 0)
+ pg_checksum_init(&checksum_ctx, CHECKSUM_TYPE_NONE);
+ else
+ pg_checksum_init(&checksum_ctx, checksum_type);
+
+ /* Actually copy the file. */
+ snprintf(ofullpath, MAXPGPATH, "%s/%s", ofulldir, de->d_name);
+ copy_file(ifullpath, ofullpath, &checksum_ctx, opt->dry_run);
+
+ /*
+ * If copy_file() performed a checksum calculation for us, then
+ * save the results (except in dry-run mode, when there's no
+ * point).
+ */
+ if (checksum_ctx.type != CHECKSUM_TYPE_NONE && !opt->dry_run)
+ {
+ checksum_payload = pg_malloc(PG_CHECKSUM_MAX_LENGTH);
+ checksum_length = pg_checksum_final(&checksum_ctx,
+ checksum_payload);
+ }
+ }
+
+ /* Generate manifest entry, if needed. */
+ if (mwriter != NULL)
+ {
+ struct stat sb;
+
+ /*
+ * In order to generate a manifest entry, we need the file size
+ * and mtime. We have no way to know the correct mtime except to
+ * stat() the file, so just do that and get the size as well.
+ *
+ * If we didn't need the mtime here, we could try to obtain the
+ * file size from the reconstruction or file copy process above,
+ * although that is actually not convenient in all cases. If we
+ * write the file ourselves then clearly we can keep a count of
+ * bytes, but if we use something like CopyFile() then it's
+ * trickier. Since we have to stat() anyway to get the mtime,
+ * there's no point in worrying about it.
+ */
+ if (stat(ofullpath, &sb) < 0)
+ pg_fatal("could not stat file \"%s\": %m", ofullpath);
+
+ /* OK, now do the work. */
+ add_file_to_manifest(mwriter, manifest_path,
+ sb.st_size, sb.st_mtime,
+ checksum_type, checksum_length,
+ checksum_payload);
+ }
+
+ /* Avoid leaking memory. */
+ if (checksum_payload != NULL)
+ pfree(checksum_payload);
+ }
+
+ closedir(dir);
+}
+
+/*
+ * Read the version number from PG_VERSION and convert it to the usual server
+ * version number format. (e.g. If PG_VERSION contains "14\n" this function
+ * will return 140000)
+ */
+static int
+read_pg_version_file(char *directory)
+{
+ char filename[MAXPGPATH];
+ StringInfoData buf;
+ int fd;
+ int version;
+ char *ep;
+
+ /* Construct pathname. */
+ snprintf(filename, MAXPGPATH, "%s/PG_VERSION", directory);
+
+ /* Open file. */
+ if ((fd = open(filename, O_RDONLY, 0)) < 0)
+ pg_fatal("could not open file \"%s\": %m", filename);
+
+ /* Read into memory. Length limit of 128 should be more than generous. */
+ initStringInfo(&buf);
+ slurp_file(fd, filename, &buf, 128);
+
+ /* Close the file. */
+ if (close(fd) != 0)
+ pg_fatal("could not close \"%s\": %m", filename);
+
+ /* Convert to integer. */
+ errno = 0;
+ version = strtoul(buf.data, &ep, 10);
+ if (errno != 0 || *ep != '\n')
+ {
+ /*
+ * Incremental backup is not relevant to very old server versions that
+ * used multi-part version number (e.g. 9.6, or 8.4). So if we see
+ * what looks like the beginning of such a version number, just bail
+ * out.
+ */
+ if (version < 10 && *ep == '.')
+ pg_fatal("%s: server version too old\n", filename);
+ pg_fatal("%s: could not parse version number\n", filename);
+ }
+
+ /* Debugging output. */
+ pg_log_debug("read server version %d from \"%s\"", version, filename);
+
+ /* Release memory and return result. */
+ pfree(buf.data);
+ return version * 10000;
+}
+
+/*
+ * Add a directory to the list of output directories to clean up.
+ */
+static void
+remember_to_cleanup_directory(char *target_path, bool rmtopdir)
+{
+ cb_cleanup_dir *dir = pg_malloc(sizeof(cb_cleanup_dir));
+
+ dir->target_path = target_path;
+ dir->rmtopdir = rmtopdir;
+ dir->next = cleanup_dir_list;
+ cleanup_dir_list = dir;
+}
+
+/*
+ * Empty out the list of directories scheduled for cleanup a exit.
+ *
+ * We want to remove the output directories only on a failure, so call this
+ * function when we know that the operation has succeeded.
+ *
+ * Since we only expect this to be called when we're about to exit, we could
+ * just set cleanup_dir_list to NULL and be done with it, but we free the
+ * memory to be tidy.
+ */
+static void
+reset_directory_cleanup_list(void)
+{
+ while (cleanup_dir_list != NULL)
+ {
+ cb_cleanup_dir *dir = cleanup_dir_list;
+
+ cleanup_dir_list = cleanup_dir_list->next;
+ pfree(dir);
+ }
+}
+
+/*
+ * Scan the pg_tblspc directory of the final input backup to get a canonical
+ * list of what tablespaces are part of the backup.
+ *
+ * 'pathname' should be the path to the toplevel backup directory for the
+ * final backup in the backup chain.
+ */
+static cb_tablespace *
+scan_for_existing_tablespaces(char *pathname, cb_options *opt)
+{
+ char pg_tblspc[MAXPGPATH];
+ DIR *dir;
+ struct dirent *de;
+ cb_tablespace *tslist = NULL;
+
+ snprintf(pg_tblspc, MAXPGPATH, "%s/pg_tblspc", pathname);
+ pg_log_debug("scanning \"%s\"", pg_tblspc);
+
+ if ((dir = opendir(pg_tblspc)) == NULL)
+ pg_fatal("could not open directory \"%s\": %m", pathname);
+
+ while (errno = 0, (de = readdir(dir)) != NULL)
+ {
+ Oid oid;
+ char tblspcdir[MAXPGPATH];
+ char link_target[MAXPGPATH];
+ int link_length;
+ cb_tablespace *ts;
+ cb_tablespace *otherts;
+ PGFileType type;
+
+ /* Silently ignore "." and ".." entries. */
+ if (strcmp(de->d_name, ".") == 0 || strcmp(de->d_name, "..") == 0)
+ continue;
+
+ /* Construct full pathname. */
+ snprintf(tblspcdir, MAXPGPATH, "%s/%s", pg_tblspc, de->d_name);
+
+ /* Ignore any file name that doesn't look like a proper OID. */
+ if (!parse_oid(de->d_name, &oid))
+ {
+ pg_log_debug("skipping \"%s\" because the filename is not a legal tablespace OID",
+ tblspcdir);
+ continue;
+ }
+
+ /* Only symbolic links and directories are tablespaces. */
+ type = get_dirent_type(tblspcdir, de, false, PG_LOG_ERROR);
+ if (type == PGFILETYPE_ERROR)
+ exit(1);
+ if (type != PGFILETYPE_LNK && type != PGFILETYPE_DIR)
+ {
+ pg_log_debug("skipping \"%s\" because it is neither a symbolic link nor a directory",
+ tblspcdir);
+ continue;
+ }
+
+ /* Create a new tablespace object. */
+ ts = pg_malloc0(sizeof(cb_tablespace));
+ ts->oid = oid;
+
+ /*
+ * If it's a link, it's not an in-place tablespace. Otherwise, it must
+ * be a directory, and thus an in-place tablespace.
+ */
+ if (type == PGFILETYPE_LNK)
+ {
+ cb_tablespace_mapping *tsmap;
+
+ /* Read the link target. */
+ link_length = readlink(tblspcdir, link_target, sizeof(link_target));
+ if (link_length < 0)
+ pg_fatal("could not read symbolic link \"%s\": %m",
+ tblspcdir);
+ if (link_length >= sizeof(link_target))
+ pg_fatal("symbolic link \"%s\" is too long", tblspcdir);
+ link_target[link_length] = '\0';
+ if (!is_absolute_path(link_target))
+ pg_fatal("symbolic link \"%s\" is relative", tblspcdir);
+
+ /* Caonicalize the link target. */
+ canonicalize_path(link_target);
+
+ /*
+ * Find the corresponding tablespace mapping and copy the relevant
+ * details into the new tablespace entry.
+ */
+ for (tsmap = opt->tsmappings; tsmap != NULL; tsmap = tsmap->next)
+ {
+ if (strcmp(tsmap->old_dir, link_target) == 0)
+ {
+ strncpy(ts->old_dir, tsmap->old_dir, MAXPGPATH);
+ strncpy(ts->new_dir, tsmap->new_dir, MAXPGPATH);
+ ts->in_place = false;
+ break;
+ }
+ }
+
+ /* Every non-in-place tablespace must be mapped. */
+ if (tsmap == NULL)
+ pg_fatal("tablespace at \"%s\" has no tablespace mapping",
+ link_target);
+ }
+ else
+ {
+ /*
+ * For an in-place tablespace, there's no separate directory, so
+ * we just record the paths within the data directories.
+ */
+ snprintf(ts->old_dir, MAXPGPATH, "%s/%s", pg_tblspc, de->d_name);
+ snprintf(ts->new_dir, MAXPGPATH, "%s/pg_tblpc/%s", opt->output,
+ de->d_name);
+ ts->in_place = true;
+ }
+
+ /* Tablespaces should not share a directory. */
+ for (otherts = tslist; otherts != NULL; otherts = otherts->next)
+ if (strcmp(ts->new_dir, otherts->new_dir) == 0)
+ pg_fatal("tablespaces with OIDs %u and %u both point at \"%s\"",
+ otherts->oid, oid, ts->new_dir);
+
+ /* Add this tablespace to the list. */
+ ts->next = tslist;
+ tslist = ts;
+ }
+
+ return tslist;
+}
+
+/*
+ * Read a file into a StringInfo.
+ *
+ * fd is used for the actual file I/O, filename for error reporting purposes.
+ * A file longer than maxlen is a fatal error.
+ */
+static void
+slurp_file(int fd, char *filename, StringInfo buf, int maxlen)
+{
+ struct stat st;
+ ssize_t rb;
+
+ /* Check file size, and complain if it's too large. */
+ if (fstat(fd, &st) != 0)
+ pg_fatal("could not stat \"%s\": %m", filename);
+ if (st.st_size > maxlen)
+ pg_fatal("file \"%s\" is too large", filename);
+
+ /* Make sure we have enough space. */
+ enlargeStringInfo(buf, st.st_size);
+
+ /* Read the data. */
+ rb = read(fd, &buf->data[buf->len], st.st_size);
+
+ /*
+ * We don't expect any concurrent changes, so we should read exactly the
+ * expected number of bytes.
+ */
+ if (rb != st.st_size)
+ {
+ if (rb < 0)
+ pg_fatal("could not read file \"%s\": %m", filename);
+ else
+ pg_fatal("could not read file \"%s\": read only %d of %d bytes",
+ filename, (int) rb, (int) st.st_size);
+ }
+
+ /* Adjust buffer length for new data and restore trailing-\0 invariant */
+ buf->len += rb;
+ buf->data[buf->len] = '\0';
+}
diff --git a/src/bin/pg_combinebackup/reconstruct.c b/src/bin/pg_combinebackup/reconstruct.c
new file mode 100644
index 0000000000..7cd457aef3
--- /dev/null
+++ b/src/bin/pg_combinebackup/reconstruct.c
@@ -0,0 +1,681 @@
+/*-------------------------------------------------------------------------
+ *
+ * reconstruct.c
+ * Reconstruct full file from incremental file and backup chain.
+ *
+ * Copyright (c) 2017-2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_combinebackup/reconstruct.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include <unistd.h>
+
+#include "backup/basebackup_incremental.h"
+#include "common/logging.h"
+#include "common/file_perm.h"
+#include "copy_file.h"
+#include "lib/stringinfo.h"
+#include "reconstruct.h"
+#include "storage/block.h"
+
+/*
+ * An rfile stores the data that we need in order to be able to use some file
+ * on disk for reconstruction. For any given output file, we create one rfile
+ * per backup that we need to consult when we constructing that output file.
+ *
+ * If we find a full version of the file in the backup chain, then only
+ * filename and fd are initialized; the remaining fields are 0 or NULL.
+ * For an incremental file, header_length, num_blocks, relative_block_numbers,
+ * and truncation_block_length are also set.
+ *
+ * num_blocks_read and highest_offset_read always start out as 0.
+ */
+typedef struct rfile
+{
+ char *filename;
+ int fd;
+ size_t header_length;
+ unsigned num_blocks;
+ BlockNumber *relative_block_numbers;
+ unsigned truncation_block_length;
+ unsigned num_blocks_read;
+ off_t highest_offset_read;
+} rfile;
+
+static void debug_reconstruction(int n_source,
+ rfile **sources,
+ bool dry_run);
+static unsigned find_reconstructed_block_length(rfile *s);
+static rfile *make_incremental_rfile(char *filename);
+static rfile *make_rfile(char *filename, bool missing_ok);
+static void write_reconstructed_file(char *input_filename,
+ char *output_filename,
+ unsigned block_length,
+ rfile **sourcemap,
+ off_t *offsetmap,
+ pg_checksum_context *checksum_ctx,
+ bool debug,
+ bool dry_run);
+static void read_bytes(rfile *rf, void *buffer, unsigned length);
+
+/*
+ * Reconstruct a full file from an incremental file and a chain of prior
+ * backups.
+ *
+ * input_filename should be the path to the incremental file, and
+ * output_filename should be the path where the reconstructed file is to be
+ * written.
+ *
+ * relative_path should be the relative path to the directory containing this
+ * file. bare_file_name should be the name of the file within that directory,
+ * without "INCREMENTAL.".
+ *
+ * n_prior_backups is the number of prior backups, and prior_backup_dirs is
+ * an array of pathnames where those backups can be found.
+ */
+void
+reconstruct_from_incremental_file(char *input_filename,
+ char *output_filename,
+ char *relative_path,
+ char *bare_file_name,
+ int n_prior_backups,
+ char **prior_backup_dirs,
+ manifest_data **manifests,
+ char *manifest_path,
+ pg_checksum_type checksum_type,
+ int *checksum_length,
+ uint8 **checksum_payload,
+ bool debug,
+ bool dry_run)
+{
+ rfile **source;
+ rfile *latest_source = NULL;
+ rfile **sourcemap;
+ off_t *offsetmap;
+ unsigned block_length;
+ unsigned i;
+ unsigned sidx = n_prior_backups;
+ bool full_copy_possible = true;
+ int copy_source_index = -1;
+ rfile *copy_source = NULL;
+ pg_checksum_context checksum_ctx;
+
+ /*
+ * Every block must come either from the latest version of the file or
+ * from one of the prior backups.
+ */
+ source = pg_malloc0(sizeof(rfile *) * (1 + n_prior_backups));
+
+ /*
+ * Use the information from the latest incremental file to figure out how
+ * long the reconstructed file should be.
+ */
+ latest_source = make_incremental_rfile(input_filename);
+ source[n_prior_backups] = latest_source;
+ block_length = find_reconstructed_block_length(latest_source);
+
+ /*
+ * For each block in the output file, we need to know from which file we
+ * need to obtain it and at what offset in that file it's stored.
+ * sourcemap gives us the first of these things, and offsetmap the latter.
+ */
+ sourcemap = pg_malloc0(sizeof(rfile *) * block_length);
+ offsetmap = pg_malloc0(sizeof(off_t) * block_length);
+
+ /*
+ * Every block that is present in the newest incremental file should be
+ * sourced from that file. If it precedes the truncation_block_length,
+ * it's a block that we would otherwise have had to find in an older
+ * backup and thus reduces the number of blocks remaining to be found by
+ * one; otherwise, it's an extra block that needs to be included in the
+ * output but would not have needed to be found in an older backup if it
+ * had not been present.
+ */
+ for (i = 0; i < latest_source->num_blocks; ++i)
+ {
+ BlockNumber b = latest_source->relative_block_numbers[i];
+
+ Assert(b < block_length);
+ sourcemap[b] = latest_source;
+ offsetmap[b] = latest_source->header_length + (i * BLCKSZ);
+
+ /*
+ * A full copy of a file from an earlier backup is only possible if no
+ * blocks are needed from any later incremental file.
+ */
+ full_copy_possible = false;
+ }
+
+ while (1)
+ {
+ char source_filename[MAXPGPATH];
+ rfile *s;
+
+ /*
+ * Move to the next backup in the chain. If there are no more, then
+ * we're done.
+ */
+ if (sidx == 0)
+ break;
+ --sidx;
+
+ /*
+ * Look for the full file in the previous backup. If not found, then
+ * look for an incremental file instead.
+ */
+ snprintf(source_filename, MAXPGPATH, "%s/%s/%s",
+ prior_backup_dirs[sidx], relative_path, bare_file_name);
+ if ((s = make_rfile(source_filename, true)) == NULL)
+ {
+ snprintf(source_filename, MAXPGPATH, "%s/%s/INCREMENTAL.%s",
+ prior_backup_dirs[sidx], relative_path, bare_file_name);
+ s = make_incremental_rfile(source_filename);
+ }
+ source[sidx] = s;
+
+ /*
+ * If s->header_length == 0, then this is a full file; otherwise, it's
+ * an incremental file.
+ */
+ if (s->header_length != 0)
+ {
+ /*
+ * Since we found another incremental file, source all blocks from
+ * it that we need but don't yet have.
+ */
+ for (i = 0; i < s->num_blocks; ++i)
+ {
+ BlockNumber b = s->relative_block_numbers[i];
+
+ if (b < latest_source->truncation_block_length &&
+ sourcemap[b] == NULL)
+ {
+ sourcemap[b] = s;
+ offsetmap[b] = s->header_length + (i * BLCKSZ);
+
+ /*
+ * A full copy of a file from an earlier backup is only
+ * possible if no blocks are needed from any later
+ * incremental file.
+ */
+ full_copy_possible = false;
+ }
+ }
+ }
+ else
+ {
+ struct stat sb;
+ BlockNumber b;
+ BlockNumber blocklength;
+
+ /* We need to know the length of the file. */
+ if (fstat(s->fd, &sb) < 0)
+ pg_fatal("could not stat \"%s\": %m", s->filename);
+
+ /*
+ * Since we found a full file, source all blocks from it that
+ * exist in the file.
+ *
+ * Note that there may be blocks that don't exist either in this
+ * file or in any incremental file but that precede
+ * truncation_block_length. These are, presumably, zero-filled
+ * blocks that result from the server extending the file but
+ * taking no action on those blocks that generated any WAL.
+ *
+ * Sadly, we have no way of validating that this is really what
+ * happened, and neither does the server. From it's perspective,
+ * an unmodified block that contains data looks exactly the same
+ * as a zero-filled block that never had any data: either way,
+ * it's not mentioned in any WAL summary and the server has no
+ * reason to read it. From our perspective, all we know is that
+ * nobody had a reason to back up the block. That certainly means
+ * that the block didn't exist at the time of the full backup, but
+ * the supposition that it was all zeroes at the time of every
+ * later backup is one that we can't validate.
+ */
+ blocklength = sb.st_size / BLCKSZ;
+ for (b = 0; b < latest_source->truncation_block_length; ++b)
+ {
+ if (sourcemap[b] == NULL && b < blocklength)
+ {
+ sourcemap[b] = s;
+ offsetmap[b] = b * BLCKSZ;
+ }
+ }
+
+ /*
+ * If a full copy looks possible, check whether the resulting file
+ * should be exactly as long as the source file is. If so, a full
+ * copy is acceptable, otherwise not.
+ */
+ if (full_copy_possible)
+ {
+ uint64 expected_length;
+
+ expected_length =
+ (uint64) latest_source->truncation_block_length;
+ expected_length *= BLCKSZ;
+ if (expected_length == sb.st_size)
+ {
+ copy_source = s;
+ copy_source_index = sidx;
+ }
+ }
+ }
+ }
+
+ /*
+ * If a checksum of the required type already exists in the
+ * backup_manifest for the relevant input directory, we can save some work
+ * by reusing that checksum instead of computing a new one.
+ */
+ if (copy_source_index >= 0 && manifests[copy_source_index] != NULL &&
+ checksum_type != CHECKSUM_TYPE_NONE)
+ {
+ manifest_file *mfile;
+
+ mfile = manifest_files_lookup(manifests[copy_source_index]->files,
+ manifest_path);
+ if (mfile == NULL)
+ {
+ /*
+ * The directory is out of sync with the backup_manifest, so emit
+ * a warning.
+ */
+ pg_log_warning("\"%s/backup_manifest\" contains no entry for \"%s\"",
+ prior_backup_dirs[copy_source_index],
+ manifest_path);
+ }
+ else if (mfile->checksum_type == checksum_type)
+ {
+ *checksum_length = mfile->checksum_length;
+ *checksum_payload = pg_malloc(*checksum_length);
+ memcpy(*checksum_payload, mfile->checksum_payload,
+ *checksum_length);
+ checksum_type = CHECKSUM_TYPE_NONE;
+ }
+ }
+
+ /* Prepare for checksum calculation, if required. */
+ pg_checksum_init(&checksum_ctx, checksum_type);
+
+ /*
+ * If the full file can be created by copying a file from an older backup
+ * in the chain without needing to overwrite any blocks or truncate the
+ * result, then forget about performing reconstruction and just copy that
+ * file in its entirety.
+ *
+ * Otherwise, reconstruct.
+ */
+ if (copy_source != NULL)
+ copy_file(copy_source->filename, output_filename,
+ &checksum_ctx, dry_run);
+ else
+ {
+ write_reconstructed_file(input_filename, output_filename,
+ block_length, sourcemap, offsetmap,
+ &checksum_ctx, debug, dry_run);
+ debug_reconstruction(n_prior_backups + 1, source, dry_run);
+ }
+
+ /* Save results of checksum calculation. */
+ if (checksum_type != CHECKSUM_TYPE_NONE)
+ {
+ *checksum_payload = pg_malloc(PG_CHECKSUM_MAX_LENGTH);
+ *checksum_length = pg_checksum_final(&checksum_ctx,
+ *checksum_payload);
+ }
+
+ /*
+ * Close files and release memory.
+ */
+ for (i = 0; i <= n_prior_backups; ++i)
+ {
+ rfile *s = source[i];
+
+ if (s == NULL)
+ continue;
+ if (close(s->fd) != 0)
+ pg_fatal("could not close \"%s\": %m", s->filename);
+ if (s->relative_block_numbers != NULL)
+ pfree(s->relative_block_numbers);
+ pg_free(s->filename);
+ }
+ pfree(sourcemap);
+ pfree(offsetmap);
+ pfree(source);
+}
+
+/*
+ * Perform post-reconstruction logging and sanity checks.
+ */
+static void
+debug_reconstruction(int n_source, rfile **sources, bool dry_run)
+{
+ unsigned i;
+
+ for (i = 0; i < n_source; ++i)
+ {
+ rfile *s = sources[i];
+
+ /* Ignore source if not used. */
+ if (s == NULL)
+ continue;
+
+ /* If no data is needed from this file, we can ignore it. */
+ if (s->num_blocks_read == 0)
+ continue;
+
+ /* Debug logging. */
+ if (dry_run)
+ pg_log_debug("would have read %u blocks from \"%s\"",
+ s->num_blocks_read, s->filename);
+ else
+ pg_log_debug("read %u blocks from \"%s\"",
+ s->num_blocks_read, s->filename);
+
+ /*
+ * In dry-run mode, we don't actually try to read data from the file,
+ * but we do try to verify that the file is long enough that we could
+ * have read the data if we'd tried.
+ *
+ * If this fails, then it means that a non-dry-run attempt would fail,
+ * complaining of not being able to read the required bytes from the
+ * file.
+ */
+ if (dry_run)
+ {
+ struct stat sb;
+
+ if (fstat(s->fd, &sb) < 0)
+ pg_fatal("could not stat \"%s\": %m", s->filename);
+ if (sb.st_size < s->highest_offset_read)
+ pg_fatal("file \"%s\" is too short: expected %llu, found %llu",
+ s->filename,
+ (unsigned long long) s->highest_offset_read,
+ (unsigned long long) sb.st_size);
+ }
+ }
+}
+
+/*
+ * When we perform reconstruction using an incremental file, the output file
+ * should be at least as long as the truncation_block_length. Any blocks
+ * present in the incremental file increase the output length as far as is
+ * necessary to include those blocks.
+ */
+static unsigned
+find_reconstructed_block_length(rfile *s)
+{
+ unsigned block_length = s->truncation_block_length;
+ unsigned i;
+
+ for (i = 0; i < s->num_blocks; ++i)
+ if (s->relative_block_numbers[i] >= block_length)
+ block_length = s->relative_block_numbers[i] + 1;
+
+ return block_length;
+}
+
+/*
+ * Initialize an incremental rfile, reading the header so that we know which
+ * blocks it contains.
+ */
+static rfile *
+make_incremental_rfile(char *filename)
+{
+ rfile *rf;
+ unsigned magic;
+
+ rf = make_rfile(filename, false);
+
+ /* Read and validate magic number. */
+ read_bytes(rf, &magic, sizeof(magic));
+ if (magic != INCREMENTAL_MAGIC)
+ pg_fatal("file \"%s\" has bad incremental magic number (0x%x not 0x%x)",
+ filename, magic, INCREMENTAL_MAGIC);
+
+ /* Read block count. */
+ read_bytes(rf, &rf->num_blocks, sizeof(rf->num_blocks));
+ if (rf->num_blocks > RELSEG_SIZE)
+ pg_fatal("file \"%s\" has block count %u in excess of segment size %u",
+ filename, rf->num_blocks, RELSEG_SIZE);
+
+ /* Read truncation block length. */
+ read_bytes(rf, &rf->truncation_block_length,
+ sizeof(rf->truncation_block_length));
+ if (rf->truncation_block_length > RELSEG_SIZE)
+ pg_fatal("file \"%s\" has truncation block length %u in excess of segment size %u",
+ filename, rf->truncation_block_length, RELSEG_SIZE);
+
+ /* Read block numbers if there are any. */
+ if (rf->num_blocks > 0)
+ {
+ rf->relative_block_numbers =
+ pg_malloc0(sizeof(BlockNumber) * rf->num_blocks);
+ read_bytes(rf, rf->relative_block_numbers,
+ sizeof(BlockNumber) * rf->num_blocks);
+ }
+
+ /* Remember length of header. */
+ rf->header_length = sizeof(magic) + sizeof(rf->num_blocks) +
+ sizeof(rf->truncation_block_length) +
+ sizeof(BlockNumber) * rf->num_blocks;
+
+ return rf;
+}
+
+/*
+ * Allocate and perform basic initialization of an rfile.
+ */
+static rfile *
+make_rfile(char *filename, bool missing_ok)
+{
+ rfile *rf;
+
+ rf = pg_malloc0(sizeof(rfile));
+ rf->filename = pstrdup(filename);
+ if ((rf->fd = open(filename, O_RDONLY | PG_BINARY, 0)) < 0)
+ {
+ if (missing_ok && errno == ENOENT)
+ {
+ pg_free(rf);
+ return NULL;
+ }
+ pg_fatal("could not open file \"%s\": %m", filename);
+ }
+
+ return rf;
+}
+
+/*
+ * Read the indicated number of bytes from an rfile into the buffer.
+ */
+static void
+read_bytes(rfile *rf, void *buffer, unsigned length)
+{
+ unsigned rb = read(rf->fd, buffer, length);
+
+ if (rb != length)
+ {
+ if (rb < 0)
+ pg_fatal("could not read file \"%s\": %m", rf->filename);
+ else
+ pg_fatal("could not read file \"%s\": read only %d of %d bytes",
+ rf->filename, (int) rb, length);
+ }
+}
+
+/*
+ * Write out a reconstructed file.
+ */
+static void
+write_reconstructed_file(char *input_filename,
+ char *output_filename,
+ unsigned block_length,
+ rfile **sourcemap,
+ off_t *offsetmap,
+ pg_checksum_context *checksum_ctx,
+ bool debug,
+ bool dry_run)
+{
+ int wfd = -1;
+ unsigned i;
+ unsigned zero_blocks = 0;
+
+ /* Debugging output. */
+ if (debug)
+ {
+ StringInfoData debug_buf;
+ unsigned start_of_range = 0;
+ unsigned current_block = 0;
+
+ /* Basic information about the output file to be produced. */
+ if (dry_run)
+ pg_log_debug("would reconstruct \"%s\" (%u blocks, checksum %s)",
+ output_filename, block_length,
+ pg_checksum_type_name(checksum_ctx->type));
+ else
+ pg_log_debug("reconstructing \"%s\" (%u blocks, checksum %s)",
+ output_filename, block_length,
+ pg_checksum_type_name(checksum_ctx->type));
+
+ /* Print out the plan for reconstructing this file. */
+ initStringInfo(&debug_buf);
+ while (current_block < block_length)
+ {
+ rfile *s = sourcemap[current_block];
+
+ /* Extend range, if possible. */
+ if (current_block + 1 < block_length &&
+ s == sourcemap[current_block + 1])
+ {
+ ++current_block;
+ continue;
+ }
+
+ /* Add details about this range. */
+ if (s == NULL)
+ {
+ if (current_block == start_of_range)
+ appendStringInfo(&debug_buf, " %u:zero", current_block);
+ else
+ appendStringInfo(&debug_buf, " %u-%u:zero",
+ start_of_range, current_block);
+ }
+ else
+ {
+ if (current_block == start_of_range)
+ appendStringInfo(&debug_buf, " %u:%s@" UINT64_FORMAT,
+ current_block,
+ s == NULL ? "ZERO" : s->filename,
+ (uint64) offsetmap[current_block]);
+ else
+ appendStringInfo(&debug_buf, " %u-%u:%s@" UINT64_FORMAT,
+ start_of_range, current_block,
+ s == NULL ? "ZERO" : s->filename,
+ (uint64) offsetmap[current_block]);
+ }
+
+ /* Begin new range. */
+ start_of_range = ++current_block;
+
+ /* If the output is very long or we are done, dump it now. */
+ if (current_block == block_length || debug_buf.len > 1024)
+ {
+ pg_log_debug("reconstruction plan:%s", debug_buf.data);
+ resetStringInfo(&debug_buf);
+ }
+ }
+
+ /* Free memory. */
+ pfree(debug_buf.data);
+ }
+
+ /* Open the output file, except in dry_run mode. */
+ if (!dry_run &&
+ (wfd = open(output_filename,
+ O_RDWR | PG_BINARY | O_CREAT | O_EXCL,
+ pg_file_create_mode)) < 0)
+ pg_fatal("could not open file \"%s\": %m", output_filename);
+
+ /* Read and write the blocks as required. */
+ for (i = 0; i < block_length; ++i)
+ {
+ uint8 buffer[BLCKSZ];
+ rfile *s = sourcemap[i];
+ unsigned wb;
+
+ /* Update accounting information. */
+ if (s == NULL)
+ ++zero_blocks;
+ else
+ {
+ s->num_blocks_read++;
+ s->highest_offset_read = Max(s->highest_offset_read,
+ offsetmap[i] + BLCKSZ);
+ }
+
+ /* Skip the rest of this in dry-run mode. */
+ if (dry_run)
+ continue;
+
+ /* Read or zero-fill the block as appropriate. */
+ if (s == NULL)
+ {
+ /*
+ * New block not mentioned in the WAL summary. Should have been an
+ * uninitialized block, so just zero-fill it.
+ */
+ memset(buffer, 0, BLCKSZ);
+ }
+ else
+ {
+ unsigned rb;
+
+ /* Read the block from the correct source, except if dry-run. */
+ rb = pg_pread(s->fd, buffer, BLCKSZ, offsetmap[i]);
+ if (rb != BLCKSZ)
+ {
+ if (rb < 0)
+ pg_fatal("could not read file \"%s\": %m", s->filename);
+ else
+ pg_fatal("could not read file \"%s\": read only %d of %d bytes at offset %u",
+ s->filename, (int) rb, BLCKSZ,
+ (unsigned) offsetmap[i]);
+ }
+ }
+
+ /* Write out the block. */
+ if ((wb = write(wfd, buffer, BLCKSZ)) != BLCKSZ)
+ {
+ if (wb < 0)
+ pg_fatal("could not write file \"%s\": %m", output_filename);
+ else
+ pg_fatal("could not write file \"%s\": wrote only %d of %d bytes",
+ output_filename, (int) wb, BLCKSZ);
+ }
+
+ /* Update the checksum computation. */
+ if (pg_checksum_update(checksum_ctx, buffer, BLCKSZ) < 0)
+ pg_fatal("could not update checksum of file \"%s\"",
+ output_filename);
+ }
+
+ /* Debugging output. */
+ if (zero_blocks > 0)
+ {
+ if (dry_run)
+ pg_log_debug("would have zero-filled %u blocks", zero_blocks);
+ else
+ pg_log_debug("zero-filled %u blocks", zero_blocks);
+ }
+
+ /* Close the output file. */
+ if (wfd >= 0 && close(wfd) != 0)
+ pg_fatal("could not close \"%s\": %m", output_filename);
+}
diff --git a/src/bin/pg_combinebackup/reconstruct.h b/src/bin/pg_combinebackup/reconstruct.h
new file mode 100644
index 0000000000..d689aeb5c2
--- /dev/null
+++ b/src/bin/pg_combinebackup/reconstruct.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * reconstruct.h
+ * Reconstruct full file from incremental file and backup chain.
+ *
+ * Copyright (c) 2017-2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/bin/pg_combinebackup/reconstruct.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef RECONSTRUCT_H
+#define RECONSTRUCT_H
+
+#include "common/checksum_helper.h"
+#include "load_manifest.h"
+
+extern void reconstruct_from_incremental_file(char *input_filename,
+ char *output_filename,
+ char *relative_path,
+ char *bare_file_name,
+ int n_prior_backups,
+ char **prior_backup_dirs,
+ manifest_data **manifests,
+ char *manifest_path,
+ pg_checksum_type checksum_type,
+ int *checksum_length,
+ uint8 **checksum_payload,
+ bool debug,
+ bool dry_run);
+
+#endif
diff --git a/src/bin/pg_combinebackup/t/001_basic.pl b/src/bin/pg_combinebackup/t/001_basic.pl
new file mode 100644
index 0000000000..fb66075d1a
--- /dev/null
+++ b/src/bin/pg_combinebackup/t/001_basic.pl
@@ -0,0 +1,23 @@
+# Copyright (c) 2021-2023, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;
+
+program_help_ok('pg_combinebackup');
+program_version_ok('pg_combinebackup');
+program_options_handling_ok('pg_combinebackup');
+
+command_fails_like(
+ ['pg_combinebackup'],
+ qr/no input directories specified/,
+ 'input directories must be specified');
+command_fails_like(
+ [ 'pg_combinebackup', $tempdir ],
+ qr/no output directory specified/,
+ 'output directory must be specified');
+
+done_testing();
diff --git a/src/bin/pg_combinebackup/t/002_compare_backups.pl b/src/bin/pg_combinebackup/t/002_compare_backups.pl
new file mode 100644
index 0000000000..0b80455aff
--- /dev/null
+++ b/src/bin/pg_combinebackup/t/002_compare_backups.pl
@@ -0,0 +1,154 @@
+# Copyright (c) 2021-2023, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+use File::Compare;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Set up a new database instance.
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(has_archiving => 1, allows_streaming => 1);
+$primary->append_conf('postgresql.conf', 'summarize_wal = on');
+$primary->start;
+
+# Create some test tables, each containing one row of data, plus a whole
+# extra database.
+$primary->safe_psql('postgres', <<EOM);
+CREATE TABLE will_change (a int, b text);
+INSERT INTO will_change VALUES (1, 'initial test row');
+CREATE TABLE will_grow (a int, b text);
+INSERT INTO will_grow VALUES (1, 'initial test row');
+CREATE TABLE will_shrink (a int, b text);
+INSERT INTO will_shrink VALUES (1, 'initial test row');
+CREATE TABLE will_get_vacuumed (a int, b text);
+INSERT INTO will_get_vacuumed VALUES (1, 'initial test row');
+CREATE TABLE will_get_dropped (a int, b text);
+INSERT INTO will_get_dropped VALUES (1, 'initial test row');
+CREATE TABLE will_get_rewritten (a int, b text);
+INSERT INTO will_get_rewritten VALUES (1, 'initial test row');
+CREATE DATABASE db_will_get_dropped;
+EOM
+
+# Take a full backup.
+my $backup1path = $primary->backup_dir . '/backup1';
+$primary->command_ok(
+ [ 'pg_basebackup', '-D', $backup1path, '--no-sync', '-cfast' ],
+ "full backup");
+
+# Now make some database changes.
+$primary->safe_psql('postgres', <<EOM);
+UPDATE will_change SET b = 'modified value' WHERE a = 1;
+INSERT INTO will_grow
+ SELECT g, 'additional row' FROM generate_series(2, 5000) g;
+TRUNCATE will_shrink;
+VACUUM will_get_vacuumed;
+DROP TABLE will_get_dropped;
+CREATE TABLE newly_created (a int, b text);
+INSERT INTO newly_created VALUES (1, 'row for new table');
+VACUUM FULL will_get_rewritten;
+DROP DATABASE db_will_get_dropped;
+CREATE DATABASE db_newly_created;
+EOM
+
+# Take an incremental backup.
+my $backup2path = $primary->backup_dir . '/backup2';
+$primary->command_ok(
+ [ 'pg_basebackup', '-D', $backup2path, '--no-sync', '-cfast',
+ '--incremental', $backup1path . '/backup_manifest' ],
+ "incremental backup");
+
+# Find an LSN to which either backup can be recovered.
+my $lsn = $primary->safe_psql('postgres', "SELECT pg_current_wal_lsn();");
+
+# Make sure that the WAL segment containing that LSN has been archived.
+# PostgreSQL won't issue two consecutive XLOG_SWITCH records, and the backup
+# just issued one, so call txid_current() to generate some WAL activity
+# before calling pg_switch_wal().
+$primary->safe_psql('postgres', 'SELECT txid_current();');
+$primary->safe_psql('postgres', 'SELECT pg_switch_wal()');
+
+# Now wait for the LSN we chose above to be archived.
+my $archive_wait_query =
+ "SELECT pg_walfile_name('$lsn') <= last_archived_wal FROM pg_stat_archiver;";
+$primary->poll_query_until('postgres', $archive_wait_query)
+ or die "Timed out while waiting for WAL segment to be archived";
+
+# Perform PITR from the full backup. Disable archive_mode so that the archive
+# doesn't find out about the new timeline; that way, the later PITR below will
+# choose the same timeline.
+my $pitr1 = PostgreSQL::Test::Cluster->new('pitr1');
+$pitr1->init_from_backup($primary, 'backup1',
+ standby => 1, has_restoring => 1);
+$pitr1->append_conf('postgresql.conf', qq{
+recovery_target_lsn = '$lsn'
+recovery_target_action = 'promote'
+archive_mode = 'off'
+});
+$pitr1->start();
+
+# Perform PITR to the same LSN from the incremental backup. Use the same
+# basic configuration as before.
+my $pitr2 = PostgreSQL::Test::Cluster->new('pitr2');
+$pitr2->init_from_backup($primary, 'backup2',
+ standby => 1, has_restoring => 1,
+ combine_with_prior => [ 'backup1' ]);
+$pitr2->append_conf('postgresql.conf', qq{
+recovery_target_lsn = '$lsn'
+recovery_target_action = 'promote'
+archive_mode = 'off'
+});
+$pitr2->start();
+
+# Wait until both servers exit recovery.
+$pitr1->poll_query_until('postgres',
+ "SELECT NOT pg_is_in_recovery();")
+ or die "Timed out while waiting apply to reach LSN $lsn";
+$pitr2->poll_query_until('postgres',
+ "SELECT NOT pg_is_in_recovery();")
+ or die "Timed out while waiting apply to reach LSN $lsn";
+
+# Perform a logical dump of each server, and check that they match.
+# It would be much nicer if we could physically compare the data files, but
+# that doesn't really work. The contents of the page hole aren't guaranteed to
+# be identical, and there can be other discrepancies as well. To make this work
+# we'd need the equivalent of each AM's rm_mask functon written or at least
+# callable from Perl, and that doesn't seem practical.
+#
+# NB: We're just using the primary's backup directory for scratch space here.
+# This could equally well be any other directory we wanted to pick.
+my $backupdir = $primary->backup_dir;
+my $dump1 = $backupdir . '/pitr1.dump';
+my $dump2 = $backupdir . '/pitr2.dump';
+$pitr1->command_ok([
+ 'pg_dumpall', '-f', $dump1, '--no-sync', '--no-unlogged-table-data',
+ '-d', $pitr1->connstr('postgres'),
+ ],
+ 'dump from PITR 1');
+$pitr1->command_ok([
+ 'pg_dumpall', '-f', $dump2, '--no-sync', '--no-unlogged-table-data',
+ '-d', $pitr1->connstr('postgres'),
+ ],
+ 'dump from PITR 2');
+
+# Compare the two dumps, there should be no differences.
+my $compare_res = compare($dump1, $dump2);
+note($dump1);
+note($dump2);
+is($compare_res, 0, "dumps are identical");
+
+# Provide more context if the dumps do not match.
+if ($compare_res != 0)
+{
+ my ($stdout, $stderr) =
+ run_command([ 'diff', '-u', $dump1, $dump2 ]);
+ print "=== diff of $dump1 and $dump2\n";
+ print "=== stdout ===\n";
+ print $stdout;
+ print "=== stderr ===\n";
+ print $stderr;
+ print "=== EOF ===\n";
+}
+
+done_testing();
diff --git a/src/bin/pg_combinebackup/t/003_timeline.pl b/src/bin/pg_combinebackup/t/003_timeline.pl
new file mode 100644
index 0000000000..bc053ca5e8
--- /dev/null
+++ b/src/bin/pg_combinebackup/t/003_timeline.pl
@@ -0,0 +1,90 @@
+# Copyright (c) 2021-2023, PostgreSQL Global Development Group
+#
+# This test aims to validate that restoring an incremental backup works
+# properly even when the reference backup is on a different timeline.
+
+use strict;
+use warnings;
+use File::Compare;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Set up a new database instance.
+my $node1 = PostgreSQL::Test::Cluster->new('node1');
+$node1->init(has_archiving => 1, allows_streaming => 1);
+$node1->append_conf('postgresql.conf', 'summarize_wal = on');
+$node1->start;
+
+# Create a table and insert a test row into it.
+$node1->safe_psql('postgres', <<EOM);
+CREATE TABLE mytable (a int, b text);
+INSERT INTO mytable VALUES (1, 'aardvark');
+EOM
+
+# Take a full backup.
+my $backup1path = $node1->backup_dir . '/backup1';
+$node1->command_ok(
+ [ 'pg_basebackup', '-D', $backup1path, '--no-sync', '-cfast' ],
+ "full backup from node1");
+
+# Insert a second row on the original node.
+$node1->safe_psql('postgres', <<EOM);
+INSERT INTO mytable VALUES (2, 'beetle');
+EOM
+
+# Now take an incremental backup.
+my $backup2path = $node1->backup_dir . '/backup2';
+$node1->command_ok(
+ [ 'pg_basebackup', '-D', $backup2path, '--no-sync', '-cfast',
+ '--incremental', $backup1path . '/backup_manifest' ],
+ "incremental backup from node1");
+
+# Restore the incremental backup and use it to create a new node.
+my $node2 = PostgreSQL::Test::Cluster->new('node2');
+$node2->init_from_backup($node1, 'backup2',
+ combine_with_prior => [ 'backup1' ]);
+$node2->start();
+
+# Insert rows on both nodes.
+$node1->safe_psql('postgres', <<EOM);
+INSERT INTO mytable VALUES (3, 'crab');
+EOM
+$node2->safe_psql('postgres', <<EOM);
+INSERT INTO mytable VALUES (4, 'dingo');
+EOM
+
+# Take another incremental backup, from node2, based on backup2 from node1.
+my $backup3path = $node1->backup_dir . '/backup3';
+$node2->command_ok(
+ [ 'pg_basebackup', '-D', $backup3path, '--no-sync', '-cfast',
+ '--incremental', $backup2path . '/backup_manifest' ],
+ "incremental backup from node2");
+
+# Restore the incremental backup and use it to create a new node.
+my $node3 = PostgreSQL::Test::Cluster->new('node3');
+$node3->init_from_backup($node1, 'backup3',
+ combine_with_prior => [ 'backup1', 'backup2' ]);
+$node3->start();
+
+# Let's insert one more row.
+$node3->safe_psql('postgres', <<EOM);
+INSERT INTO mytable VALUES (5, 'elephant');
+EOM
+
+# Now check that we have the expected rows.
+my $result = $node3->safe_psql('postgres', <<EOM);
+select string_agg(a::text, ':'), string_agg(b, ':') from mytable;
+EOM
+is($result, '1:2:4:5|aardvark:beetle:dingo:elephant');
+
+# Let's also verify all the backups.
+for my $backup_name (qw(backup1 backup2 backup3))
+{
+ $node1->command_ok(
+ [ 'pg_verifybackup', $node1->backup_dir . '/' . $backup_name ],
+ "verify backup $backup_name");
+}
+
+# OK, that's all.
+done_testing();
diff --git a/src/bin/pg_combinebackup/t/004_manifest.pl b/src/bin/pg_combinebackup/t/004_manifest.pl
new file mode 100644
index 0000000000..37de61ac06
--- /dev/null
+++ b/src/bin/pg_combinebackup/t/004_manifest.pl
@@ -0,0 +1,75 @@
+# Copyright (c) 2021-2023, PostgreSQL Global Development Group
+#
+# This test aims to validate that pg_combinebackup works in the degenerate
+# case where it is invoked on a single full backup and that it can produce
+# a new, valid manifest when it does. Secondarily, it checks that
+# pg_combinebackup does not produce a manifest when run with --no-manifest.
+
+use strict;
+use warnings;
+use File::Compare;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Set up a new database instance.
+my $node = PostgreSQL::Test::Cluster->new('node');
+$node->init(has_archiving => 1, allows_streaming => 1);
+$node->start;
+
+# Take a full backup.
+my $original_backup_path = $node->backup_dir . '/original';
+$node->command_ok(
+ [ 'pg_basebackup', '-D', $original_backup_path, '--no-sync', '-cfast' ],
+ "full backup");
+
+# Verify the full backup.
+$node->command_ok([ 'pg_verifybackup', $original_backup_path ],
+ "verify original backup");
+
+# Process the backup with pg_combinebackup using various manifest options.
+sub combine_and_test_one_backup
+{
+ my ($backup_name, $failure_pattern, @extra_options) = @_;
+ my $revised_backup_path = $node->backup_dir . '/' . $backup_name;
+ $node->command_ok(
+ [ 'pg_combinebackup', $original_backup_path, '-o', $revised_backup_path,
+ '--no-sync', @extra_options ],
+ "pg_combinebackup with @extra_options");
+ if (defined $failure_pattern)
+ {
+ $node->command_fails_like(
+ [ 'pg_verifybackup', $revised_backup_path ],
+ $failure_pattern,
+ "unable to verify backup $backup_name");
+ }
+ else
+ {
+ $node->command_ok(
+ [ 'pg_verifybackup', $revised_backup_path ],
+ "verify backup $backup_name");
+ }
+}
+combine_and_test_one_backup('nomanifest',
+ qr/could not open file.*backup_manifest/, '--no-manifest');
+combine_and_test_one_backup('csum_none',
+ undef, '--manifest-checksums=NONE');
+combine_and_test_one_backup('csum_sha224',
+ undef, '--manifest-checksums=SHA224');
+
+# Verify that SHA224 is mentioned in the SHA224 manifest lots of times.
+my $sha224_manifest =
+ slurp_file($node->backup_dir . '/csum_sha224/backup_manifest');
+my $sha224_count = (() = $sha224_manifest =~ /SHA224/mig);
+cmp_ok($sha224_count,
+ '>', 100, "SHA224 is mentioned many times in SHA224 manifest");
+
+# Verify that SHA224 is mentioned in the SHA224 manifest lots of times.
+my $nocsum_manifest =
+ slurp_file($node->backup_dir . '/csum_none/backup_manifest');
+my $nocsum_count = (() = $nocsum_manifest =~ /Checksum-Algorithm/mig);
+is($nocsum_count, 0,
+ "Checksum_Algorithm is not mentioned in no-checksum manifest");
+
+# OK, that's all.
+done_testing();
diff --git a/src/bin/pg_combinebackup/t/005_integrity.pl b/src/bin/pg_combinebackup/t/005_integrity.pl
new file mode 100644
index 0000000000..b1f63a43e0
--- /dev/null
+++ b/src/bin/pg_combinebackup/t/005_integrity.pl
@@ -0,0 +1,125 @@
+# Copyright (c) 2021-2023, PostgreSQL Global Development Group
+#
+# This test aims to validate that an incremental backup can be combined
+# with a valid prior backup and that it cannot be combined with an invalid
+# prior backup.
+
+use strict;
+use warnings;
+use File::Compare;
+use File::Path qw(rmtree);
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Set up a new database instance.
+my $node1 = PostgreSQL::Test::Cluster->new('node1');
+$node1->init(has_archiving => 1, allows_streaming => 1);
+$node1->append_conf('postgresql.conf', 'summarize_wal = on');
+$node1->start;
+
+# Set up another new database instance. We don't want to use the cached
+# INITDB_TEMPLATE for this, because we want it to be a separate cluster
+# with a different system ID.
+my $node2;
+{
+ local $ENV{'INITDB_TEMPLATE'} = undef;
+
+ $node2 = PostgreSQL::Test::Cluster->new('node2');
+ $node2->init(has_archiving => 1, allows_streaming => 1);
+ $node2->append_conf('postgresql.conf', 'summarize_wal = on');
+ $node2->start;
+}
+
+# Take a full backup from node1.
+my $backup1path = $node1->backup_dir . '/backup1';
+$node1->command_ok(
+ [ 'pg_basebackup', '-D', $backup1path, '--no-sync', '-cfast' ],
+ "full backup from node1");
+
+# Now take an incremental backup.
+my $backup2path = $node1->backup_dir . '/backup2';
+$node1->command_ok(
+ [ 'pg_basebackup', '-D', $backup2path, '--no-sync', '-cfast',
+ '--incremental', $backup1path . '/backup_manifest' ],
+ "incremental backup from node1");
+
+# Now take another incremental backup.
+my $backup3path = $node1->backup_dir . '/backup3';
+$node1->command_ok(
+ [ 'pg_basebackup', '-D', $backup3path, '--no-sync', '-cfast',
+ '--incremental', $backup2path . '/backup_manifest' ],
+ "another incremental backup from node1");
+
+# Take a full backup from node2.
+my $backupother1path = $node1->backup_dir . '/backupother1';
+$node2->command_ok(
+ [ 'pg_basebackup', '-D', $backupother1path, '--no-sync', '-cfast' ],
+ "full backup from node2");
+
+# Take an incremental backup from node2.
+my $backupother2path = $node1->backup_dir . '/backupother2';
+$node2->command_ok(
+ [ 'pg_basebackup', '-D', $backupother2path, '--no-sync', '-cfast',
+ '--incremental', $backupother1path . '/backup_manifest' ],
+ "incremental backup from node2");
+
+# Result directory.
+my $resultpath = $node1->backup_dir . '/result';
+
+# Can't combine 2 full backups.
+$node1->command_fails_like(
+ [ 'pg_combinebackup', $backup1path, $backup1path, '-o', $resultpath ],
+ qr/is a full backup, but only the first backup should be a full backup/,
+ "can't combine full backups");
+
+# Can't combine 2 incremental backups.
+$node1->command_fails_like(
+ [ 'pg_combinebackup', $backup2path, $backup2path, '-o', $resultpath ],
+ qr/is an incremental backup, but the first backup should be a full backup/,
+ "can't combine full backups");
+
+# Can't combine full backup with an incremental backup from a different system.
+$node1->command_fails_like(
+ [ 'pg_combinebackup', $backup1path, $backupother2path, '-o', $resultpath ],
+ qr/expected system identifier.*but found/,
+ "can't combine backups from different nodes");
+
+# Can't omit a required backup.
+$node1->command_fails_like(
+ [ 'pg_combinebackup', $backup1path, $backup3path, '-o', $resultpath ],
+ qr/starts at LSN.*but expected/,
+ "can't omit a required backup");
+
+# Can't combine backups in the wrong order.
+$node1->command_fails_like(
+ [ 'pg_combinebackup', $backup1path, $backup3path, $backup2path, '-o', $resultpath ],
+ qr/starts at LSN.*but expected/,
+ "can't combine backups in the wrong order");
+
+# Can combine 3 backups that match up properly.
+$node1->command_ok(
+ [ 'pg_combinebackup', $backup1path, $backup2path, $backup3path, '-o', $resultpath ],
+ "can combine 3 matching backups");
+rmtree($resultpath);
+
+# Can combine full backup with first incremental.
+my $synthetic12path = $node1->backup_dir . '/synthetic12';
+$node1->command_ok(
+ [ 'pg_combinebackup', $backup1path, $backup2path, '-o', $synthetic12path ],
+ "can combine 2 matching backups");
+
+# Can combine result of previous step with second incremental.
+$node1->command_ok(
+ [ 'pg_combinebackup', $synthetic12path, $backup3path, '-o', $resultpath ],
+ "can combine synthetic backup with later incremental");
+rmtree($resultpath);
+
+# Can't combine result of 1+2 with 2.
+$node1->command_fails_like(
+ [ 'pg_combinebackup', $synthetic12path, $backup2path, '-o', $resultpath ],
+ qr/starts at LSN.*but expected/,
+ "can't combine synthetic backup with included incremental");
+
+# OK, that's all.
+done_testing();
diff --git a/src/bin/pg_combinebackup/write_manifest.c b/src/bin/pg_combinebackup/write_manifest.c
new file mode 100644
index 0000000000..82160134d8
--- /dev/null
+++ b/src/bin/pg_combinebackup/write_manifest.c
@@ -0,0 +1,293 @@
+/*-------------------------------------------------------------------------
+ *
+ * Write a new backup manifest.
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_combinebackup/write_manifest.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include <fcntl.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "common/checksum_helper.h"
+#include "common/file_perm.h"
+#include "common/logging.h"
+#include "lib/stringinfo.h"
+#include "load_manifest.h"
+#include "mb/pg_wchar.h"
+#include "write_manifest.h"
+
+struct manifest_writer
+{
+ char pathname[MAXPGPATH];
+ int fd;
+ StringInfoData buf;
+ bool first_file;
+ bool still_checksumming;
+ pg_checksum_context manifest_ctx;
+};
+
+static void escape_json(StringInfo buf, const char *str);
+static void flush_manifest(manifest_writer *mwriter);
+static size_t hex_encode(const uint8 *src, size_t len, char *dst);
+
+/*
+ * Create a new backup manifest writer.
+ *
+ * The backup manifest will be written into a file named backup_manifest
+ * in the specified directory.
+ */
+manifest_writer *
+create_manifest_writer(char *directory)
+{
+ manifest_writer *mwriter = pg_malloc(sizeof(manifest_writer));
+
+ snprintf(mwriter->pathname, MAXPGPATH, "%s/backup_manifest", directory);
+ mwriter->fd = -1;
+ initStringInfo(&mwriter->buf);
+ mwriter->first_file = true;
+ mwriter->still_checksumming = true;
+ pg_checksum_init(&mwriter->manifest_ctx, CHECKSUM_TYPE_SHA256);
+
+ appendStringInfo(&mwriter->buf,
+ "{ \"PostgreSQL-Backup-Manifest-Version\": 1,\n"
+ "\"Files\": [");
+
+ return mwriter;
+}
+
+/*
+ * Add an entry for a file to a backup manifest.
+ *
+ * This is very similar to the backend's AddFileToBackupManifest, but
+ * various adjustments are required due to frontend/backend differences
+ * and other details.
+ */
+void
+add_file_to_manifest(manifest_writer *mwriter, const char *manifest_path,
+ size_t size, pg_time_t mtime,
+ pg_checksum_type checksum_type,
+ int checksum_length,
+ uint8 *checksum_payload)
+{
+ int pathlen = strlen(manifest_path);
+
+ if (mwriter->first_file)
+ {
+ appendStringInfoChar(&mwriter->buf, '\n');
+ mwriter->first_file = false;
+ }
+ else
+ appendStringInfoString(&mwriter->buf, ",\n");
+
+ if (pg_encoding_verifymbstr(PG_UTF8, manifest_path, pathlen) == pathlen)
+ {
+ appendStringInfoString(&mwriter->buf, "{ \"Path\": ");
+ escape_json(&mwriter->buf, manifest_path);
+ appendStringInfoString(&mwriter->buf, ", ");
+ }
+ else
+ {
+ appendStringInfoString(&mwriter->buf, "{ \"Encoded-Path\": \"");
+ enlargeStringInfo(&mwriter->buf, 2 * pathlen);
+ mwriter->buf.len += hex_encode((const uint8 *) manifest_path, pathlen,
+ &mwriter->buf.data[mwriter->buf.len]);
+ appendStringInfoString(&mwriter->buf, "\", ");
+ }
+
+ appendStringInfo(&mwriter->buf, "\"Size\": %zu, ", size);
+
+ appendStringInfoString(&mwriter->buf, "\"Last-Modified\": \"");
+ enlargeStringInfo(&mwriter->buf, 128);
+ mwriter->buf.len += strftime(&mwriter->buf.data[mwriter->buf.len], 128,
+ "%Y-%m-%d %H:%M:%S %Z",
+ gmtime(&mtime));
+ appendStringInfoChar(&mwriter->buf, '"');
+
+ if (mwriter->buf.len > 128 * 1024)
+ flush_manifest(mwriter);
+
+ if (checksum_length > 0)
+ {
+ appendStringInfo(&mwriter->buf,
+ ", \"Checksum-Algorithm\": \"%s\", \"Checksum\": \"",
+ pg_checksum_type_name(checksum_type));
+
+ enlargeStringInfo(&mwriter->buf, 2 * checksum_length);
+ mwriter->buf.len += hex_encode(checksum_payload, checksum_length,
+ &mwriter->buf.data[mwriter->buf.len]);
+
+ appendStringInfoChar(&mwriter->buf, '"');
+ }
+
+ appendStringInfoString(&mwriter->buf, " }");
+
+ if (mwriter->buf.len > 128 * 1024)
+ flush_manifest(mwriter);
+}
+
+/*
+ * Finalize the backup_manifest.
+ */
+void
+finalize_manifest(manifest_writer *mwriter,
+ manifest_wal_range *first_wal_range)
+{
+ uint8 checksumbuf[PG_SHA256_DIGEST_LENGTH];
+ int len;
+ manifest_wal_range *wal_range;
+
+ /* Terminate the list of files. */
+ appendStringInfoString(&mwriter->buf, "\n],\n");
+
+ /* Start a list of LSN ranges. */
+ appendStringInfoString(&mwriter->buf, "\"WAL-Ranges\": [\n");
+
+ for (wal_range = first_wal_range; wal_range != NULL;
+ wal_range = wal_range->next)
+ appendStringInfo(&mwriter->buf,
+ "%s{ \"Timeline\": %u, \"Start-LSN\": \"%X/%X\", \"End-LSN\": \"%X/%X\" }",
+ wal_range == first_wal_range ? "" : ",\n",
+ wal_range->tli,
+ LSN_FORMAT_ARGS(wal_range->start_lsn),
+ LSN_FORMAT_ARGS(wal_range->end_lsn));
+
+ /* Terminate the list of WAL ranges. */
+ appendStringInfoString(&mwriter->buf, "\n],\n");
+
+ /* Flush accumulated data and update checksum calculation. */
+ flush_manifest(mwriter);
+
+ /* Checksum only includes data up to this point. */
+ mwriter->still_checksumming = false;
+
+ /* Compute and insert manifest checksum. */
+ appendStringInfoString(&mwriter->buf, "\"Manifest-Checksum\": \"");
+ enlargeStringInfo(&mwriter->buf, 2 * PG_SHA256_DIGEST_STRING_LENGTH);
+ len = pg_checksum_final(&mwriter->manifest_ctx, checksumbuf);
+ Assert(len == PG_SHA256_DIGEST_LENGTH);
+ mwriter->buf.len +=
+ hex_encode(checksumbuf, len, &mwriter->buf.data[mwriter->buf.len]);
+ appendStringInfoString(&mwriter->buf, "\"}\n");
+
+ /* Flush the last manifest checksum itself. */
+ flush_manifest(mwriter);
+
+ /* Close the file. */
+ if (close(mwriter->fd) != 0)
+ pg_fatal("could not close \"%s\": %m", mwriter->pathname);
+ mwriter->fd = -1;
+}
+
+/*
+ * Produce a JSON string literal, properly escaping characters in the text.
+ */
+static void
+escape_json(StringInfo buf, const char *str)
+{
+ const char *p;
+
+ appendStringInfoCharMacro(buf, '"');
+ for (p = str; *p; p++)
+ {
+ switch (*p)
+ {
+ case '\b':
+ appendStringInfoString(buf, "\\b");
+ break;
+ case '\f':
+ appendStringInfoString(buf, "\\f");
+ break;
+ case '\n':
+ appendStringInfoString(buf, "\\n");
+ break;
+ case '\r':
+ appendStringInfoString(buf, "\\r");
+ break;
+ case '\t':
+ appendStringInfoString(buf, "\\t");
+ break;
+ case '"':
+ appendStringInfoString(buf, "\\\"");
+ break;
+ case '\\':
+ appendStringInfoString(buf, "\\\\");
+ break;
+ default:
+ if ((unsigned char) *p < ' ')
+ appendStringInfo(buf, "\\u%04x", (int) *p);
+ else
+ appendStringInfoCharMacro(buf, *p);
+ break;
+ }
+ }
+ appendStringInfoCharMacro(buf, '"');
+}
+
+/*
+ * Flush whatever portion of the backup manifest we have generated and
+ * buffered in memory out to a file on disk.
+ *
+ * The first call to this function will create the file. After that, we
+ * keep it open and just append more data.
+ */
+static void
+flush_manifest(manifest_writer *mwriter)
+{
+ char pathname[MAXPGPATH];
+
+ if (mwriter->fd == -1 &&
+ (mwriter->fd = open(mwriter->pathname,
+ O_WRONLY | O_CREAT | O_EXCL | PG_BINARY,
+ pg_file_create_mode)) < 0)
+ pg_fatal("could not open file \"%s\": %m", mwriter->pathname);
+
+ if (mwriter->buf.len > 0)
+ {
+ ssize_t wb;
+
+ wb = write(mwriter->fd, mwriter->buf.data, mwriter->buf.len);
+ if (wb != mwriter->buf.len)
+ {
+ if (wb < 0)
+ pg_fatal("could not write \"%s\": %m", mwriter->pathname);
+ else
+ pg_fatal("could not write file \"%s\": wrote only %d of %d bytes",
+ pathname, (int) wb, mwriter->buf.len);
+ }
+
+ if (mwriter->still_checksumming)
+ pg_checksum_update(&mwriter->manifest_ctx,
+ (uint8 *) mwriter->buf.data,
+ mwriter->buf.len);
+ resetStringInfo(&mwriter->buf);
+ }
+}
+
+/*
+ * Encode bytes using two hexademical digits for each one.
+ */
+static size_t
+hex_encode(const uint8 *src, size_t len, char *dst)
+{
+ const uint8 *end = src + len;
+
+ while (src < end)
+ {
+ unsigned n1 = (*src >> 4) & 0xF;
+ unsigned n2 = *src & 0xF;
+
+ *dst++ = n1 < 10 ? '0' + n1 : 'a' + n1 - 10;
+ *dst++ = n2 < 10 ? '0' + n2 : 'a' + n2 - 10;
+ ++src;
+ }
+
+ return len * 2;
+}
diff --git a/src/bin/pg_combinebackup/write_manifest.h b/src/bin/pg_combinebackup/write_manifest.h
new file mode 100644
index 0000000000..8fd7fe02c8
--- /dev/null
+++ b/src/bin/pg_combinebackup/write_manifest.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * Write a new backup manifest.
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_combinebackup/write_manifest.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef WRITE_MANIFEST_H
+#define WRITE_MANIFEST_H
+
+#include "common/checksum_helper.h"
+#include "pgtime.h"
+
+struct manifest_wal_range;
+
+struct manifest_writer;
+typedef struct manifest_writer manifest_writer;
+
+extern manifest_writer *create_manifest_writer(char *directory);
+extern void add_file_to_manifest(manifest_writer *mwriter,
+ const char *manifest_path,
+ size_t size, pg_time_t mtime,
+ pg_checksum_type checksum_type,
+ int checksum_length,
+ uint8 *checksum_payload);
+extern void finalize_manifest(manifest_writer *mwriter,
+ struct manifest_wal_range *first_wal_range);
+
+#endif /* WRITE_MANIFEST_H */
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c
index 3ae3fc06df..5407f51a4e 100644
--- a/src/bin/pg_resetwal/pg_resetwal.c
+++ b/src/bin/pg_resetwal/pg_resetwal.c
@@ -85,6 +85,7 @@ static void RewriteControlFile(void);
static void FindEndOfXLOG(void);
static void KillExistingXLOG(void);
static void KillExistingArchiveStatus(void);
+static void KillExistingWALSummaries(void);
static void WriteEmptyXLOG(void);
static void usage(void);
@@ -493,6 +494,7 @@ main(int argc, char *argv[])
RewriteControlFile();
KillExistingXLOG();
KillExistingArchiveStatus();
+ KillExistingWALSummaries();
WriteEmptyXLOG();
printf(_("Write-ahead log reset\n"));
@@ -1034,6 +1036,40 @@ KillExistingArchiveStatus(void)
pg_fatal("could not close directory \"%s\": %m", ARCHSTATDIR);
}
+/*
+ * Remove existing WAL summary files
+ */
+static void
+KillExistingWALSummaries(void)
+{
+#define WALSUMMARYDIR XLOGDIR "/summaries"
+#define WALSUMMARY_NHEXCHARS 40
+
+ DIR *xldir;
+ struct dirent *xlde;
+ char path[MAXPGPATH + sizeof(WALSUMMARYDIR)];
+
+ xldir = opendir(WALSUMMARYDIR);
+ if (xldir == NULL)
+ pg_fatal("could not open directory \"%s\": %m", WALSUMMARYDIR);
+
+ while (errno = 0, (xlde = readdir(xldir)) != NULL)
+ {
+ if (strspn(xlde->d_name, "0123456789ABCDEF") == WALSUMMARY_NHEXCHARS &&
+ strcmp(xlde->d_name + WALSUMMARY_NHEXCHARS, ".summary") == 0)
+ {
+ snprintf(path, sizeof(path), "%s/%s", WALSUMMARYDIR, xlde->d_name);
+ if (unlink(path) < 0)
+ pg_fatal("could not delete file \"%s\": %m", path);
+ }
+ }
+
+ if (errno)
+ pg_fatal("could not read directory \"%s\": %m", WALSUMMARYDIR);
+
+ if (closedir(xldir))
+ pg_fatal("could not close directory \"%s\": %m", ARCHSTATDIR);
+}
/*
* Write an empty XLOG file, containing only the checkpoint record
diff --git a/src/include/access/xlogbackup.h b/src/include/access/xlogbackup.h
index 1611358137..90e04cad56 100644
--- a/src/include/access/xlogbackup.h
+++ b/src/include/access/xlogbackup.h
@@ -28,6 +28,8 @@ typedef struct BackupState
XLogRecPtr checkpointloc; /* last checkpoint location */
pg_time_t starttime; /* backup start time */
bool started_in_recovery; /* backup started in recovery? */
+ XLogRecPtr istartpoint; /* incremental based on backup at this LSN */
+ TimeLineID istarttli; /* incremental based on backup on this TLI */
/* Fields saved at the end of backup */
XLogRecPtr stoppoint; /* backup stop WAL location */
diff --git a/src/include/backup/basebackup.h b/src/include/backup/basebackup.h
index 1432d9c206..345bd22534 100644
--- a/src/include/backup/basebackup.h
+++ b/src/include/backup/basebackup.h
@@ -34,6 +34,9 @@ typedef struct
int64 size; /* total size as sent; -1 if not known */
} tablespaceinfo;
-extern void SendBaseBackup(BaseBackupCmd *cmd);
+struct IncrementalBackupInfo;
+
+extern void SendBaseBackup(BaseBackupCmd *cmd,
+ struct IncrementalBackupInfo *ib);
#endif /* _BASEBACKUP_H */
diff --git a/src/include/backup/basebackup_incremental.h b/src/include/backup/basebackup_incremental.h
new file mode 100644
index 0000000000..c300235a2f
--- /dev/null
+++ b/src/include/backup/basebackup_incremental.h
@@ -0,0 +1,56 @@
+/*-------------------------------------------------------------------------
+ *
+ * basebackup_incremental.h
+ * API for incremental backup support
+ *
+ * Portions Copyright (c) 2010-2022, PostgreSQL Global Development Group
+ *
+ * src/include/backup/basebackup_incremental.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef BASEBACKUP_INCREMENTAL_H
+#define BASEBACKUP_INCREMENTAL_H
+
+#include "access/xlogbackup.h"
+#include "common/relpath.h"
+#include "storage/block.h"
+#include "utils/palloc.h"
+
+#define INCREMENTAL_MAGIC 0xd3ae1f0d
+
+typedef enum
+{
+ BACK_UP_FILE_FULLY,
+ BACK_UP_FILE_INCREMENTALLY,
+ DO_NOT_BACK_UP_FILE
+} FileBackupMethod;
+
+struct IncrementalBackupInfo;
+typedef struct IncrementalBackupInfo IncrementalBackupInfo;
+
+extern IncrementalBackupInfo *CreateIncrementalBackupInfo(MemoryContext);
+
+extern void AppendIncrementalManifestData(IncrementalBackupInfo *ib,
+ const char *data,
+ int len);
+extern void FinalizeIncrementalManifest(IncrementalBackupInfo *ib);
+
+extern void PrepareForIncrementalBackup(IncrementalBackupInfo *ib,
+ BackupState *backup_state);
+
+extern char *GetIncrementalFilePath(Oid dboid, Oid spcoid,
+ RelFileNumber relfilenumber,
+ ForkNumber forknum, unsigned segno);
+extern FileBackupMethod GetFileBackupMethod(IncrementalBackupInfo *ib,
+ char *path,
+ Oid dboid, Oid spcoid,
+ RelFileNumber relfilenumber,
+ ForkNumber forknum,
+ unsigned segno, size_t size,
+ unsigned *num_blocks_required,
+ BlockNumber *relative_block_numbers,
+ unsigned *truncation_block_length);
+extern size_t GetIncrementalFileSize(unsigned num_blocks_required);
+
+#endif
diff --git a/src/include/nodes/replnodes.h b/src/include/nodes/replnodes.h
index 5142a08729..c98961c329 100644
--- a/src/include/nodes/replnodes.h
+++ b/src/include/nodes/replnodes.h
@@ -108,4 +108,13 @@ typedef struct TimeLineHistoryCmd
TimeLineID timeline;
} TimeLineHistoryCmd;
+/* ----------------------
+ * UPLOAD_MANIFEST command
+ * ----------------------
+ */
+typedef struct UploadManifestCmd
+{
+ NodeTag type;
+} UploadManifestCmd;
+
#endif /* REPLNODES_H */
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index c3d46c7c70..b711d60fc4 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -779,6 +779,10 @@ a tar-format backup, pass the name of the tar program to use in the
keyword parameter tar_program. Note that tablespace tar files aren't
handled here.
+To restore from an incremental backup, pass the parameter combine_with_prior
+as a reference to an array of prior backup names with which this backup
+is to be combined using pg_combinebackup.
+
Streaming replication can be enabled on this node by passing the keyword
parameter has_streaming => 1. This is disabled by default.
@@ -816,7 +820,22 @@ sub init_from_backup
mkdir $self->archive_dir;
my $data_path = $self->data_dir;
- if (defined $params{tar_program})
+ if (defined $params{combine_with_prior})
+ {
+ my @prior_backups = @{$params{combine_with_prior}};
+ my @prior_backup_path;
+
+ for my $prior_backup_name (@prior_backups)
+ {
+ push @prior_backup_path,
+ $root_node->backup_dir . '/' . $prior_backup_name;
+ }
+
+ local %ENV = $self->_get_env();
+ PostgreSQL::Test::Utils::system_or_bail('pg_combinebackup',
+ @prior_backup_path, $backup_path, '-o', $data_path);
+ }
+ elsif (defined $params{tar_program})
{
mkdir($data_path);
PostgreSQL::Test::Utils::system_or_bail($params{tar_program}, 'xf',
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index bf81b91e20..95ae399cae 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -4009,3 +4009,15 @@ SummarizerReadLocalXLogPrivate
WalSummarizerData
WalSummaryFile
WalSummaryIO
+FileBackupMethod
+IncrementalBackupInfo
+UploadManifestCmd
+backup_file_entry
+backup_wal_range
+cb_cleanup_dir
+cb_options
+cb_tablespace
+cb_tablespace_mapping
+manifest_data
+manifest_writer
+rfile
--
2.37.1 (Apple Git-137.1)
[application/octet-stream] v8-0002-Move-src-bin-pg_verifybackup-parse_manifest.c-int.patch (4.3K, ../../CA+TgmoYiPn7GE_GFvGf+150XOt-Pkdm8h4rzy42jYCRrceUWhg@mail.gmail.com/7-v8-0002-Move-src-bin-pg_verifybackup-parse_manifest.c-int.patch)
download | inline diff:
From 571b192c1655d1e74d0a5d3b0b3915cc69e7463f Mon Sep 17 00:00:00 2001
From: Robert Haas <[email protected]>
Date: Tue, 3 Oct 2023 13:32:45 -0400
Subject: [PATCH v8 2/6] Move src/bin/pg_verifybackup/parse_manifest.c into
src/common.
This makes it possible for the code to be easily reused by other
client-side tools, and/or by the server.
---
src/bin/pg_verifybackup/Makefile | 1 -
src/bin/pg_verifybackup/meson.build | 1 -
src/bin/pg_verifybackup/pg_verifybackup.c | 2 +-
src/common/Makefile | 1 +
src/common/meson.build | 1 +
src/{bin/pg_verifybackup => common}/parse_manifest.c | 4 ++--
src/{bin/pg_verifybackup => include/common}/parse_manifest.h | 2 +-
7 files changed, 6 insertions(+), 6 deletions(-)
rename src/{bin/pg_verifybackup => common}/parse_manifest.c (99%)
rename src/{bin/pg_verifybackup => include/common}/parse_manifest.h (97%)
diff --git a/src/bin/pg_verifybackup/Makefile b/src/bin/pg_verifybackup/Makefile
index c96323faa9..7c045f142e 100644
--- a/src/bin/pg_verifybackup/Makefile
+++ b/src/bin/pg_verifybackup/Makefile
@@ -21,7 +21,6 @@ LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)
OBJS = \
$(WIN32RES) \
- parse_manifest.o \
pg_verifybackup.o
all: pg_verifybackup
diff --git a/src/bin/pg_verifybackup/meson.build b/src/bin/pg_verifybackup/meson.build
index 9369da1bc6..58f780d1a6 100644
--- a/src/bin/pg_verifybackup/meson.build
+++ b/src/bin/pg_verifybackup/meson.build
@@ -1,7 +1,6 @@
# Copyright (c) 2022-2023, PostgreSQL Global Development Group
pg_verifybackup_sources = files(
- 'parse_manifest.c',
'pg_verifybackup.c'
)
diff --git a/src/bin/pg_verifybackup/pg_verifybackup.c b/src/bin/pg_verifybackup/pg_verifybackup.c
index 059836f0e6..ce423a03d4 100644
--- a/src/bin/pg_verifybackup/pg_verifybackup.c
+++ b/src/bin/pg_verifybackup/pg_verifybackup.c
@@ -20,9 +20,9 @@
#include "common/hashfn.h"
#include "common/logging.h"
+#include "common/parse_manifest.h"
#include "fe_utils/simple_list.h"
#include "getopt_long.h"
-#include "parse_manifest.h"
#include "pgtime.h"
/*
diff --git a/src/common/Makefile b/src/common/Makefile
index ce4535d7fe..1092dc63df 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -66,6 +66,7 @@ OBJS_COMMON = \
kwlookup.o \
link-canary.o \
md5_common.o \
+ parse_manifest.o \
percentrepl.o \
pg_get_line.o \
pg_lzcompress.o \
diff --git a/src/common/meson.build b/src/common/meson.build
index 8be145c0fb..d52dd12bc9 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -18,6 +18,7 @@ common_sources = files(
'kwlookup.c',
'link-canary.c',
'md5_common.c',
+ 'parse_manifest.c',
'percentrepl.c',
'pg_get_line.c',
'pg_lzcompress.c',
diff --git a/src/bin/pg_verifybackup/parse_manifest.c b/src/common/parse_manifest.c
similarity index 99%
rename from src/bin/pg_verifybackup/parse_manifest.c
rename to src/common/parse_manifest.c
index bf0227c668..ee6f74e5d5 100644
--- a/src/bin/pg_verifybackup/parse_manifest.c
+++ b/src/common/parse_manifest.c
@@ -6,15 +6,15 @@
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
- * src/bin/pg_verifybackup/parse_manifest.c
+ * src/common/parse_manifest.c
*
*-------------------------------------------------------------------------
*/
#include "postgres_fe.h"
-#include "parse_manifest.h"
#include "common/jsonapi.h"
+#include "common/parse_manifest.h"
/*
* Semantic states for JSON manifest parsing.
diff --git a/src/bin/pg_verifybackup/parse_manifest.h b/src/include/common/parse_manifest.h
similarity index 97%
rename from src/bin/pg_verifybackup/parse_manifest.h
rename to src/include/common/parse_manifest.h
index 7387a917a2..7b24c5d785 100644
--- a/src/bin/pg_verifybackup/parse_manifest.h
+++ b/src/include/common/parse_manifest.h
@@ -6,7 +6,7 @@
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
- * src/bin/pg_verifybackup/parse_manifest.h
+ * src/include/common/parse_manifest.h
*
*-------------------------------------------------------------------------
*/
--
2.37.1 (Apple Git-137.1)
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: trying again to get incremental backup
2023-10-24 16:08 Re: trying again to get incremental backup Robert Haas <[email protected]>
2023-10-30 14:45 ` Re: trying again to get incremental backup Robert Haas <[email protected]>
2023-10-30 18:46 ` Re: trying again to get incremental backup Andres Freund <[email protected]>
2023-11-06 20:36 ` Re: trying again to get incremental backup Robert Haas <[email protected]>
@ 2023-11-10 11:27 ` Dilip Kumar <[email protected]>
2023-11-13 19:22 ` Re: trying again to get incremental backup Robert Haas <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Dilip Kumar @ 2023-11-10 11:27 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Andres Freund <[email protected]>; Peter Eisentraut <[email protected]>; pgsql-hackers
On Tue, Nov 7, 2023 at 2:06 AM Robert Haas <[email protected]> wrote:
>
> On Mon, Oct 30, 2023 at 2:46 PM Andres Freund <[email protected]> wrote:
> > After playing with this for a while, I don't see a reason for wal_summarize_mb
> > from a memory usage POV at least.
>
> Here's v8. Changes:
Review comments, based on what I reviewed so far.
- I think 0001 looks good improvement irrespective of the patch series.
- review 0003
1.
+ be enabled either on a primary or on a standby. WAL summarization can
+ cannot be enabled when <varname>wal_level</varname> is set to
+ <literal>minimal</literal>.
Grammatical error
"WAL summarization can cannot" -> WAL summarization cannot
2.
+ <varlistentry id="guc-wal-summarize-keep-time"
xreflabel="wal_summarize_keep_time">
+ <term><varname>wal_summarize_keep_time</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>wal_summarize_keep_time</varname>
configuration parameter</primary>
+ </indexterm>
I feel the name of the guy should be either wal_summarizer_keep_time
or wal_summaries_keep_time, I mean either we should refer to the
summarizer process or to the way summaries files.
3.
+XLogGetOldestSegno(TimeLineID tli)
+{
+
+ /* Ignore files that are not XLOG segments */
+ if (!IsXLogFileName(xlde->d_name))
+ continue;
+
+ /* Parse filename to get TLI and segno. */
+ XLogFromFileName(xlde->d_name, &file_tli, &file_segno,
+ wal_segment_size);
+
+ /* Ignore anything that's not from the TLI of interest. */
+ if (tli != file_tli)
+ continue;
+
+ /* If it's the oldest so far, update oldest_segno. */
Some of the single-line comments end with a full stop whereas others
do not, so better to be consistent.
4.
+ * If start_lsn != InvalidXLogRecPtr, only summaries that end before the
+ * indicated LSN will be included.
+ *
+ * If end_lsn != InvalidXLogRecPtr, only summaries that start before the
+ * indicated LSN will be included.
+ *
+ * The intent is that you can call GetWalSummaries(tli, start_lsn, end_lsn)
+ * to get all WAL summaries on the indicated timeline that overlap the
+ * specified LSN range.
+ */
+List *
+GetWalSummaries(TimeLineID tli, XLogRecPtr start_lsn, XLogRecPtr end_lsn)
Instead of "If start_lsn != InvalidXLogRecPtr, only summaries that end
before the" it should be "If start_lsn != InvalidXLogRecPtr, only
summaries that end after the" because only if the summary files are
Ending after the start_lsn then it will have some overlapping and we
need to return them if ending before start lsn then those files are
not overlapping at all, right?
5.
In FilterWalSummaries() header also the comment is wrong same as for
GetWalSummaries() function.
6.
+ * If the whole range of LSNs is covered, returns true, otherwise false.
+ * If false is returned, *missing_lsn is set either to InvalidXLogRecPtr
+ * if there are no WAL summary files in the input list, or to the first LSN
+ * in the range that is not covered by a WAL summary file in the input list.
+ */
+bool
+WalSummariesAreComplete(List *wslist, XLogRecPtr start_lsn,
I did not see the usage of this function, but I think if the whole
range is not covered why not keep the behavior uniform w.r.t. what we
set for '*missing_lsn', I mean suppose there is no file then
missing_lsn is the start_lsn because a very first LSN is missing.
7.
+ nbytes = FileRead(io->file, data, length, io->filepos,
+ WAIT_EVENT_WAL_SUMMARY_READ);
+ if (nbytes < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write file \"%s\": %m",
+ FilePathName(io->file))));
/could not write file/ could not read file
8.
+/*
+ * Comparator to sort a List of WalSummaryFile objects by start_lsn.
+ */
+static int
+ListComparatorForWalSummaryFiles(const ListCell *a, const ListCell *b)
+{
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: trying again to get incremental backup
2023-10-24 16:08 Re: trying again to get incremental backup Robert Haas <[email protected]>
2023-10-30 14:45 ` Re: trying again to get incremental backup Robert Haas <[email protected]>
2023-10-30 18:46 ` Re: trying again to get incremental backup Andres Freund <[email protected]>
2023-11-06 20:36 ` Re: trying again to get incremental backup Robert Haas <[email protected]>
2023-11-10 11:27 ` Re: trying again to get incremental backup Dilip Kumar <[email protected]>
@ 2023-11-13 19:22 ` Robert Haas <[email protected]>
2023-11-14 04:28 ` Re: trying again to get incremental backup Dilip Kumar <[email protected]>
0 siblings, 1 reply; 77+ messages in thread
From: Robert Haas @ 2023-11-13 19:22 UTC (permalink / raw)
To: Dilip Kumar <[email protected]>; +Cc: Andres Freund <[email protected]>; Peter Eisentraut <[email protected]>; pgsql-hackers
On Fri, Nov 10, 2023 at 6:27 AM Dilip Kumar <[email protected]> wrote:
> - I think 0001 looks good improvement irrespective of the patch series.
OK, perhaps that can be independently committed, then, if nobody objects.
Thanks for the review; I've fixed a bunch of things that you
mentioned. I'll just comment on the ones I haven't yet done anything
about below.
> 2.
> + <varlistentry id="guc-wal-summarize-keep-time"
> xreflabel="wal_summarize_keep_time">
> + <term><varname>wal_summarize_keep_time</varname> (<type>boolean</type>)
> + <indexterm>
> + <primary><varname>wal_summarize_keep_time</varname>
> configuration parameter</primary>
> + </indexterm>
>
> I feel the name of the guy should be either wal_summarizer_keep_time
> or wal_summaries_keep_time, I mean either we should refer to the
> summarizer process or to the way summaries files.
How about wal_summary_keep_time?
> 6.
> + * If the whole range of LSNs is covered, returns true, otherwise false.
> + * If false is returned, *missing_lsn is set either to InvalidXLogRecPtr
> + * if there are no WAL summary files in the input list, or to the first LSN
> + * in the range that is not covered by a WAL summary file in the input list.
> + */
> +bool
> +WalSummariesAreComplete(List *wslist, XLogRecPtr start_lsn,
>
> I did not see the usage of this function, but I think if the whole
> range is not covered why not keep the behavior uniform w.r.t. what we
> set for '*missing_lsn', I mean suppose there is no file then
> missing_lsn is the start_lsn because a very first LSN is missing.
It's used later in the patch series. I think the way that I have it
makes for a more understandable error message.
> 8.
> +/*
> + * Comparator to sort a List of WalSummaryFile objects by start_lsn.
> + */
> +static int
> +ListComparatorForWalSummaryFiles(const ListCell *a, const ListCell *b)
> +{
I'm not sure what needs fixing here.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 77+ messages in thread
* Re: trying again to get incremental backup
2023-10-24 16:08 Re: trying again to get incremental backup Robert Haas <[email protected]>
2023-10-30 14:45 ` Re: trying again to get incremental backup Robert Haas <[email protected]>
2023-10-30 18:46 ` Re: trying again to get incremental backup Andres Freund <[email protected]>
2023-11-06 20:36 ` Re: trying again to get incremental backup Robert Haas <[email protected]>
2023-11-10 11:27 ` Re: trying again to get incremental backup Dilip Kumar <[email protected]>
2023-11-13 19:22 ` Re: trying again to get incremental backup Robert Haas <[email protected]>
@ 2023-11-14 04:28 ` Dilip Kumar <[email protected]>
0 siblings, 0 replies; 77+ messages in thread
From: Dilip Kumar @ 2023-11-14 04:28 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Andres Freund <[email protected]>; Peter Eisentraut <[email protected]>; pgsql-hackers
On Tue, Nov 14, 2023 at 12:52 AM Robert Haas <[email protected]> wrote:
>
> On Fri, Nov 10, 2023 at 6:27 AM Dilip Kumar <[email protected]> wrote:
> > - I think 0001 looks good improvement irrespective of the patch series.
>
> OK, perhaps that can be independently committed, then, if nobody objects.
>
> Thanks for the review; I've fixed a bunch of things that you
> mentioned. I'll just comment on the ones I haven't yet done anything
> about below.
>
> > 2.
> > + <varlistentry id="guc-wal-summarize-keep-time"
> > xreflabel="wal_summarize_keep_time">
> > + <term><varname>wal_summarize_keep_time</varname> (<type>boolean</type>)
> > + <indexterm>
> > + <primary><varname>wal_summarize_keep_time</varname>
> > configuration parameter</primary>
> > + </indexterm>
> >
> > I feel the name of the guy should be either wal_summarizer_keep_time
> > or wal_summaries_keep_time, I mean either we should refer to the
> > summarizer process or to the way summaries files.
>
> How about wal_summary_keep_time?
Yes, that looks perfect to me.
> > 6.
> > + * If the whole range of LSNs is covered, returns true, otherwise false.
> > + * If false is returned, *missing_lsn is set either to InvalidXLogRecPtr
> > + * if there are no WAL summary files in the input list, or to the first LSN
> > + * in the range that is not covered by a WAL summary file in the input list.
> > + */
> > +bool
> > +WalSummariesAreComplete(List *wslist, XLogRecPtr start_lsn,
> >
> > I did not see the usage of this function, but I think if the whole
> > range is not covered why not keep the behavior uniform w.r.t. what we
> > set for '*missing_lsn', I mean suppose there is no file then
> > missing_lsn is the start_lsn because a very first LSN is missing.
>
> It's used later in the patch series. I think the way that I have it
> makes for a more understandable error message.
Okay
> > 8.
> > +/*
> > + * Comparator to sort a List of WalSummaryFile objects by start_lsn.
> > + */
> > +static int
> > +ListComparatorForWalSummaryFiles(const ListCell *a, const ListCell *b)
> > +{
>
> I'm not sure what needs fixing here.
I think I copy-pasted it by mistake, just ignore it.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 77+ messages in thread
end of thread, other threads:[~2024-01-31 17:39 UTC | newest]
Thread overview: 77+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-12-19 03:43 [PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test. Kyotaro Horiguchi <[email protected]>
2018-12-19 03:43 [PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test. Kyotaro Horiguchi <[email protected]>
2018-12-19 03:43 [PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test. Kyotaro Horiguchi <[email protected]>
2018-12-19 03:43 [PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test. Kyotaro Horiguchi <[email protected]>
2018-12-19 03:43 [PATCH v19 1/3] Add primary_slot_name to init_from_backup in TAP test. Kyotaro Horiguchi <[email protected]>
2019-12-20 01:09 [PATCH v30 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v32 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v37 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v29 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v39 3/8] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v31 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v24 04/15] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v25 04/15] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v30 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v23 04/15] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v26 04/10] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v27 3/9] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v29 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v39 3/8] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v38 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v30 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v38 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v38 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v37 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v27 3/9] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v28 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v29 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v29 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v24 04/15] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v37 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v30 03/11] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2019-12-20 01:09 [PATCH v39 3/8] Allow to prolong life span of transition tables until transaction end Yugo Nagata <[email protected]>
2022-03-23 18:50 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2022-03-29 12:34 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2022-08-02 18:56 ` Re: speed up a logical replica setup Jacob Champion <[email protected]>
2024-01-08 07:05 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-18 09:18 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-18 09:37 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-22 06:55 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-22 09:08 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:22 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-23 23:44 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 09:27 ` Re: speed up a logical replica setup Peter Eisentraut <[email protected]>
2024-01-25 10:16 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 13:03 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
2024-01-18 09:43 ` Re: speed up a logical replica setup Amit Kapila <[email protected]>
2024-01-19 08:48 ` Re: speed up a logical replica setup Shubham Khanna <[email protected]>
2024-01-25 05:54 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 07:06 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-22 09:30 ` Re: speed up a logical replica setup Shlok Kyal <[email protected]>
2024-01-23 02:11 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-24 06:11 ` Re: speed up a logical replica setup Shubham Khanna <[email protected]>
2024-01-24 11:43 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-25 09:05 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-26 00:28 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 05:51 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-24 01:29 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-25 23:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-26 07:55 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-29 01:10 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-29 09:57 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-30 02:01 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-30 09:26 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 12:52 ` RE: speed up a logical replica setup Hayato Kuroda (Fujitsu) <[email protected]>
2024-01-31 14:25 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
2024-01-31 14:34 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-31 14:55 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
2024-01-31 15:37 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2024-01-31 17:39 ` Re: speed up a logical replica setup Fabrízio de Royes Mello <[email protected]>
2024-01-23 23:58 ` Re: speed up a logical replica setup Euler Taveira <[email protected]>
2023-10-24 16:08 Re: trying again to get incremental backup Robert Haas <[email protected]>
2023-10-30 14:45 ` Re: trying again to get incremental backup Robert Haas <[email protected]>
2023-10-30 18:46 ` Re: trying again to get incremental backup Andres Freund <[email protected]>
2023-11-06 20:36 ` Re: trying again to get incremental backup Robert Haas <[email protected]>
2023-11-10 11:27 ` Re: trying again to get incremental backup Dilip Kumar <[email protected]>
2023-11-13 19:22 ` Re: trying again to get incremental backup Robert Haas <[email protected]>
2023-11-14 04:28 ` Re: trying again to get incremental backup Dilip Kumar <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox