public inbox for [email protected]help / color / mirror / Atom feed
[PATCH 1/5] bootstrap: convert Typ to a List* 4+ messages / 4 participants [nested] [flat]
* [PATCH 1/5] bootstrap: convert Typ to a List* @ 2020-11-20 02:48 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 4+ messages in thread From: Justin Pryzby @ 2020-11-20 02:48 UTC (permalink / raw) --- src/backend/bootstrap/bootstrap.c | 69 ++++++++++++++----------------- 1 file changed, 31 insertions(+), 38 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..18eb62ca47 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -159,7 +159,7 @@ struct typmap FormData_pg_type am_typ; }; -static struct typmap **Typ = NULL; +static List *Typ = NIL; /* List of struct typmap* */ static struct typmap *Ap = NULL; static Datum values[MAXATTR]; /* current row's attribute values */ @@ -597,7 +597,7 @@ boot_openrel(char *relname) * pg_type must be filled before any OPEN command is executed, hence we * can now populate the Typ array if we haven't yet. */ - if (Typ == NULL) + if (Typ == NIL) populate_typ_array(); if (boot_reldesc != NULL) @@ -688,7 +688,7 @@ DefineAttr(char *name, char *type, int attnum, int nullness) typeoid = gettype(type); - if (Typ != NULL) + if (Typ != NIL) { attrtypes[attnum]->atttypid = Ap->am_oid; attrtypes[attnum]->attlen = Ap->am_typ.typlen; @@ -877,36 +877,25 @@ populate_typ_array(void) Relation rel; TableScanDesc scan; HeapTuple tup; - int nalloc; - int i; - - Assert(Typ == NULL); - nalloc = 512; - Typ = (struct typmap **) - MemoryContextAlloc(TopMemoryContext, nalloc * sizeof(struct typmap *)); + Assert(Typ == NIL); rel = table_open(TypeRelationId, NoLock); scan = table_beginscan_catalog(rel, 0, NULL); - i = 0; while ((tup = heap_getnext(scan, ForwardScanDirection)) != NULL) { Form_pg_type typForm = (Form_pg_type) GETSTRUCT(tup); + struct typmap *newtyp; + MemoryContext old; - /* make sure there will be room for a trailing NULL pointer */ - if (i >= nalloc - 1) - { - nalloc *= 2; - Typ = (struct typmap **) - repalloc(Typ, nalloc * sizeof(struct typmap *)); - } - Typ[i] = (struct typmap *) - MemoryContextAlloc(TopMemoryContext, sizeof(struct typmap)); - Typ[i]->am_oid = typForm->oid; - memcpy(&(Typ[i]->am_typ), typForm, sizeof(Typ[i]->am_typ)); - i++; + old = MemoryContextSwitchTo(TopMemoryContext); + newtyp = (struct typmap *) palloc(sizeof(struct typmap)); + Typ = lappend(Typ, newtyp); + MemoryContextSwitchTo(old); + + newtyp->am_oid = typForm->oid; + memcpy(&newtyp->am_typ, typForm, sizeof(newtyp->am_typ)); } - Typ[i] = NULL; /* Fill trailing NULL pointer */ table_endscan(scan); table_close(rel, NoLock); } @@ -925,16 +914,17 @@ populate_typ_array(void) static Oid gettype(char *type) { - if (Typ != NULL) + if (Typ != NIL) { - struct typmap **app; + ListCell *lc; - for (app = Typ; *app != NULL; app++) + foreach (lc, Typ) { - if (strncmp(NameStr((*app)->am_typ.typname), type, NAMEDATALEN) == 0) + struct typmap *app = lfirst(lc); + if (strncmp(NameStr(app->am_typ.typname), type, NAMEDATALEN) == 0) { - Ap = *app; - return (*app)->am_oid; + Ap = app; + return app->am_oid; } } } @@ -980,14 +970,17 @@ boot_get_type_io_data(Oid typid, if (Typ != NULL) { /* We have the boot-time contents of pg_type, so use it */ - struct typmap **app; - struct typmap *ap; - - app = Typ; - while (*app && (*app)->am_oid != typid) - ++app; - ap = *app; - if (ap == NULL) + struct typmap *ap = NULL; + ListCell *lc; + + foreach (lc, Typ) + { + ap = lfirst(lc); + if (ap->am_oid == typid) + break; + } + + if (!ap || ap->am_oid != typid) elog(ERROR, "type OID %u not found in Typ list", typid); *typlen = ap->am_typ.typlen; -- 2.26.2 --------------614DDB87AFFED893713AC0E9 Content-Type: text/x-patch; charset=UTF-8; name="0002-Allow-composite-types-in-bootstrap-20210304.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Allow-composite-types-in-bootstrap-20210304.patch" ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2022-01-21 05:33 Masahiko Sawada <[email protected]> 0 siblings, 1 reply; 4+ messages in thread From: Masahiko Sawada @ 2022-01-21 05:33 UTC (permalink / raw) To: Peter Eisentraut <[email protected]>; +Cc: pgsql-hackers On Wed, Dec 15, 2021 at 7:13 AM Peter Eisentraut <[email protected]> wrote: > > On 31.10.21 11:08, Peter Eisentraut wrote: > > I want to reactivate $subject. I took Petr Jelinek's patch from [0], > > rebased it, added a bit of testing. It basically works, but as > > mentioned in [0], there are various issues to work out. > > > > The idea is that the standby runs a background worker to periodically > > fetch replication slot information from the primary. On failover, a > > logical subscriber would then ideally find up-to-date replication slots > > on the new publisher and can just continue normally. > > > So, again, this isn't anywhere near ready, but there is already a lot > > here to gather feedback about how it works, how it should work, how to > > configure it, and how it fits into an overall replication and HA > > architecture. > > The second, > standby_slot_names, is set on the primary. It holds back logical > replication until the listed physical standbys have caught up. That > way, when failover is necessary, the promoted standby is not behind the > logical replication consumers. I might be missing something but isn’t it okay even if the new primary server is behind the subscribers? IOW, even if two slot's LSNs (i.e., restart_lsn and confirm_flush_lsn) are behind the subscriber's remote LSN (i.e., pg_replication_origin.remote_lsn), the primary sends only transactions that were committed after the remote_lsn. So the subscriber can resume logical replication with the new primary without any data loss. The new primary should not be ahead of the subscribers because it forwards the logical replication start LSN to the slot’s confirm_flush_lsn in this case. But it cannot happen since the remote LSN of the subscriber’s origin is always updated first, then the confirm_flush_lsn of the slot on the primary is updated, and then the confirm_flush_lsn of the slot on the standby is synchronized. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/ ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2022-01-21 23:02 Hsu, John <[email protected]> parent: Masahiko Sawada <[email protected]> 0 siblings, 1 reply; 4+ messages in thread From: Hsu, John @ 2022-01-21 23:02 UTC (permalink / raw) To: Masahiko Sawada <[email protected]>; Peter Eisentraut <[email protected]>; +Cc: pgsql-hackers > I might be missing something but isn’t it okay even if the new primary > server is behind the subscribers? IOW, even if two slot's LSNs (i.e., > restart_lsn and confirm_flush_lsn) are behind the subscriber's remote > LSN (i.e., pg_replication_origin.remote_lsn), the primary sends only > transactions that were committed after the remote_lsn. So the > subscriber can resume logical replication with the new primary without > any data loss. Maybe I'm misreading, but I thought the purpose of this to make sure that the logical subscriber does not have data that has not been replicated to the new primary. The use-case I can think of would be if synchronous_commit were enabled and fail-over occurs. If we didn't have this set, isn't it possible that this logical subscriber has extra commits that aren't present on the newly promoted primary? And sorry I accidentally started a new thread in my last reply. Re-pasting some of my previous questions/comments: wait_for_standby_confirmation does not update standby_slot_names once it's in a loop and can't be fixed with SIGHUP. Similarly, synchronize_slot_names isn't updated once the worker is launched. If a logical slot was dropped on the writer, should the worker drop logical slots that it was previously synchronizing but are no longer present? Or should we leave that to the user to manage? I'm trying to think why users would want to sync logical slots to a reader but not have that be dropped as well if it's no longer present. Is there a reason we're deciding to use one-worker syncing per database instead of one general worker that syncs across all the databases? I imagine I'm missing something obvious here. As for how standby_slot_names should be configured, I'd prefer the flexibility similar to what we have for synchronus_standby_names since that seems the most analogous. It'd provide flexibility for failovers, which I imagine is the most common use-case. On 1/20/22, 9:34 PM, "Masahiko Sawada" <[email protected]> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. On Wed, Dec 15, 2021 at 7:13 AM Peter Eisentraut <[email protected]> wrote: > > On 31.10.21 11:08, Peter Eisentraut wrote: > > I want to reactivate $subject. I took Petr Jelinek's patch from [0], > > rebased it, added a bit of testing. It basically works, but as > > mentioned in [0], there are various issues to work out. > > > > The idea is that the standby runs a background worker to periodically > > fetch replication slot information from the primary. On failover, a > > logical subscriber would then ideally find up-to-date replication slots > > on the new publisher and can just continue normally. > > > So, again, this isn't anywhere near ready, but there is already a lot > > here to gather feedback about how it works, how it should work, how to > > configure it, and how it fits into an overall replication and HA > > architecture. > > The second, > standby_slot_names, is set on the primary. It holds back logical > replication until the listed physical standbys have caught up. That > way, when failover is necessary, the promoted standby is not behind the > logical replication consumers. I might be missing something but isn’t it okay even if the new primary server is behind the subscribers? IOW, even if two slot's LSNs (i.e., restart_lsn and confirm_flush_lsn) are behind the subscriber's remote LSN (i.e., pg_replication_origin.remote_lsn), the primary sends only transactions that were committed after the remote_lsn. So the subscriber can resume logical replication with the new primary without any data loss. The new primary should not be ahead of the subscribers because it forwards the logical replication start LSN to the slot’s confirm_flush_lsn in this case. But it cannot happen since the remote LSN of the subscriber’s origin is always updated first, then the confirm_flush_lsn of the slot on the primary is updated, and then the confirm_flush_lsn of the slot on the standby is synchronized. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/ ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2022-02-04 17:51 Ashutosh Sharma <[email protected]> parent: Hsu, John <[email protected]> 0 siblings, 0 replies; 4+ messages in thread From: Ashutosh Sharma @ 2022-02-04 17:51 UTC (permalink / raw) To: Hsu, John <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Peter Eisentraut <[email protected]>; pgsql-hackers On Sat, Jan 22, 2022 at 4:33 AM Hsu, John <[email protected]> wrote: > > > I might be missing something but isn’t it okay even if the new primary > > server is behind the subscribers? IOW, even if two slot's LSNs (i.e., > > restart_lsn and confirm_flush_lsn) are behind the subscriber's remote > > LSN (i.e., pg_replication_origin.remote_lsn), the primary sends only > > transactions that were committed after the remote_lsn. So the > > subscriber can resume logical replication with the new primary without > > any data loss. > > Maybe I'm misreading, but I thought the purpose of this to make > sure that the logical subscriber does not have data that has not been > replicated to the new primary. The use-case I can think of would be > if synchronous_commit were enabled and fail-over occurs. If > we didn't have this set, isn't it possible that this logical subscriber > has extra commits that aren't present on the newly promoted primary? > This is very much possible if the new primary used to be asynchronous standby. But, it seems like the current patch is trying to hold the logical replication until the data has been replicated to the physical standby when synchronous_slot_names is set. This will ensure that the logical subscriber is never ahead of the new primary. However, AFAIU that's not the primary use-case of this patch; instead this is to ensure that the logical subscribers continue getting data from the new primary when the failover occurs. > > If a logical slot was dropped on the writer, should the worker drop logical > slots that it was previously synchronizing but are no longer present? Or > should we leave that to the user to manage? I'm trying to think why users > would want to sync logical slots to a reader but not have that be dropped > as well if it's no longer present. > AFAIU this should be taken care of by the background worker used to synchronize the replication slot. -- With Regards, Ashutosh Sharma. ^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2022-02-04 17:51 UTC | newest] Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-11-20 02:48 [PATCH 1/5] bootstrap: convert Typ to a List* Justin Pryzby <[email protected]> 2022-01-21 05:33 Re: Synchronizing slots from primary to standby Masahiko Sawada <[email protected]> 2022-01-21 23:02 ` Re: Synchronizing slots from primary to standby Hsu, John <[email protected]> 2022-02-04 17:51 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox