public inbox for [email protected]
help / color / mirror / Atom feedRe: Synchronizing slots from primary to standby
59+ messages / 11 participants
[nested] [flat]
* Re: Synchronizing slots from primary to standby
@ 2022-02-07 08:08 Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Ashutosh Sharma @ 2022-02-07 08:08 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; pgsql-hackers
Hi Andres,
Are you talking about this scenario - what if the logical replication
slot on the publisher is dropped, but is being referenced by the
standby where the slot is synchronized? Should the redo function for
the drop replication slot have the capability to drop it on standby
and its subscribers (if any) as well?
--
With Regards,
Ashutosh Sharma.
On Sun, Feb 6, 2022 at 1:29 AM Andres Freund <[email protected]> wrote:
>
> Hi,
>
> On 2022-01-03 14:46:52 +0100, Peter Eisentraut wrote:
> > From ec00dc6ab8bafefc00e9b1c78ac9348b643b8a87 Mon Sep 17 00:00:00 2001
> > From: Peter Eisentraut <[email protected]>
> > Date: Mon, 3 Jan 2022 14:43:36 +0100
> > Subject: [PATCH v3] Synchronize logical replication slots from primary to
> > standby
>
> I've just skimmed the patch and the related threads. As far as I can tell this
> cannot be safely used without the conflict handling in [1], is that correct?
>
>
> Greetings,
>
> Andres Freund
>
> [1] https://postgr.es/m/CA%2BTgmoZd-JqNL1-R3RJ0jQRD%2B-dc94X0nPJgh%2BdwdDF0rFuE3g%40mail.gmail.com
>
>
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
@ 2022-02-07 20:32 ` Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Andres Freund @ 2022-02-07 20:32 UTC (permalink / raw)
To: Ashutosh Sharma <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; pgsql-hackers
Hi,
On 2022-02-07 13:38:38 +0530, Ashutosh Sharma wrote:
> Are you talking about this scenario - what if the logical replication
> slot on the publisher is dropped, but is being referenced by the
> standby where the slot is synchronized?
It's a bit hard to say, because neither in this thread nor in the patch I've
found a clear description of what the syncing needs to & tries to
guarantee. It might be that that was discussed in one of the precursor
threads, but...
Generally I don't think we can permit scenarios where a slot can be in a
"corrupt" state, i.e. missing required catalog entries, after "normal"
administrative commands (i.e. not mucking around in catalog entries / on-disk
files). Even if the sequence of commands may be a bit weird. All such cases
need to be either prevented or detected.
As far as I can tell, the way this patch keeps slots on physical replicas
"valid" is solely by reorderbuffer.c blocking during replay via
wait_for_standby_confirmation().
Which means that if e.g. the standby_slot_names GUC differs from
synchronize_slot_names on the physical replica, the slots synchronized on the
physical replica are not going to be valid. Or if the primary drops its
logical slots.
> Should the redo function for the drop replication slot have the capability
> to drop it on standby and its subscribers (if any) as well?
Slots are not WAL logged (and shouldn't be).
I think you pretty much need the recovery conflict handling infrastructure I
referenced upthread, which recognized during replay if a record has a conflict
with a slot on a standby. And then ontop of that you can build something like
this patch.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
@ 2022-02-08 14:57 ` Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Ashutosh Sharma @ 2022-02-08 14:57 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; pgsql-hackers
On Tue, Feb 8, 2022 at 2:02 AM Andres Freund <[email protected]> wrote:
>
> Hi,
>
> On 2022-02-07 13:38:38 +0530, Ashutosh Sharma wrote:
> > Are you talking about this scenario - what if the logical replication
> > slot on the publisher is dropped, but is being referenced by the
> > standby where the slot is synchronized?
>
> It's a bit hard to say, because neither in this thread nor in the patch I've
> found a clear description of what the syncing needs to & tries to
> guarantee. It might be that that was discussed in one of the precursor
> threads, but...
>
> Generally I don't think we can permit scenarios where a slot can be in a
> "corrupt" state, i.e. missing required catalog entries, after "normal"
> administrative commands (i.e. not mucking around in catalog entries / on-disk
> files). Even if the sequence of commands may be a bit weird. All such cases
> need to be either prevented or detected.
>
>
> As far as I can tell, the way this patch keeps slots on physical replicas
> "valid" is solely by reorderbuffer.c blocking during replay via
> wait_for_standby_confirmation().
>
> Which means that if e.g. the standby_slot_names GUC differs from
> synchronize_slot_names on the physical replica, the slots synchronized on the
> physical replica are not going to be valid. Or if the primary drops its
> logical slots.
>
>
> > Should the redo function for the drop replication slot have the capability
> > to drop it on standby and its subscribers (if any) as well?
>
> Slots are not WAL logged (and shouldn't be).
>
> I think you pretty much need the recovery conflict handling infrastructure I
> referenced upthread, which recognized during replay if a record has a conflict
> with a slot on a standby. And then ontop of that you can build something like
> this patch.
>
OK. Understood, thanks Andres.
--
With Regards,
Ashutosh Sharma.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
@ 2022-02-10 21:47 ` Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Bruce Momjian @ 2022-02-10 21:47 UTC (permalink / raw)
To: Ashutosh Sharma <[email protected]>; +Cc: Andres Freund <[email protected]>; Peter Eisentraut <[email protected]>; pgsql-hackers
On Tue, Feb 8, 2022 at 08:27:32PM +0530, Ashutosh Sharma wrote:
> > Which means that if e.g. the standby_slot_names GUC differs from
> > synchronize_slot_names on the physical replica, the slots synchronized on the
> > physical replica are not going to be valid. Or if the primary drops its
> > logical slots.
> >
> >
> > > Should the redo function for the drop replication slot have the capability
> > > to drop it on standby and its subscribers (if any) as well?
> >
> > Slots are not WAL logged (and shouldn't be).
> >
> > I think you pretty much need the recovery conflict handling infrastructure I
> > referenced upthread, which recognized during replay if a record has a conflict
> > with a slot on a standby. And then ontop of that you can build something like
> > this patch.
> >
>
> OK. Understood, thanks Andres.
I would love to see this feature in PG 15. Can someone explain its
current status? Thanks.
--
Bruce Momjian <[email protected]> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
@ 2022-02-11 14:26 ` Peter Eisentraut <[email protected]>
2022-02-18 15:25 ` Re: Synchronizing slots from primary to standby James Coleman <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
0 siblings, 2 replies; 59+ messages in thread
From: Peter Eisentraut @ 2022-02-11 14:26 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
On 10.02.22 22:47, Bruce Momjian wrote:
> On Tue, Feb 8, 2022 at 08:27:32PM +0530, Ashutosh Sharma wrote:
>>> Which means that if e.g. the standby_slot_names GUC differs from
>>> synchronize_slot_names on the physical replica, the slots synchronized on the
>>> physical replica are not going to be valid. Or if the primary drops its
>>> logical slots.
>>>
>>>
>>>> Should the redo function for the drop replication slot have the capability
>>>> to drop it on standby and its subscribers (if any) as well?
>>>
>>> Slots are not WAL logged (and shouldn't be).
>>>
>>> I think you pretty much need the recovery conflict handling infrastructure I
>>> referenced upthread, which recognized during replay if a record has a conflict
>>> with a slot on a standby. And then ontop of that you can build something like
>>> this patch.
>>>
>>
>> OK. Understood, thanks Andres.
>
> I would love to see this feature in PG 15. Can someone explain its
> current status? Thanks.
The way I understand it:
1. This feature (probably) depends on the "Minimal logical decoding on
standbys" patch. The details there aren't totally clear (to me). That
patch had some activity lately but I don't see it in a state that it's
nearing readiness.
2. I think the way this (my) patch is currently written needs some
refactoring about how we launch and manage workers. Right now, it's all
mangled together with logical replication, since that is a convenient
way to launch and manage workers, but it really doesn't need to be tied
to logical replication, since it can also be used for other logical slots.
3. It's an open question how to configure this. My patch show a very
minimal configuration that allows you to keep all logical slots always
behind one physical slot, which addresses one particular use case. In
general, you might have things like, one set of logical slots should
stay behind one physical slot, another set behind another physical slot,
another set should not care, etc. This could turn into something like
the synchronous replication feature, where it ends up with its own
configuration language.
Each of these are clearly significant jobs on their own.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
@ 2022-02-18 15:25 ` James Coleman <[email protected]>
1 sibling, 0 replies; 59+ messages in thread
From: James Coleman @ 2022-02-18 15:25 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
On Fri, Feb 11, 2022 at 9:26 AM Peter Eisentraut
<[email protected]> wrote:
>
> On 10.02.22 22:47, Bruce Momjian wrote:
> > I would love to see this feature in PG 15. Can someone explain its
> > current status? Thanks.
>
> The way I understand it:
> ...
Hi Peter,
I'm starting to review this patch, and last time I checked I noticed
it didn't seem to apply cleanly to master anymore. Would you be able
to send a rebased version?
Thanks,
James Coleman
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
@ 2022-11-15 09:02 ` Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
1 sibling, 1 reply; 59+ messages in thread
From: Drouvot, Bertrand @ 2022-11-15 09:02 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
Hi,
On 2/11/22 3:26 PM, Peter Eisentraut wrote:
> On 10.02.22 22:47, Bruce Momjian wrote:
>> On Tue, Feb 8, 2022 at 08:27:32PM +0530, Ashutosh Sharma wrote:
>>>> Which means that if e.g. the standby_slot_names GUC differs from
>>>> synchronize_slot_names on the physical replica, the slots
>>>> synchronized on the
>>>> physical replica are not going to be valid. Or if the primary drops
>>>> its
>>>> logical slots.
>>>>
>>>>
>>>>> Should the redo function for the drop replication slot have the
>>>>> capability
>>>>> to drop it on standby and its subscribers (if any) as well?
>>>>
>>>> Slots are not WAL logged (and shouldn't be).
>>>>
>>>> I think you pretty much need the recovery conflict handling
>>>> infrastructure I
>>>> referenced upthread, which recognized during replay if a record has
>>>> a conflict
>>>> with a slot on a standby. And then ontop of that you can build
>>>> something like
>>>> this patch.
>>>>
>>>
>>> OK. Understood, thanks Andres.
>>
>> I would love to see this feature in PG 15. Can someone explain its
>> current status? Thanks.
>
> The way I understand it:
>
> 1. This feature (probably) depends on the "Minimal logical decoding on
> standbys" patch. The details there aren't totally clear (to me). That
> patch had some activity lately but I don't see it in a state that it's
> nearing readiness.
>
FWIW, a proposal has been submitted in [1] to add information in the WAL
records in preparation for logical slot conflict handling.
[1]:
https://www.postgresql.org/message-id/[email protected]
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
@ 2023-04-14 13:22 ` Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Drouvot, Bertrand @ 2023-04-14 13:22 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
Hi,
On 11/15/22 10:02 AM, Drouvot, Bertrand wrote:
> Hi,
>
> On 2/11/22 3:26 PM, Peter Eisentraut wrote:
>> On 10.02.22 22:47, Bruce Momjian wrote:
>>> On Tue, Feb 8, 2022 at 08:27:32PM +0530, Ashutosh Sharma wrote:
>>>>> Which means that if e.g. the standby_slot_names GUC differs from
>>>>> synchronize_slot_names on the physical replica, the slots synchronized on the
>>>>> physical replica are not going to be valid. Or if the primary drops its
>>>>> logical slots.
>>>>>
>>>>>
>>>>>> Should the redo function for the drop replication slot have the capability
>>>>>> to drop it on standby and its subscribers (if any) as well?
>>>>>
>>>>> Slots are not WAL logged (and shouldn't be).
>>>>>
>>>>> I think you pretty much need the recovery conflict handling infrastructure I
>>>>> referenced upthread, which recognized during replay if a record has a conflict
>>>>> with a slot on a standby. And then ontop of that you can build something like
>>>>> this patch.
>>>>>
>>>>
>>>> OK. Understood, thanks Andres.
>>>
>>> I would love to see this feature in PG 15. Can someone explain its
>>> current status? Thanks.
>>
>> The way I understand it:
>>
>> 1. This feature (probably) depends on the "Minimal logical decoding on standbys" patch. The details there aren't totally clear (to me). That patch had some activity lately but I don't see it in a state that it's nearing readiness.
>>
>
> FWIW, a proposal has been submitted in [1] to add information in the WAL records in preparation for logical slot conflict handling.
>
> [1]: https://www.postgresql.org/message-id/[email protected]
>
Now that the "Minimal logical decoding on standby" patch series (mentioned up-thread) has been
committed, I think we can resume working on this one ("Synchronizing slots from primary to standby").
I'll work on a rebase and share it once done (unless someone already started working on a rebase).
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
@ 2023-04-17 14:05 ` Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-06-29 10:22 ` RE: Synchronizing slots from primary to standby Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 2 replies; 59+ messages in thread
From: Drouvot, Bertrand @ 2023-04-17 14:05 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
Hi,
On 4/14/23 3:22 PM, Drouvot, Bertrand wrote:
> Now that the "Minimal logical decoding on standby" patch series (mentioned up-thread) has been
> committed, I think we can resume working on this one ("Synchronizing slots from primary to standby").
>
> I'll work on a rebase and share it once done (unless someone already started working on a rebase).
>
Please find attached V5 (a rebase of V4 posted up-thread).
In addition to the "rebasing" work, the TAP test adds a test about conflict handling (logical slot invalidation)
relying on the work done in the "Minimal logical decoding on standby" patch series.
I did not look more at the patch (than what's was needed for the rebase) but plan to do so.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
From 655359eedf37d8f2e522aeb1ec8c48adfc1759b1 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <[email protected]>
Date: Thu, 13 Apr 2023 11:32:28 +0000
Subject: [PATCH v5] Synchronize logical replication slots from primary to
standby
---
doc/src/sgml/config.sgml | 34 ++
src/backend/commands/subscriptioncmds.c | 4 +-
src/backend/postmaster/bgworker.c | 3 +
.../libpqwalreceiver/libpqwalreceiver.c | 95 ++++
src/backend/replication/logical/Makefile | 1 +
src/backend/replication/logical/launcher.c | 263 +++++++----
src/backend/replication/logical/meson.build | 1 +
.../replication/logical/reorderbuffer.c | 86 ++++
src/backend/replication/logical/slotsync.c | 413 ++++++++++++++++++
src/backend/replication/logical/tablesync.c | 13 +-
src/backend/replication/logical/worker.c | 3 +-
src/backend/replication/repl_gram.y | 32 +-
src/backend/replication/repl_scanner.l | 2 +
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 195 +++++++++
src/backend/utils/activity/wait_event.c | 3 +
src/backend/utils/misc/guc_tables.c | 26 ++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/commands/subscriptioncmds.h | 3 +
src/include/nodes/replnodes.h | 9 +
src/include/replication/logicallauncher.h | 2 +
src/include/replication/logicalworker.h | 9 +
src/include/replication/slot.h | 5 +-
src/include/replication/walreceiver.h | 20 +
src/include/replication/worker_internal.h | 8 +-
src/include/utils/wait_event.h | 1 +
src/test/recovery/meson.build | 1 +
src/test/recovery/t/037_slot_sync.pl | 130 ++++++
28 files changed, 1272 insertions(+), 94 deletions(-)
3.8% doc/src/sgml/
7.1% src/backend/replication/libpqwalreceiver/
54.7% src/backend/replication/logical/
14.9% src/backend/replication/
3.3% src/backend/
4.0% src/include/replication/
10.9% src/test/recovery/t/
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 091a79d4f3..1360885208 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4466,6 +4466,23 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-standby-slot-names" xreflabel="standby_slot_names">
+ <term><varname>standby_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>standby_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ List of physical replication slots that logical replication waits for.
+ If a logical replication connection is meant to switch to a physical
+ standby after the standby is promoted, the physical replication slot
+ for the standby should be listed here. This ensures that logical
+ replication is not ahead of the physical standby.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
@@ -4649,6 +4666,23 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
</listitem>
</varlistentry>
+ <varlistentry id="guc-synchronize_slot_names" xreflabel="synchronize_slot_names">
+ <term><varname>synchronize_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>synchronize_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies a list of logical replication slots that a physical standby
+ should synchronize from the primary server. This is necessary to be
+ able to retarget those logical replication connections to this standby
+ if it gets promoted. Specify <literal>*</literal> to synchronize all
+ logical replication slots. The default is empty.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 3251d89ba8..8721706b79 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -991,7 +991,7 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data,
RemoveSubscriptionRel(sub->oid, relid);
- logicalrep_worker_stop(sub->oid, relid);
+ logicalrep_worker_stop(MyDatabaseId, sub->oid, relid);
/*
* For READY state, we would have already dropped the
@@ -1589,7 +1589,7 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel)
{
LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc);
- logicalrep_worker_stop(w->subid, w->relid);
+ logicalrep_worker_stop(w->dbid, w->subid, w->relid);
}
list_free(subworkers);
diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c
index 0dd22b2351..a89d1f10a1 100644
--- a/src/backend/postmaster/bgworker.c
+++ b/src/backend/postmaster/bgworker.c
@@ -129,6 +129,9 @@ static const struct
{
"ApplyWorkerMain", ApplyWorkerMain
},
+ {
+ "ReplSlotSyncMain", ReplSlotSyncMain
+ },
{
"ParallelApplyWorkerMain", ParallelApplyWorkerMain
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 052505e46f..4f7417c49a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -34,6 +34,7 @@
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
#include "utils/tuplestore.h"
+#include "utils/varlena.h"
PG_MODULE_MAGIC;
@@ -58,6 +59,7 @@ static void libpqrcv_get_senderinfo(WalReceiverConn *conn,
char **sender_host, int *sender_port);
static char *libpqrcv_identify_system(WalReceiverConn *conn,
TimeLineID *primary_tli);
+static List *libpqrcv_list_slots(WalReceiverConn *conn, const char *slot_names);
static int libpqrcv_server_version(WalReceiverConn *conn);
static void libpqrcv_readtimelinehistoryfile(WalReceiverConn *conn,
TimeLineID tli, char **filename,
@@ -96,6 +98,7 @@ static WalReceiverFunctionsType PQWalReceiverFunctions = {
.walrcv_receive = libpqrcv_receive,
.walrcv_send = libpqrcv_send,
.walrcv_create_slot = libpqrcv_create_slot,
+ .walrcv_list_slots = libpqrcv_list_slots,
.walrcv_get_backend_pid = libpqrcv_get_backend_pid,
.walrcv_exec = libpqrcv_exec,
.walrcv_disconnect = libpqrcv_disconnect
@@ -409,6 +412,98 @@ libpqrcv_server_version(WalReceiverConn *conn)
return PQserverVersion(conn->streamConn);
}
+/*
+ * Get list of slots from primary.
+ */
+static List *
+libpqrcv_list_slots(WalReceiverConn *conn, const char *slot_names)
+{
+ PGresult *res;
+ List *slotlist = NIL;
+ int ntuples;
+ StringInfoData s;
+ WalRecvReplicationSlotData *slot_data;
+
+ initStringInfo(&s);
+ appendStringInfoString(&s, "LIST_SLOTS");
+
+ if (strcmp(slot_names, "") != 0 && strcmp(slot_names, "*") != 0)
+ {
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+
+ appendStringInfoChar(&s, ' ');
+ rawname = pstrdup(slot_names);
+ SplitIdentifierString(rawname, ',', &namelist);
+ foreach (lc, namelist)
+ {
+ if (lc != list_head(namelist))
+ appendStringInfoChar(&s, ',');
+ appendStringInfo(&s, "%s",
+ quote_identifier(lfirst(lc)));
+ }
+ }
+
+ res = libpqrcv_PQexec(conn->streamConn, s.data);
+ pfree(s.data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ PQclear(res);
+ ereport(ERROR,
+ (errmsg("could not receive list of slots the primary server: %s",
+ pchomp(PQerrorMessage(conn->streamConn)))));
+ }
+ if (PQnfields(res) < 10)
+ {
+ int nfields = PQnfields(res);
+
+ PQclear(res);
+ ereport(ERROR,
+ (errmsg("invalid response from primary server"),
+ errdetail("Could not get list of slots: got %d fields, expected %d or more fields.",
+ nfields, 10)));
+ }
+
+ ntuples = PQntuples(res);
+ for (int i = 0; i < ntuples; i++)
+ {
+ char *slot_type;
+
+ slot_data = palloc0(sizeof(WalRecvReplicationSlotData));
+ namestrcpy(&slot_data->persistent_data.name, PQgetvalue(res, i, 0));
+ if (!PQgetisnull(res, i, 1))
+ namestrcpy(&slot_data->persistent_data.plugin, PQgetvalue(res, i, 1));
+ slot_type = PQgetvalue(res, i, 2);
+ if (!PQgetisnull(res, i, 3))
+ slot_data->persistent_data.database = atooid(PQgetvalue(res, i, 3));
+ if (strcmp(slot_type, "physical") == 0)
+ {
+ if (OidIsValid(slot_data->persistent_data.database))
+ elog(ERROR, "unexpected physical replication slot with database set");
+ }
+ if (pg_strtoint32(PQgetvalue(res, i, 5)) == 1)
+ slot_data->persistent_data.persistency = RS_TEMPORARY;
+ else
+ slot_data->persistent_data.persistency = RS_PERSISTENT;
+ if (!PQgetisnull(res, i, 6))
+ slot_data->persistent_data.xmin = atooid(PQgetvalue(res, i, 6));
+ if (!PQgetisnull(res, i, 7))
+ slot_data->persistent_data.catalog_xmin = atooid(PQgetvalue(res, i, 7));
+ if (!PQgetisnull(res, i, 8))
+ slot_data->persistent_data.restart_lsn = strtou64(PQgetvalue(res, i, 8), NULL, 10);
+ if (!PQgetisnull(res, i, 9))
+ slot_data->persistent_data.confirmed_flush = strtou64(PQgetvalue(res, i, 9), NULL, 10);
+
+ slot_data->last_sync_time = 0;
+ slotlist = lappend(slotlist, slot_data);
+ }
+
+ PQclear(res);
+
+ return slotlist;
+}
+
/*
* Start streaming WAL data from given streaming options.
*
diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile
index 2dc25e37bb..ba03eeff1c 100644
--- a/src/backend/replication/logical/Makefile
+++ b/src/backend/replication/logical/Makefile
@@ -25,6 +25,7 @@ OBJS = \
proto.o \
relation.o \
reorderbuffer.o \
+ slotsync.o \
snapbuild.o \
tablesync.o \
worker.o
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 970d170e73..14af724639 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -22,6 +22,7 @@
#include "access/htup_details.h"
#include "access/tableam.h"
#include "access/xact.h"
+#include "catalog/pg_authid.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
#include "funcapi.h"
@@ -246,7 +247,7 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker,
* We are only interested in the leader apply worker or table sync worker.
*/
LogicalRepWorker *
-logicalrep_worker_find(Oid subid, Oid relid, bool only_running)
+logicalrep_worker_find(Oid dbid, Oid subid, Oid relid, bool only_running)
{
int i;
LogicalRepWorker *res = NULL;
@@ -262,8 +263,8 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running)
if (isParallelApplyWorker(w))
continue;
- if (w->in_use && w->subid == subid && w->relid == relid &&
- (!only_running || w->proc))
+ if (w->in_use && w->dbid == dbid && w->subid == subid &&
+ w->relid == relid && (!only_running || w->proc))
{
res = w;
break;
@@ -320,9 +321,13 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid,
/* Sanity check - tablesync worker cannot be a subworker */
Assert(!(is_parallel_apply_worker && OidIsValid(relid)));
- ereport(DEBUG1,
- (errmsg_internal("starting logical replication worker for subscription \"%s\"",
- subname)));
+ if (OidIsValid(subid))
+ ereport(DEBUG1,
+ (errmsg_internal("starting logical replication worker for subscription \"%s\"",
+ subname)));
+ else
+ ereport(DEBUG1,
+ (errmsg_internal("starting replication slot synchronization worker")));
/* Report this after the initial starting message for consistency. */
if (max_replication_slots == 0)
@@ -359,7 +364,9 @@ retry:
* reason we do this is because if some worker failed to start up and its
* parent has crashed while waiting, the in_use state was never cleared.
*/
- if (worker == NULL || nsyncworkers >= max_sync_workers_per_subscription)
+ if (worker == NULL ||
+ (OidIsValid(relid) &&
+ nsyncworkers >= max_sync_workers_per_subscription))
{
bool did_cleanup = false;
@@ -455,15 +462,20 @@ retry:
memset(&bgw, 0, sizeof(bgw));
bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
BGWORKER_BACKEND_DATABASE_CONNECTION;
- bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ bgw.bgw_start_time = BgWorkerStart_ConsistentState;
snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres");
- if (is_parallel_apply_worker)
+ if (!OidIsValid(subid))
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ReplSlotSyncMain");
+ else if (is_parallel_apply_worker)
snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ParallelApplyWorkerMain");
else
snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain");
- if (OidIsValid(relid))
+ if (!OidIsValid(subid))
+ snprintf(bgw.bgw_name, BGW_MAXLEN,
+ "replication slot synchronization worker");
+ else if (OidIsValid(relid))
snprintf(bgw.bgw_name, BGW_MAXLEN,
"logical replication worker for subscription %u sync %u", subid, relid);
else if (is_parallel_apply_worker)
@@ -591,13 +603,13 @@ logicalrep_worker_stop_internal(LogicalRepWorker *worker, int signo)
* Stop the logical replication worker for subid/relid, if any.
*/
void
-logicalrep_worker_stop(Oid subid, Oid relid)
+logicalrep_worker_stop(Oid dbid, Oid subid, Oid relid)
{
LogicalRepWorker *worker;
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- worker = logicalrep_worker_find(subid, relid, false);
+ worker = logicalrep_worker_find(dbid, subid, relid, false);
if (worker)
{
@@ -640,13 +652,13 @@ logicalrep_pa_worker_stop(int slot_no, uint16 generation)
* Wake up (using latch) any logical replication worker for specified sub/rel.
*/
void
-logicalrep_worker_wakeup(Oid subid, Oid relid)
+logicalrep_worker_wakeup(Oid dbid, Oid subid, Oid relid)
{
LogicalRepWorker *worker;
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- worker = logicalrep_worker_find(subid, relid, true);
+ worker = logicalrep_worker_find(dbid, subid, relid, true);
if (worker)
logicalrep_worker_wakeup_ptr(worker);
@@ -888,7 +900,7 @@ ApplyLauncherRegister(void)
memset(&bgw, 0, sizeof(bgw));
bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
BGWORKER_BACKEND_DATABASE_CONNECTION;
- bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ bgw.bgw_start_time = BgWorkerStart_ConsistentState;
snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres");
snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyLauncherMain");
snprintf(bgw.bgw_name, BGW_MAXLEN,
@@ -1071,6 +1083,157 @@ ApplyLauncherWakeup(void)
kill(LogicalRepCtx->launcher_pid, SIGUSR1);
}
+static void
+ApplyLauncherStartSlotSync(long *wait_time)
+{
+ WalReceiverConn *wrconn;
+ char *err;
+ List *slots;
+ ListCell *lc;
+ MemoryContext tmpctx;
+ MemoryContext oldctx;
+
+ if (strcmp(synchronize_slot_names, "") == 0)
+ return;
+
+ wrconn = walrcv_connect(PrimaryConnInfo, false, false,
+ "Logical Replication Launcher", &err);
+ if (!wrconn)
+ ereport(ERROR,
+ (errmsg("could not connect to the primary server: %s", err)));
+
+ /* Use temporary context for the slot list and worker info. */
+ tmpctx = AllocSetContextCreate(TopMemoryContext,
+ "Logical Replication Launcher slot sync ctx",
+ ALLOCSET_DEFAULT_SIZES);
+ oldctx = MemoryContextSwitchTo(tmpctx);
+
+ slots = walrcv_list_slots(wrconn, synchronize_slot_names);
+
+ foreach(lc, slots)
+ {
+ WalRecvReplicationSlotData *slot_data = lfirst(lc);
+ LogicalRepWorker *w;
+ TimestampTz last_sync;
+ TimestampTz now;
+ long elapsed;
+
+ if (!OidIsValid(slot_data->persistent_data.database))
+ continue;
+
+ LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
+ w = logicalrep_worker_find(slot_data->persistent_data.database, InvalidOid,
+ InvalidOid, false);
+ LWLockRelease(LogicalRepWorkerLock);
+
+ if (w != NULL)
+ continue; /* worker is running already */
+
+ /*
+ * If the worker is eligible to start now, launch it. Otherwise,
+ * adjust wait_time so that we'll wake up as soon as it can be
+ * started.
+ *
+ * Each apply worker can only be restarted once per
+ * wal_retrieve_retry_interval, so that errors do not cause us to
+ * repeatedly restart the worker as fast as possible.
+ */
+ last_sync = slot_data->last_sync_time;
+ now = GetCurrentTimestamp();
+ if (last_sync == 0 ||
+ (elapsed = TimestampDifferenceMilliseconds(last_sync, now)) >= wal_retrieve_retry_interval)
+ {
+ slot_data->last_sync_time = now;
+ logicalrep_worker_launch(slot_data->persistent_data.database,
+ InvalidOid, NULL,
+ BOOTSTRAP_SUPERUSERID, InvalidOid,
+ DSM_HANDLE_INVALID);
+ }
+ else
+ {
+ *wait_time = Min(*wait_time,
+ wal_retrieve_retry_interval - elapsed);
+ }
+ }
+
+ /* Switch back to original memory context. */
+ MemoryContextSwitchTo(oldctx);
+ /* Clean the temporary memory. */
+ MemoryContextDelete(tmpctx);
+
+ walrcv_disconnect(wrconn);
+}
+
+static void
+ApplyLauncherStartSubs(long *wait_time)
+{
+ List *sublist;
+ ListCell *lc;
+ MemoryContext subctx;
+ MemoryContext oldctx;
+
+ /* Use temporary context to avoid leaking memory across cycles. */
+ subctx = AllocSetContextCreate(TopMemoryContext,
+ "Logical Replication Launcher sublist",
+ ALLOCSET_DEFAULT_SIZES);
+ oldctx = MemoryContextSwitchTo(subctx);
+
+ /* Start the missing workers for enabled subscriptions. */
+ sublist = get_subscription_list();
+ foreach(lc, sublist)
+ {
+ Subscription *sub = (Subscription *) lfirst(lc);
+ LogicalRepWorker *w;
+ TimestampTz last_start;
+ TimestampTz now;
+ long elapsed;
+
+ if (!sub->enabled)
+ continue;
+
+ LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
+ w = logicalrep_worker_find(sub->dbid, sub->oid, InvalidOid, false);
+ LWLockRelease(LogicalRepWorkerLock);
+
+ if (w != NULL)
+ continue; /* worker is running already */
+
+ /*
+ * If the worker is eligible to start now, launch it. Otherwise,
+ * adjust wait_time so that we'll wake up as soon as it can be
+ * started.
+ *
+ * Each subscription's apply worker can only be restarted once per
+ * wal_retrieve_retry_interval, so that errors do not cause us to
+ * repeatedly restart the worker as fast as possible. In cases
+ * where a restart is expected (e.g., subscription parameter
+ * changes), another process should remove the last-start entry
+ * for the subscription so that the worker can be restarted
+ * without waiting for wal_retrieve_retry_interval to elapse.
+ */
+ last_start = ApplyLauncherGetWorkerStartTime(sub->oid);
+ now = GetCurrentTimestamp();
+ if (last_start == 0 ||
+ (elapsed = TimestampDifferenceMilliseconds(last_start, now)) >= wal_retrieve_retry_interval)
+ {
+ ApplyLauncherSetWorkerStartTime(sub->oid, now);
+ logicalrep_worker_launch(sub->dbid, sub->oid, sub->name,
+ sub->owner, InvalidOid,
+ DSM_HANDLE_INVALID);
+ }
+ else
+ {
+ *wait_time = Min(*wait_time,
+ wal_retrieve_retry_interval - elapsed);
+ }
+ }
+
+ /* Switch back to original memory context. */
+ MemoryContextSwitchTo(oldctx);
+ /* Clean the temporary memory. */
+ MemoryContextDelete(subctx);
+}
+
/*
* Main loop for the apply launcher process.
*/
@@ -1096,78 +1259,20 @@ ApplyLauncherMain(Datum main_arg)
*/
BackgroundWorkerInitializeConnection(NULL, NULL, 0);
+ load_file("libpqwalreceiver", false);
+
/* Enter main loop */
for (;;)
{
int rc;
- List *sublist;
- ListCell *lc;
- MemoryContext subctx;
- MemoryContext oldctx;
long wait_time = DEFAULT_NAPTIME_PER_CYCLE;
CHECK_FOR_INTERRUPTS();
- /* Use temporary context to avoid leaking memory across cycles. */
- subctx = AllocSetContextCreate(TopMemoryContext,
- "Logical Replication Launcher sublist",
- ALLOCSET_DEFAULT_SIZES);
- oldctx = MemoryContextSwitchTo(subctx);
-
- /* Start any missing workers for enabled subscriptions. */
- sublist = get_subscription_list();
- foreach(lc, sublist)
- {
- Subscription *sub = (Subscription *) lfirst(lc);
- LogicalRepWorker *w;
- TimestampTz last_start;
- TimestampTz now;
- long elapsed;
-
- if (!sub->enabled)
- continue;
-
- LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- w = logicalrep_worker_find(sub->oid, InvalidOid, false);
- LWLockRelease(LogicalRepWorkerLock);
-
- if (w != NULL)
- continue; /* worker is running already */
-
- /*
- * If the worker is eligible to start now, launch it. Otherwise,
- * adjust wait_time so that we'll wake up as soon as it can be
- * started.
- *
- * Each subscription's apply worker can only be restarted once per
- * wal_retrieve_retry_interval, so that errors do not cause us to
- * repeatedly restart the worker as fast as possible. In cases
- * where a restart is expected (e.g., subscription parameter
- * changes), another process should remove the last-start entry
- * for the subscription so that the worker can be restarted
- * without waiting for wal_retrieve_retry_interval to elapse.
- */
- last_start = ApplyLauncherGetWorkerStartTime(sub->oid);
- now = GetCurrentTimestamp();
- if (last_start == 0 ||
- (elapsed = TimestampDifferenceMilliseconds(last_start, now)) >= wal_retrieve_retry_interval)
- {
- ApplyLauncherSetWorkerStartTime(sub->oid, now);
- logicalrep_worker_launch(sub->dbid, sub->oid, sub->name,
- sub->owner, InvalidOid,
- DSM_HANDLE_INVALID);
- }
- else
- {
- wait_time = Min(wait_time,
- wal_retrieve_retry_interval - elapsed);
- }
- }
-
- /* Switch back to original memory context. */
- MemoryContextSwitchTo(oldctx);
- /* Clean the temporary memory. */
- MemoryContextDelete(subctx);
+ if (!RecoveryInProgress())
+ ApplyLauncherStartSubs(&wait_time);
+ else
+ ApplyLauncherStartSlotSync(&wait_time);
/* Wait for more work. */
rc = WaitLatch(MyLatch,
diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build
index d48cd4c590..9e52ec421f 100644
--- a/src/backend/replication/logical/meson.build
+++ b/src/backend/replication/logical/meson.build
@@ -11,6 +11,7 @@ backend_sources += files(
'proto.c',
'relation.c',
'reorderbuffer.c',
+ 'slotsync.c',
'snapbuild.c',
'tablesync.c',
'worker.c',
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 9f44974473..1519b0ec64 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -95,11 +95,14 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "replication/logical.h"
+#include "replication/logicalworker.h"
#include "replication/reorderbuffer.h"
#include "replication/slot.h"
#include "replication/snapbuild.h" /* just for SnapBuildSnapDecRefcount */
#include "storage/bufmgr.h"
#include "storage/fd.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
@@ -107,6 +110,7 @@
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
+#include "utils/varlena.h"
/* entry for a hash table we use to map from xid to our transaction state */
@@ -2053,6 +2057,85 @@ ReorderBufferResetTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
}
+static void
+wait_for_standby_confirmation(XLogRecPtr commit_lsn)
+{
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+ XLogRecPtr flush_pos = InvalidXLogRecPtr;
+
+ if (strcmp(standby_slot_names, "") == 0)
+ return;
+
+ rawname = pstrdup(standby_slot_names);
+ SplitIdentifierString(rawname, ',', &namelist);
+
+ while (true)
+ {
+ int wait_slots_remaining;
+ XLogRecPtr oldest_flush_pos = InvalidXLogRecPtr;
+ int rc;
+
+ wait_slots_remaining = list_length(namelist);
+
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (int i = 0; i < max_replication_slots; i++)
+ {
+ ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i];
+ bool inlist;
+
+ if (!s->in_use)
+ continue;
+
+ inlist = false;
+ foreach (lc, namelist)
+ {
+ char *name = lfirst(lc);
+ if (strcmp(name, NameStr(s->data.name)) == 0)
+ {
+ inlist = true;
+ break;
+ }
+ }
+ if (!inlist)
+ continue;
+
+ SpinLockAcquire(&s->mutex);
+
+ if (s->data.database == InvalidOid)
+ /* Physical slots advance restart_lsn on flush and ignore confirmed_flush_lsn */
+ flush_pos = s->data.restart_lsn;
+ else
+ /* For logical slots we must wait for commit and flush */
+ flush_pos = s->data.confirmed_flush;
+
+ SpinLockRelease(&s->mutex);
+
+ /* We want to find out the min(flush pos) over all named slots */
+ if (oldest_flush_pos == InvalidXLogRecPtr
+ || oldest_flush_pos > flush_pos)
+ oldest_flush_pos = flush_pos;
+
+ if (flush_pos >= commit_lsn && wait_slots_remaining > 0)
+ wait_slots_remaining --;
+ }
+ LWLockRelease(ReplicationSlotControlLock);
+
+ if (wait_slots_remaining == 0)
+ return;
+
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ 1000L, PG_WAIT_EXTENSION);
+
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+
+ CHECK_FOR_INTERRUPTS();
+ }
+}
+
/*
* Helper function for ReorderBufferReplay and ReorderBufferStreamTXN.
*
@@ -2502,6 +2585,9 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
* Call either PREPARE (for two-phase transactions) or COMMIT (for
* regular ones).
*/
+
+ wait_for_standby_confirmation(commit_lsn);
+
if (rbtxn_prepared(txn))
rb->prepare(rb, txn, commit_lsn);
else
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
new file mode 100644
index 0000000000..529ddb21ae
--- /dev/null
+++ b/src/backend/replication/logical/slotsync.c
@@ -0,0 +1,413 @@
+/*-------------------------------------------------------------------------
+ * slotsync.c
+ * PostgreSQL worker for synchronizing slots to a standby from primary
+ *
+ * Copyright (c) 2016-2018, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/replication/logical/slotsync.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "commands/dbcommands.h"
+#include "pgstat.h"
+#include "postmaster/bgworker.h"
+#include "replication/logicallauncher.h"
+#include "replication/logicalworker.h"
+#include "replication/walreceiver.h"
+#include "replication/worker_internal.h"
+#include "storage/ipc.h"
+#include "storage/procarray.h"
+#include "utils/builtins.h"
+#include "utils/pg_lsn.h"
+#include "utils/varlena.h"
+
+char *synchronize_slot_names;
+char *standby_slot_names;
+
+/*
+ * Wait for remote slot to pass localy reserved position.
+ */
+static void
+wait_for_primary_slot_catchup(WalReceiverConn *wrconn, char *slot_name,
+ XLogRecPtr min_lsn)
+{
+ WalRcvExecResult *res;
+ TupleTableSlot *slot;
+ Oid slotRow[1] = {LSNOID};
+ StringInfoData cmd;
+ bool isnull;
+ XLogRecPtr restart_lsn;
+
+ for (;;)
+ {
+ int rc;
+
+ CHECK_FOR_INTERRUPTS();
+
+ initStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT restart_lsn"
+ " FROM pg_catalog.pg_replication_slots"
+ " WHERE slot_name = %s",
+ quote_literal_cstr(slot_name));
+ res = walrcv_exec(wrconn, cmd.data, 1, slotRow);
+
+ if (res->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errmsg("could not fetch slot info for slot \"%s\" from primary: %s",
+ slot_name, res->err)));
+
+ slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+ if (!tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+ ereport(ERROR,
+ (errmsg("slot \"%s\" disapeared from provider",
+ slot_name)));
+
+ restart_lsn = DatumGetLSN(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ ExecClearTuple(slot);
+ walrcv_clear_result(res);
+
+ if (restart_lsn >= min_lsn)
+ break;
+
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ wal_retrieve_retry_interval,
+ WAIT_EVENT_REPL_SLOT_SYNC_MAIN);
+
+ ResetLatch(MyLatch);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+ }
+}
+
+/*
+ * Synchronize single slot to given position.
+ *
+ * This optionally creates new slot if there is no existing one.
+ */
+static void
+synchronize_one_slot(WalReceiverConn *wrconn, char *slot_name, char *database,
+ char *plugin_name, XLogRecPtr target_lsn)
+{
+ bool found = false;
+ XLogRecPtr endlsn;
+
+ /* Search for the named slot and mark it active if we find it. */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (int i = 0; i < max_replication_slots; i++)
+ {
+ ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i];
+
+ if (!s->in_use)
+ continue;
+
+ if (strcmp(NameStr(s->data.name), slot_name) == 0)
+ {
+ found = true;
+ break;
+ }
+ }
+ LWLockRelease(ReplicationSlotControlLock);
+
+ StartTransactionCommand();
+
+ /* Already existing slot, acquire */
+ if (found)
+ {
+ ReplicationSlotAcquire(slot_name, true);
+
+ if (target_lsn < MyReplicationSlot->data.confirmed_flush)
+ {
+ elog(DEBUG1,
+ "not synchronizing slot %s; synchronization would move it backward",
+ slot_name);
+
+ ReplicationSlotRelease();
+ CommitTransactionCommand();
+ return;
+ }
+ }
+ /* Otherwise create the slot first. */
+ else
+ {
+ TransactionId xmin_horizon = InvalidTransactionId;
+ ReplicationSlot *slot;
+
+ ReplicationSlotCreate(slot_name, true, RS_EPHEMERAL, false);
+ slot = MyReplicationSlot;
+
+ SpinLockAcquire(&slot->mutex);
+ slot->data.database = get_database_oid(database, false);
+ namestrcpy(&slot->data.plugin, plugin_name);
+ SpinLockRelease(&slot->mutex);
+
+ ReplicationSlotReserveWal();
+
+ LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
+ xmin_horizon = GetOldestSafeDecodingTransactionId(true);
+ slot->effective_catalog_xmin = xmin_horizon;
+ slot->data.catalog_xmin = xmin_horizon;
+ ReplicationSlotsComputeRequiredXmin(true);
+ LWLockRelease(ProcArrayLock);
+
+ if (target_lsn < MyReplicationSlot->data.restart_lsn)
+ {
+ ereport(LOG,
+ errmsg("waiting for remote slot \"%s\" LSN (%X/%X) to pass local slot LSN (%X/%X)",
+ slot_name,
+ LSN_FORMAT_ARGS(target_lsn), LSN_FORMAT_ARGS(MyReplicationSlot->data.restart_lsn)));
+
+ wait_for_primary_slot_catchup(wrconn, slot_name,
+ MyReplicationSlot->data.restart_lsn);
+ }
+
+ ReplicationSlotPersist();
+ }
+
+ endlsn = pg_logical_replication_slot_advance(target_lsn);
+
+ elog(DEBUG3, "synchronized slot %s to lsn (%X/%X)",
+ slot_name, LSN_FORMAT_ARGS(endlsn));
+
+ ReplicationSlotRelease();
+ CommitTransactionCommand();
+}
+
+static void
+synchronize_slots(void)
+{
+ WalRcvExecResult *res;
+ WalReceiverConn *wrconn = NULL;
+ TupleTableSlot *slot;
+ Oid slotRow[3] = {TEXTOID, TEXTOID, LSNOID};
+ StringInfoData s;
+ char *database;
+ char *err;
+ MemoryContext oldctx = CurrentMemoryContext;
+
+ if (!WalRcv)
+ return;
+
+ /* syscache access needs a transaction env. */
+ StartTransactionCommand();
+ /* make dbname live outside TX context */
+ MemoryContextSwitchTo(oldctx);
+
+ database = get_database_name(MyDatabaseId);
+ initStringInfo(&s);
+ appendStringInfo(&s, "%s dbname=%s", PrimaryConnInfo, database);
+ wrconn = walrcv_connect(s.data, true, false, "slot_sync", &err);
+
+ if (wrconn == NULL)
+ ereport(ERROR,
+ (errmsg("could not connect to the primary server: %s", err)));
+
+ resetStringInfo(&s);
+ appendStringInfo(&s,
+ "SELECT slot_name, plugin, confirmed_flush_lsn"
+ " FROM pg_catalog.pg_replication_slots"
+ " WHERE database = %s",
+ quote_literal_cstr(database));
+ if (strcmp(synchronize_slot_names, "") != 0 && strcmp(synchronize_slot_names, "*") != 0)
+ {
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+
+ rawname = pstrdup(synchronize_slot_names);
+ SplitIdentifierString(rawname, ',', &namelist);
+
+ appendStringInfoString(&s, " AND slot_name IN (");
+ foreach (lc, namelist)
+ {
+ if (lc != list_head(namelist))
+ appendStringInfoChar(&s, ',');
+ appendStringInfo(&s, "%s",
+ quote_literal_cstr(lfirst(lc)));
+ }
+ appendStringInfoChar(&s, ')');
+ }
+
+ res = walrcv_exec(wrconn, s.data, 3, slotRow);
+ pfree(s.data);
+
+ if (res->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errmsg("could not fetch slot info from primary: %s",
+ res->err)));
+
+ CommitTransactionCommand();
+ /* CommitTransactionCommand switches to TopMemoryContext */
+ MemoryContextSwitchTo(oldctx);
+
+ slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+ {
+ char *slot_name;
+ char *plugin_name;
+ XLogRecPtr confirmed_flush_lsn;
+ bool isnull;
+
+ slot_name = TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ plugin_name = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
+ Assert(!isnull);
+
+ confirmed_flush_lsn = DatumGetLSN(slot_getattr(slot, 3, &isnull));
+ Assert(!isnull);
+
+ synchronize_one_slot(wrconn, slot_name, database, plugin_name,
+ confirmed_flush_lsn);
+
+ ExecClearTuple(slot);
+ }
+
+ walrcv_clear_result(res);
+ pfree(database);
+
+ walrcv_disconnect(wrconn);
+}
+
+/*
+ * The main loop of our worker process.
+ */
+void
+ReplSlotSyncMain(Datum main_arg)
+{
+ int worker_slot = DatumGetInt32(main_arg);
+
+ /* Attach to slot */
+ logicalrep_worker_attach(worker_slot);
+
+ /* Establish signal handlers. */
+ BackgroundWorkerUnblockSignals();
+
+ /* Load the libpq-specific functions */
+ load_file("libpqwalreceiver", false);
+
+ /* Connect to our database. */
+ BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->userid,
+ 0);
+
+ StartTransactionCommand();
+ ereport(LOG,
+ (errmsg("replication slot synchronization worker for database \"%s\" has started",
+ get_database_name(MyLogicalRepWorker->dbid))));
+ CommitTransactionCommand();
+
+ /* Main wait loop. */
+ for (;;)
+ {
+ int rc;
+
+ CHECK_FOR_INTERRUPTS();
+
+ if (!RecoveryInProgress())
+ return;
+
+ if (strcmp(synchronize_slot_names, "") == 0)
+ return;
+
+ synchronize_slots();
+
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ wal_retrieve_retry_interval,
+ WAIT_EVENT_REPL_SLOT_SYNC_MAIN);
+
+ ResetLatch(MyLatch);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+ }
+}
+
+/*
+ * Routines for handling the GUC variable(s)
+ */
+
+bool
+check_synchronize_slot_names(char **newval, void **extra, GucSource source)
+{
+ /* Special handling for "*" which means all. */
+ if (strcmp(*newval, "*") == 0)
+ {
+ return true;
+ }
+ else
+ {
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Parse string into list of identifiers */
+ if (!SplitIdentifierString(rawname, ',', &namelist))
+ {
+ /* syntax error in name list */
+ GUC_check_errdetail("List syntax is invalid.");
+ pfree(rawname);
+ list_free(namelist);
+ return false;
+ }
+
+ foreach(lc, namelist)
+ {
+ char *curname = (char *) lfirst(lc);
+
+ ReplicationSlotValidateName(curname, ERROR);
+ }
+
+ pfree(rawname);
+ list_free(namelist);
+ }
+
+ return true;
+}
+
+
+bool
+check_standby_slot_names(char **newval, void **extra, GucSource source)
+{
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Parse string into list of identifiers */
+ if (!SplitIdentifierString(rawname, ',', &namelist))
+ {
+ /* syntax error in name list */
+ GUC_check_errdetail("List syntax is invalid.");
+ pfree(rawname);
+ list_free(namelist);
+ return false;
+ }
+
+ foreach(lc, namelist)
+ {
+ char *curname = (char *) lfirst(lc);
+
+ ReplicationSlotValidateName(curname, ERROR);
+ }
+
+ pfree(rawname);
+ list_free(namelist);
+
+ return true;
+}
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 6dce355633..2307d187e4 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -100,6 +100,7 @@
#include "catalog/pg_subscription_rel.h"
#include "catalog/pg_type.h"
#include "commands/copy.h"
+#include "commands/subscriptioncmds.h"
#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "parser/parse_relation.h"
@@ -155,7 +156,8 @@ finish_sync_worker(void)
CommitTransactionCommand();
/* Find the leader apply worker and signal it. */
- logicalrep_worker_wakeup(MyLogicalRepWorker->subid, InvalidOid);
+ logicalrep_worker_wakeup(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid, InvalidOid);
/* Stop gracefully */
proc_exit(0);
@@ -195,7 +197,8 @@ wait_for_relation_state_change(Oid relid, char expected_state)
/* Check if the sync worker is still running and bail if not. */
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- worker = logicalrep_worker_find(MyLogicalRepWorker->subid, relid,
+ worker = logicalrep_worker_find(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid, relid,
false);
LWLockRelease(LogicalRepWorkerLock);
if (!worker)
@@ -242,7 +245,8 @@ wait_for_worker_state_change(char expected_state)
* waiting.
*/
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- worker = logicalrep_worker_find(MyLogicalRepWorker->subid,
+ worker = logicalrep_worker_find(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid,
InvalidOid, false);
if (worker && worker->proc)
logicalrep_worker_wakeup_ptr(worker);
@@ -508,7 +512,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn)
*/
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- syncworker = logicalrep_worker_find(MyLogicalRepWorker->subid,
+ syncworker = logicalrep_worker_find(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid,
rstate->relid, false);
if (syncworker)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 3d58910c14..b9354bd023 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1600,7 +1600,8 @@ apply_handle_stream_start(StringInfo s)
* Signal the leader apply worker, as it may be waiting for
* us.
*/
- logicalrep_worker_wakeup(MyLogicalRepWorker->subid, InvalidOid);
+ logicalrep_worker_wakeup(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid, InvalidOid);
}
parallel_stream_nchanges = 0;
diff --git a/src/backend/replication/repl_gram.y b/src/backend/replication/repl_gram.y
index 0c874e33cf..12a4b74368 100644
--- a/src/backend/replication/repl_gram.y
+++ b/src/backend/replication/repl_gram.y
@@ -76,11 +76,12 @@ Node *replication_parse_result;
%token K_EXPORT_SNAPSHOT
%token K_NOEXPORT_SNAPSHOT
%token K_USE_SNAPSHOT
+%token K_LIST_SLOTS
%type <node> command
%type <node> base_backup start_replication start_logical_replication
create_replication_slot drop_replication_slot identify_system
- read_replication_slot timeline_history show
+ read_replication_slot timeline_history show list_slots
%type <list> generic_option_list
%type <defelt> generic_option
%type <uintval> opt_timeline
@@ -91,6 +92,7 @@ Node *replication_parse_result;
%type <boolval> opt_temporary
%type <list> create_slot_options create_slot_legacy_opt_list
%type <defelt> create_slot_legacy_opt
+%type <list> slot_name_list slot_name_list_opt
%%
@@ -114,6 +116,7 @@ command:
| read_replication_slot
| timeline_history
| show
+ | list_slots
;
/*
@@ -126,6 +129,33 @@ identify_system:
}
;
+slot_name_list:
+ IDENT
+ {
+ $$ = list_make1($1);
+ }
+ | slot_name_list ',' IDENT
+ {
+ $$ = lappend($1, $3);
+ }
+
+slot_name_list_opt:
+ slot_name_list { $$ = $1; }
+ | /* EMPTY */ { $$ = NIL; }
+ ;
+
+/*
+ * LIST_SLOTS
+ */
+list_slots:
+ K_LIST_SLOTS slot_name_list_opt
+ {
+ ListSlotsCmd *cmd = makeNode(ListSlotsCmd);
+ cmd->slot_names = $2;
+ $$ = (Node *) cmd;
+ }
+ ;
+
/*
* READ_REPLICATION_SLOT %s
*/
diff --git a/src/backend/replication/repl_scanner.l b/src/backend/replication/repl_scanner.l
index cb467ca46f..9501df38eb 100644
--- a/src/backend/replication/repl_scanner.l
+++ b/src/backend/replication/repl_scanner.l
@@ -128,6 +128,7 @@ DROP_REPLICATION_SLOT { return K_DROP_REPLICATION_SLOT; }
TIMELINE_HISTORY { return K_TIMELINE_HISTORY; }
PHYSICAL { return K_PHYSICAL; }
RESERVE_WAL { return K_RESERVE_WAL; }
+LIST_SLOTS { return K_LIST_SLOTS; }
LOGICAL { return K_LOGICAL; }
SLOT { return K_SLOT; }
TEMPORARY { return K_TEMPORARY; }
@@ -304,6 +305,7 @@ replication_scanner_is_replication_command(void)
case K_READ_REPLICATION_SLOT:
case K_TIMELINE_HISTORY:
case K_SHOW:
+ case K_LIST_SLOTS:
/* Yes; push back the first token so we can parse later. */
repl_pushed_back_token = first_token;
return true;
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 6035cf4816..83ada6db6a 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -467,7 +467,7 @@ pg_physical_replication_slot_advance(XLogRecPtr moveto)
* WAL and removal of old catalog tuples. As decoding is done in fast_forward
* mode, no changes are generated anyway.
*/
-static XLogRecPtr
+XLogRecPtr
pg_logical_replication_slot_advance(XLogRecPtr moveto)
{
LogicalDecodingContext *ctx;
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 45b8b3684f..0d01b8967a 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -473,6 +473,194 @@ IdentifySystem(void)
end_tup_output(tstate);
}
+static int
+pg_qsort_namecmp(const void *a, const void *b)
+{
+ return strncmp(NameStr(*(Name) a), NameStr(*(Name) b), NAMEDATALEN);
+}
+
+/*
+ * Handle the LIST_SLOTS command.
+ */
+static void
+ListSlots(ListSlotsCmd *cmd)
+{
+ DestReceiver *dest;
+ TupOutputState *tstate;
+ TupleDesc tupdesc;
+ NameData *slot_names;
+ int numslot_names;
+
+ numslot_names = list_length(cmd->slot_names);
+ if (numslot_names)
+ {
+ ListCell *lc;
+ int i = 0;
+
+ slot_names = palloc(numslot_names * sizeof(NameData));
+ foreach(lc, cmd->slot_names)
+ {
+ char *slot_name = lfirst(lc);
+
+ ReplicationSlotValidateName(slot_name, ERROR);
+ namestrcpy(&slot_names[i++], slot_name);
+ }
+
+ qsort(slot_names, numslot_names, sizeof(NameData), pg_qsort_namecmp);
+ }
+
+ dest = CreateDestReceiver(DestRemoteSimple);
+
+ /* need a tuple descriptor representing four columns */
+ tupdesc = CreateTemplateTupleDesc(10);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 1, "slot_name",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 2, "plugin",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 3, "slot_type",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 4, "datoid",
+ INT8OID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 5, "database",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 6, "temporary",
+ INT4OID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 7, "xmin",
+ INT8OID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 8, "catalog_xmin",
+ INT8OID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 9, "restart_lsn",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 10, "confirmed_flush",
+ TEXTOID, -1, 0);
+
+ /* prepare for projection of tuples */
+ tstate = begin_tup_output_tupdesc(dest, tupdesc, &TTSOpsVirtual);
+
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (int slotno = 0; slotno < max_replication_slots; slotno++)
+ {
+ ReplicationSlot *slot = &ReplicationSlotCtl->replication_slots[slotno];
+ char restart_lsn_str[MAXFNAMELEN];
+ char confirmed_flush_lsn_str[MAXFNAMELEN];
+ Datum values[10];
+ bool nulls[10];
+
+ ReplicationSlotPersistency persistency;
+ TransactionId xmin;
+ TransactionId catalog_xmin;
+ XLogRecPtr restart_lsn;
+ XLogRecPtr confirmed_flush_lsn;
+ Oid datoid;
+ NameData slot_name;
+ NameData plugin;
+ int i;
+ int64 tmpbigint;
+
+ if (!slot->in_use)
+ continue;
+
+ SpinLockAcquire(&slot->mutex);
+
+ xmin = slot->data.xmin;
+ catalog_xmin = slot->data.catalog_xmin;
+ datoid = slot->data.database;
+ restart_lsn = slot->data.restart_lsn;
+ confirmed_flush_lsn = slot->data.confirmed_flush;
+ namestrcpy(&slot_name, NameStr(slot->data.name));
+ namestrcpy(&plugin, NameStr(slot->data.plugin));
+ persistency = slot->data.persistency;
+
+ SpinLockRelease(&slot->mutex);
+
+ if (numslot_names &&
+ !bsearch((void *) &slot_name, (void *) slot_names,
+ numslot_names, sizeof(NameData), pg_qsort_namecmp))
+ continue;
+
+ memset(nulls, 0, sizeof(nulls));
+
+ i = 0;
+ values[i++] = CStringGetTextDatum(NameStr(slot_name));
+
+ if (datoid == InvalidOid)
+ nulls[i++] = true;
+ else
+ values[i++] = CStringGetTextDatum(NameStr(plugin));
+
+ if (datoid == InvalidOid)
+ values[i++] = CStringGetTextDatum("physical");
+ else
+ values[i++] = CStringGetTextDatum("logical");
+
+ if (datoid == InvalidOid)
+ nulls[i++] = true;
+ else
+ {
+ tmpbigint = datoid;
+ values[i++] = Int64GetDatum(tmpbigint);
+ }
+
+ if (datoid == InvalidOid)
+ nulls[i++] = true;
+ else
+ {
+ MemoryContext cur = CurrentMemoryContext;
+
+ /* syscache access needs a transaction env. */
+ StartTransactionCommand();
+ /* make dbname live outside TX context */
+ MemoryContextSwitchTo(cur);
+ values[i++] = CStringGetTextDatum(get_database_name(datoid));
+ CommitTransactionCommand();
+ /* CommitTransactionCommand switches to TopMemoryContext */
+ MemoryContextSwitchTo(cur);
+ }
+
+ values[i++] = Int32GetDatum(persistency == RS_TEMPORARY ? 1 : 0);
+
+ if (xmin != InvalidTransactionId)
+ {
+ tmpbigint = xmin;
+ values[i++] = Int64GetDatum(tmpbigint);
+ }
+ else
+ nulls[i++] = true;
+
+ if (catalog_xmin != InvalidTransactionId)
+ {
+ tmpbigint = catalog_xmin;
+ values[i++] = Int64GetDatum(tmpbigint);
+ }
+ else
+ nulls[i++] = true;
+
+ if (restart_lsn != InvalidXLogRecPtr)
+ {
+ snprintf(restart_lsn_str, sizeof(restart_lsn_str), "%X/%X",
+ LSN_FORMAT_ARGS(restart_lsn));
+ values[i++] = CStringGetTextDatum(restart_lsn_str);
+ }
+ else
+ nulls[i++] = true;
+
+ if (confirmed_flush_lsn != InvalidXLogRecPtr)
+ {
+ snprintf(confirmed_flush_lsn_str, sizeof(confirmed_flush_lsn_str),
+ "%X/%X", LSN_FORMAT_ARGS(confirmed_flush_lsn));
+ values[i++] = CStringGetTextDatum(confirmed_flush_lsn_str);
+ }
+ else
+ nulls[i++] = true;
+
+ /* send it to dest */
+ do_tup_output(tstate, values, nulls);
+ }
+ LWLockRelease(ReplicationSlotControlLock);
+
+ end_tup_output(tstate);
+}
+
/* Handle READ_REPLICATION_SLOT command */
static void
ReadReplicationSlot(ReadReplicationSlotCmd *cmd)
@@ -1820,6 +2008,13 @@ exec_replication_command(const char *cmd_string)
EndReplicationCommand(cmdtag);
break;
+ case T_ListSlotsCmd:
+ cmdtag = "LIST_SLOTS";
+ set_ps_display(cmdtag);
+ ListSlots((ListSlotsCmd *) cmd_node);
+ EndReplicationCommand(cmdtag);
+ break;
+
case T_StartReplicationCmd:
{
StartReplicationCmd *cmd = (StartReplicationCmd *) cmd_node;
diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c
index 7940d64639..f2a9517091 100644
--- a/src/backend/utils/activity/wait_event.c
+++ b/src/backend/utils/activity/wait_event.c
@@ -230,6 +230,9 @@ pgstat_get_wait_activity(WaitEventActivity w)
case WAIT_EVENT_LOGICAL_LAUNCHER_MAIN:
event_name = "LogicalLauncherMain";
break;
+ case WAIT_EVENT_REPL_SLOT_SYNC_MAIN:
+ event_name = "ReplSlotSyncMain";
+ break;
case WAIT_EVENT_LOGICAL_PARALLEL_APPLY_MAIN:
event_name = "LogicalParallelApplyMain";
break;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index cab3ddbe11..0ee7ad1348 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -63,8 +63,12 @@
#include "postmaster/syslogger.h"
#include "postmaster/walwriter.h"
#include "replication/logicallauncher.h"
+#include "replication/logicalworker.h"
+#include "replication/reorderbuffer.h"
#include "replication/slot.h"
#include "replication/syncrep.h"
+#include "replication/walreceiver.h"
+#include "replication/walsender.h"
#include "storage/bufmgr.h"
#include "storage/large_object.h"
#include "storage/pg_shmem.h"
@@ -4587,6 +4591,28 @@ struct config_string ConfigureNamesString[] =
check_io_direct, assign_io_direct, NULL
},
+ {
+ {"synchronize_slot_names", PGC_SIGHUP, REPLICATION_STANDBY,
+ gettext_noop("Sets the names of replication slots which to synchronize from primary to standby."),
+ gettext_noop("Value of \"*\" means all."),
+ GUC_LIST_INPUT | GUC_LIST_QUOTE
+ },
+ &synchronize_slot_names,
+ "",
+ check_synchronize_slot_names, NULL, NULL
+ },
+
+ {
+ {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY,
+ gettext_noop("List of physical slots that must confirm changes before changes are sent to logical replication consumers."),
+ NULL,
+ GUC_LIST_INPUT | GUC_LIST_QUOTE
+ },
+ &standby_slot_names,
+ "",
+ check_standby_slot_names, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index dce5049bc2..2ff2188c02 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -330,6 +330,7 @@
# and comma-separated list of application_name
# from standby(s); '*' = all
#vacuum_defer_cleanup_age = 0 # number of xacts by which cleanup is delayed
+#standby_slot_names = '' # physical standby slot names that logical replication waits for
# - Standby Servers -
@@ -357,6 +358,7 @@
#wal_retrieve_retry_interval = 5s # time to wait before retrying to
# retrieve WAL after a failed attempt
#recovery_min_apply_delay = 0 # minimum delay for applying changes during recovery
+#synchronize_slot_names = '' # logical replication slots to sync to standby
# - Subscribers -
diff --git a/src/include/commands/subscriptioncmds.h b/src/include/commands/subscriptioncmds.h
index 214dc6c29e..0e77f9ee5c 100644
--- a/src/include/commands/subscriptioncmds.h
+++ b/src/include/commands/subscriptioncmds.h
@@ -17,6 +17,7 @@
#include "catalog/objectaddress.h"
#include "parser/parse_node.h"
+#include "replication/walreceiver.h"
extern ObjectAddress CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
bool isTopLevel);
@@ -28,4 +29,6 @@ extern void AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId);
extern char defGetStreamingMode(DefElem *def);
+extern void ReplicationSlotDropAtPubNode(WalReceiverConn *wrconn, char *slotname, bool missing_ok);
+
#endif /* SUBSCRIPTIONCMDS_H */
diff --git a/src/include/nodes/replnodes.h b/src/include/nodes/replnodes.h
index 4321ba8f86..980e0b2ee2 100644
--- a/src/include/nodes/replnodes.h
+++ b/src/include/nodes/replnodes.h
@@ -33,6 +33,15 @@ typedef struct IdentifySystemCmd
NodeTag type;
} IdentifySystemCmd;
+/* ----------------------
+ * LIST_SLOTS command
+ * ----------------------
+ */
+typedef struct ListSlotsCmd
+{
+ NodeTag type;
+ List *slot_names;
+} ListSlotsCmd;
/* ----------------------
* BASE_BACKUP command
diff --git a/src/include/replication/logicallauncher.h b/src/include/replication/logicallauncher.h
index a07c9cb311..80fdbf9657 100644
--- a/src/include/replication/logicallauncher.h
+++ b/src/include/replication/logicallauncher.h
@@ -31,4 +31,6 @@ extern bool IsLogicalLauncher(void);
extern pid_t GetLeaderApplyWorkerPid(pid_t pid);
+extern PGDLLIMPORT char *PrimaryConnInfo;
+
#endif /* LOGICALLAUNCHER_H */
diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h
index 39588da79f..6408753557 100644
--- a/src/include/replication/logicalworker.h
+++ b/src/include/replication/logicalworker.h
@@ -14,10 +14,16 @@
#include <signal.h>
+#include "utils/guc.h"
+
+extern char *synchronize_slot_names;
+extern char *standby_slot_names;
+
extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending;
extern void ApplyWorkerMain(Datum main_arg);
extern void ParallelApplyWorkerMain(Datum main_arg);
+extern void ReplSlotSyncMain(Datum main_arg);
extern bool IsLogicalWorker(void);
extern bool IsLogicalParallelApplyWorker(void);
@@ -29,4 +35,7 @@ extern void LogicalRepWorkersWakeupAtCommit(Oid subid);
extern void AtEOXact_LogicalRepWorkers(bool isCommit);
+extern bool check_synchronize_slot_names(char **newval, void **extra, GucSource source);
+extern bool check_standby_slot_names(char **newval, void **extra, GucSource source);
+
#endif /* LOGICALWORKER_H */
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index a8a89dc784..5dc2e0d30d 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -15,7 +15,6 @@
#include "storage/lwlock.h"
#include "storage/shmem.h"
#include "storage/spin.h"
-#include "replication/walreceiver.h"
/*
* Behaviour of replication slots, upon release or crash.
@@ -238,7 +237,6 @@ extern ReplicationSlot *SearchNamedReplicationSlot(const char *name, bool need_l
extern int ReplicationSlotIndex(ReplicationSlot *slot);
extern bool ReplicationSlotName(int index, Name name);
extern void ReplicationSlotNameForTablesync(Oid suboid, Oid relid, char *syncslotname, Size szslot);
-extern void ReplicationSlotDropAtPubNode(WalReceiverConn *wrconn, char *slotname, bool missing_ok);
extern void StartupReplicationSlots(void);
extern void CheckPointReplicationSlots(void);
@@ -246,4 +244,7 @@ extern void CheckPointReplicationSlots(void);
extern void CheckSlotRequirements(void);
extern void CheckSlotPermissions(void);
+extern XLogRecPtr pg_logical_replication_slot_advance(XLogRecPtr moveto);
+
+
#endif /* SLOT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 281626fa6f..9e9d64faf2 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -20,6 +20,7 @@
#include "pgtime.h"
#include "port/atomics.h"
#include "replication/logicalproto.h"
+#include "replication/slot.h"
#include "replication/walsender.h"
#include "storage/condition_variable.h"
#include "storage/latch.h"
@@ -191,6 +192,17 @@ typedef struct
} proto;
} WalRcvStreamOptions;
+/*
+ * Slot information receiver from remote.
+ *
+ * Currently same as ReplicationSlotPersistentData except last_sync_time
+ */
+typedef struct WalRecvReplicationSlotData
+{
+ ReplicationSlotPersistentData persistent_data;
+ TimestampTz last_sync_time;
+} WalRecvReplicationSlotData;
+
struct WalReceiverConn;
typedef struct WalReceiverConn WalReceiverConn;
@@ -280,6 +292,11 @@ typedef void (*walrcv_get_senderinfo_fn) (WalReceiverConn *conn,
typedef char *(*walrcv_identify_system_fn) (WalReceiverConn *conn,
TimeLineID *primary_tli);
+/*
+ * TODO
+ */
+typedef List *(*walrcv_list_slots_fn) (WalReceiverConn *conn, const char *slots);
+
/*
* walrcv_server_version_fn
*
@@ -393,6 +410,7 @@ typedef struct WalReceiverFunctionsType
walrcv_get_conninfo_fn walrcv_get_conninfo;
walrcv_get_senderinfo_fn walrcv_get_senderinfo;
walrcv_identify_system_fn walrcv_identify_system;
+ walrcv_list_slots_fn walrcv_list_slots;
walrcv_server_version_fn walrcv_server_version;
walrcv_readtimelinehistoryfile_fn walrcv_readtimelinehistoryfile;
walrcv_startstreaming_fn walrcv_startstreaming;
@@ -417,6 +435,8 @@ extern PGDLLIMPORT WalReceiverFunctionsType *WalReceiverFunctions;
WalReceiverFunctions->walrcv_get_senderinfo(conn, sender_host, sender_port)
#define walrcv_identify_system(conn, primary_tli) \
WalReceiverFunctions->walrcv_identify_system(conn, primary_tli)
+#define walrcv_list_slots(conn, slots) \
+ WalReceiverFunctions->walrcv_list_slots(conn, slots)
#define walrcv_server_version(conn) \
WalReceiverFunctions->walrcv_server_version(conn)
#define walrcv_readtimelinehistoryfile(conn, tli, filename, content, size) \
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index dce71d2c50..5b4fda2fd9 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -65,7 +65,7 @@ typedef struct LogicalRepWorker
* would be created for each transaction which will be deleted after the
* transaction is finished.
*/
- FileSet *stream_fileset;
+ struct FileSet *stream_fileset;
/*
* PID of leader apply worker if this slot is used for a parallel apply
@@ -226,15 +226,15 @@ extern PGDLLIMPORT LogicalRepWorker *MyLogicalRepWorker;
extern PGDLLIMPORT bool in_remote_transaction;
extern void logicalrep_worker_attach(int slot);
-extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid,
+extern LogicalRepWorker *logicalrep_worker_find(Oid dbid, Oid subid, Oid relid,
bool only_running);
extern List *logicalrep_workers_find(Oid subid, bool only_running);
extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname,
Oid userid, Oid relid,
dsm_handle subworker_dsm);
-extern void logicalrep_worker_stop(Oid subid, Oid relid);
+extern void logicalrep_worker_stop(Oid dbid, Oid subid, Oid relid);
extern void logicalrep_pa_worker_stop(int slot_no, uint16 generation);
-extern void logicalrep_worker_wakeup(Oid subid, Oid relid);
+extern void logicalrep_worker_wakeup(Oid dbid, Oid subid, Oid relid);
extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker);
extern int logicalrep_sync_worker_count(Oid subid);
diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h
index 518d3b0a1f..cccd4d9d32 100644
--- a/src/include/utils/wait_event.h
+++ b/src/include/utils/wait_event.h
@@ -43,6 +43,7 @@ typedef enum
WAIT_EVENT_LOGICAL_APPLY_MAIN,
WAIT_EVENT_LOGICAL_LAUNCHER_MAIN,
WAIT_EVENT_LOGICAL_PARALLEL_APPLY_MAIN,
+ WAIT_EVENT_REPL_SLOT_SYNC_MAIN,
WAIT_EVENT_RECOVERY_WAL_STREAM,
WAIT_EVENT_SYSLOGGER_MAIN,
WAIT_EVENT_WAL_RECEIVER_MAIN,
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index 2008958010..b6fcc8704e 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -42,6 +42,7 @@ tests += {
't/034_create_database.pl',
't/035_standby_logical_decoding.pl',
't/036_truncated_dropped.pl',
+ 't/037_slot_sync.pl',
],
},
}
diff --git a/src/test/recovery/t/037_slot_sync.pl b/src/test/recovery/t/037_slot_sync.pl
new file mode 100644
index 0000000000..0520042d96
--- /dev/null
+++ b/src/test/recovery/t/037_slot_sync.pl
@@ -0,0 +1,130 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+my $node_phys_standby = PostgreSQL::Test::Cluster->new('phys_standby');
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+
+# find $pat in logfile of $node after $off-th byte
+sub find_in_log
+{
+ my ($node, $pat, $off) = @_;
+
+ $off = 0 unless defined $off;
+ my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile);
+ return 0 if (length($log) <= $off);
+
+ $log = substr($log, $off);
+
+ return $log =~ m/$pat/;
+}
+
+# Check invalidation in the logfile
+sub check_for_invalidation
+{
+ my ($log_start, $test_name) = @_;
+
+ # message should be issued
+ ok( find_in_log(
+ $node_phys_standby,
+ "invalidating obsolete replication slot \"sub1\"", $log_start),
+ "sub1 slot invalidation is logged $test_name");
+}
+
+# Check conflicting status in pg_replication_slots.
+sub check_slots_conflicting_status
+{
+ my $res = $node_phys_standby->safe_psql(
+ 'postgres', qq(
+ select bool_and(conflicting) from pg_replication_slots;));
+
+ is($res, 't',
+ "Logical slot is reported as conflicting");
+}
+
+$node_primary->init(allows_streaming => 'logical');
+$node_primary->append_conf('postgresql.conf', "standby_slot_names = 'pslot1'");
+$node_primary->start;
+$node_primary->psql('postgres', q{SELECT pg_create_physical_replication_slot('pslot1');});
+
+$node_primary->backup('backup');
+
+$node_phys_standby->init_from_backup($node_primary, 'backup', has_streaming => 1);
+$node_phys_standby->append_conf('postgresql.conf', q{
+synchronize_slot_names = '*'
+primary_slot_name = 'pslot1'
+hot_standby_feedback = off
+});
+
+$node_phys_standby->start;
+
+$node_primary->safe_psql('postgres', "CREATE TABLE t1 (a int PRIMARY KEY)");
+$node_primary->safe_psql('postgres', "INSERT INTO t1 VALUES (1), (2), (3)");
+
+# Some tests need to wait for VACUUM to be replayed. But vacuum does not flush
+# WAL. An insert into flush_wal outside transaction does guarantee a flush.
+$node_primary->psql('postgres', q[CREATE TABLE flush_wal();]);
+
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+$node_subscriber->safe_psql('postgres', "CREATE TABLE t1 (a int PRIMARY KEY)");
+
+$node_primary->safe_psql('postgres', "CREATE PUBLICATION pub1 FOR TABLE t1");
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1 CONNECTION '" . ($node_primary->connstr . ' dbname=postgres') . "' PUBLICATION pub1");
+
+# Wait for initial sync of all subscriptions
+my $synced_query =
+ "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result = $node_primary->safe_psql('postgres',
+ "SELECT slot_name, plugin, database FROM pg_replication_slots WHERE slot_type = 'logical'");
+
+is($result, qq(sub1|pgoutput|postgres), 'logical slot on primary');
+
+# FIXME: standby needs restart to pick up new slots
+$node_phys_standby->restart;
+sleep 3;
+
+$result = $node_phys_standby->safe_psql('postgres',
+ "SELECT slot_name, plugin, database FROM pg_replication_slots");
+
+is($result, qq(sub1|pgoutput|postgres), 'logical slot on standby');
+
+$node_primary->safe_psql('postgres', "INSERT INTO t1 VALUES (4), (5), (6)");
+$node_primary->wait_for_catchup('sub1');
+
+$node_primary->wait_for_catchup($node_phys_standby->name);
+
+# Logical subscriber and physical replica are caught up at this point.
+
+# Drop the subscription so that catalog_xmin is unknown on the primary
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub1");
+
+# This should trigger a conflict as hot_standby_feedback is off on the standby
+$node_primary->safe_psql('postgres', qq[
+ CREATE TABLE conflict_test(x integer, y text);
+ DROP TABLE conflict_test;
+ VACUUM full pg_class;
+ INSERT INTO flush_wal DEFAULT VALUES; -- see create table flush_wal
+]);
+
+# Ensure physical replay catches up
+$node_primary->wait_for_catchup($node_phys_standby);
+
+# Check invalidation in the logfile
+check_for_invalidation(1, 'with vacuum FULL on pg_class');
+
+# Check conflicting status in pg_replication_slots.
+check_slots_conflicting_status();
+
+done_testing();
--
2.34.1
Attachments:
[text/plain] v5-0001-Synchronize-logical-replication-slots-from-primar.patch (62.6K, ../../[email protected]/2-v5-0001-Synchronize-logical-replication-slots-from-primar.patch)
download | inline diff:
From 655359eedf37d8f2e522aeb1ec8c48adfc1759b1 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <[email protected]>
Date: Thu, 13 Apr 2023 11:32:28 +0000
Subject: [PATCH v5] Synchronize logical replication slots from primary to
standby
---
doc/src/sgml/config.sgml | 34 ++
src/backend/commands/subscriptioncmds.c | 4 +-
src/backend/postmaster/bgworker.c | 3 +
.../libpqwalreceiver/libpqwalreceiver.c | 95 ++++
src/backend/replication/logical/Makefile | 1 +
src/backend/replication/logical/launcher.c | 263 +++++++----
src/backend/replication/logical/meson.build | 1 +
.../replication/logical/reorderbuffer.c | 86 ++++
src/backend/replication/logical/slotsync.c | 413 ++++++++++++++++++
src/backend/replication/logical/tablesync.c | 13 +-
src/backend/replication/logical/worker.c | 3 +-
src/backend/replication/repl_gram.y | 32 +-
src/backend/replication/repl_scanner.l | 2 +
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 195 +++++++++
src/backend/utils/activity/wait_event.c | 3 +
src/backend/utils/misc/guc_tables.c | 26 ++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/commands/subscriptioncmds.h | 3 +
src/include/nodes/replnodes.h | 9 +
src/include/replication/logicallauncher.h | 2 +
src/include/replication/logicalworker.h | 9 +
src/include/replication/slot.h | 5 +-
src/include/replication/walreceiver.h | 20 +
src/include/replication/worker_internal.h | 8 +-
src/include/utils/wait_event.h | 1 +
src/test/recovery/meson.build | 1 +
src/test/recovery/t/037_slot_sync.pl | 130 ++++++
28 files changed, 1272 insertions(+), 94 deletions(-)
3.8% doc/src/sgml/
7.1% src/backend/replication/libpqwalreceiver/
54.7% src/backend/replication/logical/
14.9% src/backend/replication/
3.3% src/backend/
4.0% src/include/replication/
10.9% src/test/recovery/t/
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 091a79d4f3..1360885208 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4466,6 +4466,23 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-standby-slot-names" xreflabel="standby_slot_names">
+ <term><varname>standby_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>standby_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ List of physical replication slots that logical replication waits for.
+ If a logical replication connection is meant to switch to a physical
+ standby after the standby is promoted, the physical replication slot
+ for the standby should be listed here. This ensures that logical
+ replication is not ahead of the physical standby.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
@@ -4649,6 +4666,23 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
</listitem>
</varlistentry>
+ <varlistentry id="guc-synchronize_slot_names" xreflabel="synchronize_slot_names">
+ <term><varname>synchronize_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>synchronize_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies a list of logical replication slots that a physical standby
+ should synchronize from the primary server. This is necessary to be
+ able to retarget those logical replication connections to this standby
+ if it gets promoted. Specify <literal>*</literal> to synchronize all
+ logical replication slots. The default is empty.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 3251d89ba8..8721706b79 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -991,7 +991,7 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data,
RemoveSubscriptionRel(sub->oid, relid);
- logicalrep_worker_stop(sub->oid, relid);
+ logicalrep_worker_stop(MyDatabaseId, sub->oid, relid);
/*
* For READY state, we would have already dropped the
@@ -1589,7 +1589,7 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel)
{
LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc);
- logicalrep_worker_stop(w->subid, w->relid);
+ logicalrep_worker_stop(w->dbid, w->subid, w->relid);
}
list_free(subworkers);
diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c
index 0dd22b2351..a89d1f10a1 100644
--- a/src/backend/postmaster/bgworker.c
+++ b/src/backend/postmaster/bgworker.c
@@ -129,6 +129,9 @@ static const struct
{
"ApplyWorkerMain", ApplyWorkerMain
},
+ {
+ "ReplSlotSyncMain", ReplSlotSyncMain
+ },
{
"ParallelApplyWorkerMain", ParallelApplyWorkerMain
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 052505e46f..4f7417c49a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -34,6 +34,7 @@
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
#include "utils/tuplestore.h"
+#include "utils/varlena.h"
PG_MODULE_MAGIC;
@@ -58,6 +59,7 @@ static void libpqrcv_get_senderinfo(WalReceiverConn *conn,
char **sender_host, int *sender_port);
static char *libpqrcv_identify_system(WalReceiverConn *conn,
TimeLineID *primary_tli);
+static List *libpqrcv_list_slots(WalReceiverConn *conn, const char *slot_names);
static int libpqrcv_server_version(WalReceiverConn *conn);
static void libpqrcv_readtimelinehistoryfile(WalReceiverConn *conn,
TimeLineID tli, char **filename,
@@ -96,6 +98,7 @@ static WalReceiverFunctionsType PQWalReceiverFunctions = {
.walrcv_receive = libpqrcv_receive,
.walrcv_send = libpqrcv_send,
.walrcv_create_slot = libpqrcv_create_slot,
+ .walrcv_list_slots = libpqrcv_list_slots,
.walrcv_get_backend_pid = libpqrcv_get_backend_pid,
.walrcv_exec = libpqrcv_exec,
.walrcv_disconnect = libpqrcv_disconnect
@@ -409,6 +412,98 @@ libpqrcv_server_version(WalReceiverConn *conn)
return PQserverVersion(conn->streamConn);
}
+/*
+ * Get list of slots from primary.
+ */
+static List *
+libpqrcv_list_slots(WalReceiverConn *conn, const char *slot_names)
+{
+ PGresult *res;
+ List *slotlist = NIL;
+ int ntuples;
+ StringInfoData s;
+ WalRecvReplicationSlotData *slot_data;
+
+ initStringInfo(&s);
+ appendStringInfoString(&s, "LIST_SLOTS");
+
+ if (strcmp(slot_names, "") != 0 && strcmp(slot_names, "*") != 0)
+ {
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+
+ appendStringInfoChar(&s, ' ');
+ rawname = pstrdup(slot_names);
+ SplitIdentifierString(rawname, ',', &namelist);
+ foreach (lc, namelist)
+ {
+ if (lc != list_head(namelist))
+ appendStringInfoChar(&s, ',');
+ appendStringInfo(&s, "%s",
+ quote_identifier(lfirst(lc)));
+ }
+ }
+
+ res = libpqrcv_PQexec(conn->streamConn, s.data);
+ pfree(s.data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ PQclear(res);
+ ereport(ERROR,
+ (errmsg("could not receive list of slots the primary server: %s",
+ pchomp(PQerrorMessage(conn->streamConn)))));
+ }
+ if (PQnfields(res) < 10)
+ {
+ int nfields = PQnfields(res);
+
+ PQclear(res);
+ ereport(ERROR,
+ (errmsg("invalid response from primary server"),
+ errdetail("Could not get list of slots: got %d fields, expected %d or more fields.",
+ nfields, 10)));
+ }
+
+ ntuples = PQntuples(res);
+ for (int i = 0; i < ntuples; i++)
+ {
+ char *slot_type;
+
+ slot_data = palloc0(sizeof(WalRecvReplicationSlotData));
+ namestrcpy(&slot_data->persistent_data.name, PQgetvalue(res, i, 0));
+ if (!PQgetisnull(res, i, 1))
+ namestrcpy(&slot_data->persistent_data.plugin, PQgetvalue(res, i, 1));
+ slot_type = PQgetvalue(res, i, 2);
+ if (!PQgetisnull(res, i, 3))
+ slot_data->persistent_data.database = atooid(PQgetvalue(res, i, 3));
+ if (strcmp(slot_type, "physical") == 0)
+ {
+ if (OidIsValid(slot_data->persistent_data.database))
+ elog(ERROR, "unexpected physical replication slot with database set");
+ }
+ if (pg_strtoint32(PQgetvalue(res, i, 5)) == 1)
+ slot_data->persistent_data.persistency = RS_TEMPORARY;
+ else
+ slot_data->persistent_data.persistency = RS_PERSISTENT;
+ if (!PQgetisnull(res, i, 6))
+ slot_data->persistent_data.xmin = atooid(PQgetvalue(res, i, 6));
+ if (!PQgetisnull(res, i, 7))
+ slot_data->persistent_data.catalog_xmin = atooid(PQgetvalue(res, i, 7));
+ if (!PQgetisnull(res, i, 8))
+ slot_data->persistent_data.restart_lsn = strtou64(PQgetvalue(res, i, 8), NULL, 10);
+ if (!PQgetisnull(res, i, 9))
+ slot_data->persistent_data.confirmed_flush = strtou64(PQgetvalue(res, i, 9), NULL, 10);
+
+ slot_data->last_sync_time = 0;
+ slotlist = lappend(slotlist, slot_data);
+ }
+
+ PQclear(res);
+
+ return slotlist;
+}
+
/*
* Start streaming WAL data from given streaming options.
*
diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile
index 2dc25e37bb..ba03eeff1c 100644
--- a/src/backend/replication/logical/Makefile
+++ b/src/backend/replication/logical/Makefile
@@ -25,6 +25,7 @@ OBJS = \
proto.o \
relation.o \
reorderbuffer.o \
+ slotsync.o \
snapbuild.o \
tablesync.o \
worker.o
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 970d170e73..14af724639 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -22,6 +22,7 @@
#include "access/htup_details.h"
#include "access/tableam.h"
#include "access/xact.h"
+#include "catalog/pg_authid.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
#include "funcapi.h"
@@ -246,7 +247,7 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker,
* We are only interested in the leader apply worker or table sync worker.
*/
LogicalRepWorker *
-logicalrep_worker_find(Oid subid, Oid relid, bool only_running)
+logicalrep_worker_find(Oid dbid, Oid subid, Oid relid, bool only_running)
{
int i;
LogicalRepWorker *res = NULL;
@@ -262,8 +263,8 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running)
if (isParallelApplyWorker(w))
continue;
- if (w->in_use && w->subid == subid && w->relid == relid &&
- (!only_running || w->proc))
+ if (w->in_use && w->dbid == dbid && w->subid == subid &&
+ w->relid == relid && (!only_running || w->proc))
{
res = w;
break;
@@ -320,9 +321,13 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid,
/* Sanity check - tablesync worker cannot be a subworker */
Assert(!(is_parallel_apply_worker && OidIsValid(relid)));
- ereport(DEBUG1,
- (errmsg_internal("starting logical replication worker for subscription \"%s\"",
- subname)));
+ if (OidIsValid(subid))
+ ereport(DEBUG1,
+ (errmsg_internal("starting logical replication worker for subscription \"%s\"",
+ subname)));
+ else
+ ereport(DEBUG1,
+ (errmsg_internal("starting replication slot synchronization worker")));
/* Report this after the initial starting message for consistency. */
if (max_replication_slots == 0)
@@ -359,7 +364,9 @@ retry:
* reason we do this is because if some worker failed to start up and its
* parent has crashed while waiting, the in_use state was never cleared.
*/
- if (worker == NULL || nsyncworkers >= max_sync_workers_per_subscription)
+ if (worker == NULL ||
+ (OidIsValid(relid) &&
+ nsyncworkers >= max_sync_workers_per_subscription))
{
bool did_cleanup = false;
@@ -455,15 +462,20 @@ retry:
memset(&bgw, 0, sizeof(bgw));
bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
BGWORKER_BACKEND_DATABASE_CONNECTION;
- bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ bgw.bgw_start_time = BgWorkerStart_ConsistentState;
snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres");
- if (is_parallel_apply_worker)
+ if (!OidIsValid(subid))
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ReplSlotSyncMain");
+ else if (is_parallel_apply_worker)
snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ParallelApplyWorkerMain");
else
snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain");
- if (OidIsValid(relid))
+ if (!OidIsValid(subid))
+ snprintf(bgw.bgw_name, BGW_MAXLEN,
+ "replication slot synchronization worker");
+ else if (OidIsValid(relid))
snprintf(bgw.bgw_name, BGW_MAXLEN,
"logical replication worker for subscription %u sync %u", subid, relid);
else if (is_parallel_apply_worker)
@@ -591,13 +603,13 @@ logicalrep_worker_stop_internal(LogicalRepWorker *worker, int signo)
* Stop the logical replication worker for subid/relid, if any.
*/
void
-logicalrep_worker_stop(Oid subid, Oid relid)
+logicalrep_worker_stop(Oid dbid, Oid subid, Oid relid)
{
LogicalRepWorker *worker;
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- worker = logicalrep_worker_find(subid, relid, false);
+ worker = logicalrep_worker_find(dbid, subid, relid, false);
if (worker)
{
@@ -640,13 +652,13 @@ logicalrep_pa_worker_stop(int slot_no, uint16 generation)
* Wake up (using latch) any logical replication worker for specified sub/rel.
*/
void
-logicalrep_worker_wakeup(Oid subid, Oid relid)
+logicalrep_worker_wakeup(Oid dbid, Oid subid, Oid relid)
{
LogicalRepWorker *worker;
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- worker = logicalrep_worker_find(subid, relid, true);
+ worker = logicalrep_worker_find(dbid, subid, relid, true);
if (worker)
logicalrep_worker_wakeup_ptr(worker);
@@ -888,7 +900,7 @@ ApplyLauncherRegister(void)
memset(&bgw, 0, sizeof(bgw));
bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
BGWORKER_BACKEND_DATABASE_CONNECTION;
- bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ bgw.bgw_start_time = BgWorkerStart_ConsistentState;
snprintf(bgw.bgw_library_name, BGW_MAXLEN, "postgres");
snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyLauncherMain");
snprintf(bgw.bgw_name, BGW_MAXLEN,
@@ -1071,6 +1083,157 @@ ApplyLauncherWakeup(void)
kill(LogicalRepCtx->launcher_pid, SIGUSR1);
}
+static void
+ApplyLauncherStartSlotSync(long *wait_time)
+{
+ WalReceiverConn *wrconn;
+ char *err;
+ List *slots;
+ ListCell *lc;
+ MemoryContext tmpctx;
+ MemoryContext oldctx;
+
+ if (strcmp(synchronize_slot_names, "") == 0)
+ return;
+
+ wrconn = walrcv_connect(PrimaryConnInfo, false, false,
+ "Logical Replication Launcher", &err);
+ if (!wrconn)
+ ereport(ERROR,
+ (errmsg("could not connect to the primary server: %s", err)));
+
+ /* Use temporary context for the slot list and worker info. */
+ tmpctx = AllocSetContextCreate(TopMemoryContext,
+ "Logical Replication Launcher slot sync ctx",
+ ALLOCSET_DEFAULT_SIZES);
+ oldctx = MemoryContextSwitchTo(tmpctx);
+
+ slots = walrcv_list_slots(wrconn, synchronize_slot_names);
+
+ foreach(lc, slots)
+ {
+ WalRecvReplicationSlotData *slot_data = lfirst(lc);
+ LogicalRepWorker *w;
+ TimestampTz last_sync;
+ TimestampTz now;
+ long elapsed;
+
+ if (!OidIsValid(slot_data->persistent_data.database))
+ continue;
+
+ LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
+ w = logicalrep_worker_find(slot_data->persistent_data.database, InvalidOid,
+ InvalidOid, false);
+ LWLockRelease(LogicalRepWorkerLock);
+
+ if (w != NULL)
+ continue; /* worker is running already */
+
+ /*
+ * If the worker is eligible to start now, launch it. Otherwise,
+ * adjust wait_time so that we'll wake up as soon as it can be
+ * started.
+ *
+ * Each apply worker can only be restarted once per
+ * wal_retrieve_retry_interval, so that errors do not cause us to
+ * repeatedly restart the worker as fast as possible.
+ */
+ last_sync = slot_data->last_sync_time;
+ now = GetCurrentTimestamp();
+ if (last_sync == 0 ||
+ (elapsed = TimestampDifferenceMilliseconds(last_sync, now)) >= wal_retrieve_retry_interval)
+ {
+ slot_data->last_sync_time = now;
+ logicalrep_worker_launch(slot_data->persistent_data.database,
+ InvalidOid, NULL,
+ BOOTSTRAP_SUPERUSERID, InvalidOid,
+ DSM_HANDLE_INVALID);
+ }
+ else
+ {
+ *wait_time = Min(*wait_time,
+ wal_retrieve_retry_interval - elapsed);
+ }
+ }
+
+ /* Switch back to original memory context. */
+ MemoryContextSwitchTo(oldctx);
+ /* Clean the temporary memory. */
+ MemoryContextDelete(tmpctx);
+
+ walrcv_disconnect(wrconn);
+}
+
+static void
+ApplyLauncherStartSubs(long *wait_time)
+{
+ List *sublist;
+ ListCell *lc;
+ MemoryContext subctx;
+ MemoryContext oldctx;
+
+ /* Use temporary context to avoid leaking memory across cycles. */
+ subctx = AllocSetContextCreate(TopMemoryContext,
+ "Logical Replication Launcher sublist",
+ ALLOCSET_DEFAULT_SIZES);
+ oldctx = MemoryContextSwitchTo(subctx);
+
+ /* Start the missing workers for enabled subscriptions. */
+ sublist = get_subscription_list();
+ foreach(lc, sublist)
+ {
+ Subscription *sub = (Subscription *) lfirst(lc);
+ LogicalRepWorker *w;
+ TimestampTz last_start;
+ TimestampTz now;
+ long elapsed;
+
+ if (!sub->enabled)
+ continue;
+
+ LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
+ w = logicalrep_worker_find(sub->dbid, sub->oid, InvalidOid, false);
+ LWLockRelease(LogicalRepWorkerLock);
+
+ if (w != NULL)
+ continue; /* worker is running already */
+
+ /*
+ * If the worker is eligible to start now, launch it. Otherwise,
+ * adjust wait_time so that we'll wake up as soon as it can be
+ * started.
+ *
+ * Each subscription's apply worker can only be restarted once per
+ * wal_retrieve_retry_interval, so that errors do not cause us to
+ * repeatedly restart the worker as fast as possible. In cases
+ * where a restart is expected (e.g., subscription parameter
+ * changes), another process should remove the last-start entry
+ * for the subscription so that the worker can be restarted
+ * without waiting for wal_retrieve_retry_interval to elapse.
+ */
+ last_start = ApplyLauncherGetWorkerStartTime(sub->oid);
+ now = GetCurrentTimestamp();
+ if (last_start == 0 ||
+ (elapsed = TimestampDifferenceMilliseconds(last_start, now)) >= wal_retrieve_retry_interval)
+ {
+ ApplyLauncherSetWorkerStartTime(sub->oid, now);
+ logicalrep_worker_launch(sub->dbid, sub->oid, sub->name,
+ sub->owner, InvalidOid,
+ DSM_HANDLE_INVALID);
+ }
+ else
+ {
+ *wait_time = Min(*wait_time,
+ wal_retrieve_retry_interval - elapsed);
+ }
+ }
+
+ /* Switch back to original memory context. */
+ MemoryContextSwitchTo(oldctx);
+ /* Clean the temporary memory. */
+ MemoryContextDelete(subctx);
+}
+
/*
* Main loop for the apply launcher process.
*/
@@ -1096,78 +1259,20 @@ ApplyLauncherMain(Datum main_arg)
*/
BackgroundWorkerInitializeConnection(NULL, NULL, 0);
+ load_file("libpqwalreceiver", false);
+
/* Enter main loop */
for (;;)
{
int rc;
- List *sublist;
- ListCell *lc;
- MemoryContext subctx;
- MemoryContext oldctx;
long wait_time = DEFAULT_NAPTIME_PER_CYCLE;
CHECK_FOR_INTERRUPTS();
- /* Use temporary context to avoid leaking memory across cycles. */
- subctx = AllocSetContextCreate(TopMemoryContext,
- "Logical Replication Launcher sublist",
- ALLOCSET_DEFAULT_SIZES);
- oldctx = MemoryContextSwitchTo(subctx);
-
- /* Start any missing workers for enabled subscriptions. */
- sublist = get_subscription_list();
- foreach(lc, sublist)
- {
- Subscription *sub = (Subscription *) lfirst(lc);
- LogicalRepWorker *w;
- TimestampTz last_start;
- TimestampTz now;
- long elapsed;
-
- if (!sub->enabled)
- continue;
-
- LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- w = logicalrep_worker_find(sub->oid, InvalidOid, false);
- LWLockRelease(LogicalRepWorkerLock);
-
- if (w != NULL)
- continue; /* worker is running already */
-
- /*
- * If the worker is eligible to start now, launch it. Otherwise,
- * adjust wait_time so that we'll wake up as soon as it can be
- * started.
- *
- * Each subscription's apply worker can only be restarted once per
- * wal_retrieve_retry_interval, so that errors do not cause us to
- * repeatedly restart the worker as fast as possible. In cases
- * where a restart is expected (e.g., subscription parameter
- * changes), another process should remove the last-start entry
- * for the subscription so that the worker can be restarted
- * without waiting for wal_retrieve_retry_interval to elapse.
- */
- last_start = ApplyLauncherGetWorkerStartTime(sub->oid);
- now = GetCurrentTimestamp();
- if (last_start == 0 ||
- (elapsed = TimestampDifferenceMilliseconds(last_start, now)) >= wal_retrieve_retry_interval)
- {
- ApplyLauncherSetWorkerStartTime(sub->oid, now);
- logicalrep_worker_launch(sub->dbid, sub->oid, sub->name,
- sub->owner, InvalidOid,
- DSM_HANDLE_INVALID);
- }
- else
- {
- wait_time = Min(wait_time,
- wal_retrieve_retry_interval - elapsed);
- }
- }
-
- /* Switch back to original memory context. */
- MemoryContextSwitchTo(oldctx);
- /* Clean the temporary memory. */
- MemoryContextDelete(subctx);
+ if (!RecoveryInProgress())
+ ApplyLauncherStartSubs(&wait_time);
+ else
+ ApplyLauncherStartSlotSync(&wait_time);
/* Wait for more work. */
rc = WaitLatch(MyLatch,
diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build
index d48cd4c590..9e52ec421f 100644
--- a/src/backend/replication/logical/meson.build
+++ b/src/backend/replication/logical/meson.build
@@ -11,6 +11,7 @@ backend_sources += files(
'proto.c',
'relation.c',
'reorderbuffer.c',
+ 'slotsync.c',
'snapbuild.c',
'tablesync.c',
'worker.c',
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 9f44974473..1519b0ec64 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -95,11 +95,14 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "replication/logical.h"
+#include "replication/logicalworker.h"
#include "replication/reorderbuffer.h"
#include "replication/slot.h"
#include "replication/snapbuild.h" /* just for SnapBuildSnapDecRefcount */
#include "storage/bufmgr.h"
#include "storage/fd.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
@@ -107,6 +110,7 @@
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
+#include "utils/varlena.h"
/* entry for a hash table we use to map from xid to our transaction state */
@@ -2053,6 +2057,85 @@ ReorderBufferResetTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
}
+static void
+wait_for_standby_confirmation(XLogRecPtr commit_lsn)
+{
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+ XLogRecPtr flush_pos = InvalidXLogRecPtr;
+
+ if (strcmp(standby_slot_names, "") == 0)
+ return;
+
+ rawname = pstrdup(standby_slot_names);
+ SplitIdentifierString(rawname, ',', &namelist);
+
+ while (true)
+ {
+ int wait_slots_remaining;
+ XLogRecPtr oldest_flush_pos = InvalidXLogRecPtr;
+ int rc;
+
+ wait_slots_remaining = list_length(namelist);
+
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (int i = 0; i < max_replication_slots; i++)
+ {
+ ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i];
+ bool inlist;
+
+ if (!s->in_use)
+ continue;
+
+ inlist = false;
+ foreach (lc, namelist)
+ {
+ char *name = lfirst(lc);
+ if (strcmp(name, NameStr(s->data.name)) == 0)
+ {
+ inlist = true;
+ break;
+ }
+ }
+ if (!inlist)
+ continue;
+
+ SpinLockAcquire(&s->mutex);
+
+ if (s->data.database == InvalidOid)
+ /* Physical slots advance restart_lsn on flush and ignore confirmed_flush_lsn */
+ flush_pos = s->data.restart_lsn;
+ else
+ /* For logical slots we must wait for commit and flush */
+ flush_pos = s->data.confirmed_flush;
+
+ SpinLockRelease(&s->mutex);
+
+ /* We want to find out the min(flush pos) over all named slots */
+ if (oldest_flush_pos == InvalidXLogRecPtr
+ || oldest_flush_pos > flush_pos)
+ oldest_flush_pos = flush_pos;
+
+ if (flush_pos >= commit_lsn && wait_slots_remaining > 0)
+ wait_slots_remaining --;
+ }
+ LWLockRelease(ReplicationSlotControlLock);
+
+ if (wait_slots_remaining == 0)
+ return;
+
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ 1000L, PG_WAIT_EXTENSION);
+
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+
+ CHECK_FOR_INTERRUPTS();
+ }
+}
+
/*
* Helper function for ReorderBufferReplay and ReorderBufferStreamTXN.
*
@@ -2502,6 +2585,9 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
* Call either PREPARE (for two-phase transactions) or COMMIT (for
* regular ones).
*/
+
+ wait_for_standby_confirmation(commit_lsn);
+
if (rbtxn_prepared(txn))
rb->prepare(rb, txn, commit_lsn);
else
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
new file mode 100644
index 0000000000..529ddb21ae
--- /dev/null
+++ b/src/backend/replication/logical/slotsync.c
@@ -0,0 +1,413 @@
+/*-------------------------------------------------------------------------
+ * slotsync.c
+ * PostgreSQL worker for synchronizing slots to a standby from primary
+ *
+ * Copyright (c) 2016-2018, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/replication/logical/slotsync.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "commands/dbcommands.h"
+#include "pgstat.h"
+#include "postmaster/bgworker.h"
+#include "replication/logicallauncher.h"
+#include "replication/logicalworker.h"
+#include "replication/walreceiver.h"
+#include "replication/worker_internal.h"
+#include "storage/ipc.h"
+#include "storage/procarray.h"
+#include "utils/builtins.h"
+#include "utils/pg_lsn.h"
+#include "utils/varlena.h"
+
+char *synchronize_slot_names;
+char *standby_slot_names;
+
+/*
+ * Wait for remote slot to pass localy reserved position.
+ */
+static void
+wait_for_primary_slot_catchup(WalReceiverConn *wrconn, char *slot_name,
+ XLogRecPtr min_lsn)
+{
+ WalRcvExecResult *res;
+ TupleTableSlot *slot;
+ Oid slotRow[1] = {LSNOID};
+ StringInfoData cmd;
+ bool isnull;
+ XLogRecPtr restart_lsn;
+
+ for (;;)
+ {
+ int rc;
+
+ CHECK_FOR_INTERRUPTS();
+
+ initStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT restart_lsn"
+ " FROM pg_catalog.pg_replication_slots"
+ " WHERE slot_name = %s",
+ quote_literal_cstr(slot_name));
+ res = walrcv_exec(wrconn, cmd.data, 1, slotRow);
+
+ if (res->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errmsg("could not fetch slot info for slot \"%s\" from primary: %s",
+ slot_name, res->err)));
+
+ slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+ if (!tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+ ereport(ERROR,
+ (errmsg("slot \"%s\" disapeared from provider",
+ slot_name)));
+
+ restart_lsn = DatumGetLSN(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ ExecClearTuple(slot);
+ walrcv_clear_result(res);
+
+ if (restart_lsn >= min_lsn)
+ break;
+
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ wal_retrieve_retry_interval,
+ WAIT_EVENT_REPL_SLOT_SYNC_MAIN);
+
+ ResetLatch(MyLatch);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+ }
+}
+
+/*
+ * Synchronize single slot to given position.
+ *
+ * This optionally creates new slot if there is no existing one.
+ */
+static void
+synchronize_one_slot(WalReceiverConn *wrconn, char *slot_name, char *database,
+ char *plugin_name, XLogRecPtr target_lsn)
+{
+ bool found = false;
+ XLogRecPtr endlsn;
+
+ /* Search for the named slot and mark it active if we find it. */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (int i = 0; i < max_replication_slots; i++)
+ {
+ ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i];
+
+ if (!s->in_use)
+ continue;
+
+ if (strcmp(NameStr(s->data.name), slot_name) == 0)
+ {
+ found = true;
+ break;
+ }
+ }
+ LWLockRelease(ReplicationSlotControlLock);
+
+ StartTransactionCommand();
+
+ /* Already existing slot, acquire */
+ if (found)
+ {
+ ReplicationSlotAcquire(slot_name, true);
+
+ if (target_lsn < MyReplicationSlot->data.confirmed_flush)
+ {
+ elog(DEBUG1,
+ "not synchronizing slot %s; synchronization would move it backward",
+ slot_name);
+
+ ReplicationSlotRelease();
+ CommitTransactionCommand();
+ return;
+ }
+ }
+ /* Otherwise create the slot first. */
+ else
+ {
+ TransactionId xmin_horizon = InvalidTransactionId;
+ ReplicationSlot *slot;
+
+ ReplicationSlotCreate(slot_name, true, RS_EPHEMERAL, false);
+ slot = MyReplicationSlot;
+
+ SpinLockAcquire(&slot->mutex);
+ slot->data.database = get_database_oid(database, false);
+ namestrcpy(&slot->data.plugin, plugin_name);
+ SpinLockRelease(&slot->mutex);
+
+ ReplicationSlotReserveWal();
+
+ LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
+ xmin_horizon = GetOldestSafeDecodingTransactionId(true);
+ slot->effective_catalog_xmin = xmin_horizon;
+ slot->data.catalog_xmin = xmin_horizon;
+ ReplicationSlotsComputeRequiredXmin(true);
+ LWLockRelease(ProcArrayLock);
+
+ if (target_lsn < MyReplicationSlot->data.restart_lsn)
+ {
+ ereport(LOG,
+ errmsg("waiting for remote slot \"%s\" LSN (%X/%X) to pass local slot LSN (%X/%X)",
+ slot_name,
+ LSN_FORMAT_ARGS(target_lsn), LSN_FORMAT_ARGS(MyReplicationSlot->data.restart_lsn)));
+
+ wait_for_primary_slot_catchup(wrconn, slot_name,
+ MyReplicationSlot->data.restart_lsn);
+ }
+
+ ReplicationSlotPersist();
+ }
+
+ endlsn = pg_logical_replication_slot_advance(target_lsn);
+
+ elog(DEBUG3, "synchronized slot %s to lsn (%X/%X)",
+ slot_name, LSN_FORMAT_ARGS(endlsn));
+
+ ReplicationSlotRelease();
+ CommitTransactionCommand();
+}
+
+static void
+synchronize_slots(void)
+{
+ WalRcvExecResult *res;
+ WalReceiverConn *wrconn = NULL;
+ TupleTableSlot *slot;
+ Oid slotRow[3] = {TEXTOID, TEXTOID, LSNOID};
+ StringInfoData s;
+ char *database;
+ char *err;
+ MemoryContext oldctx = CurrentMemoryContext;
+
+ if (!WalRcv)
+ return;
+
+ /* syscache access needs a transaction env. */
+ StartTransactionCommand();
+ /* make dbname live outside TX context */
+ MemoryContextSwitchTo(oldctx);
+
+ database = get_database_name(MyDatabaseId);
+ initStringInfo(&s);
+ appendStringInfo(&s, "%s dbname=%s", PrimaryConnInfo, database);
+ wrconn = walrcv_connect(s.data, true, false, "slot_sync", &err);
+
+ if (wrconn == NULL)
+ ereport(ERROR,
+ (errmsg("could not connect to the primary server: %s", err)));
+
+ resetStringInfo(&s);
+ appendStringInfo(&s,
+ "SELECT slot_name, plugin, confirmed_flush_lsn"
+ " FROM pg_catalog.pg_replication_slots"
+ " WHERE database = %s",
+ quote_literal_cstr(database));
+ if (strcmp(synchronize_slot_names, "") != 0 && strcmp(synchronize_slot_names, "*") != 0)
+ {
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+
+ rawname = pstrdup(synchronize_slot_names);
+ SplitIdentifierString(rawname, ',', &namelist);
+
+ appendStringInfoString(&s, " AND slot_name IN (");
+ foreach (lc, namelist)
+ {
+ if (lc != list_head(namelist))
+ appendStringInfoChar(&s, ',');
+ appendStringInfo(&s, "%s",
+ quote_literal_cstr(lfirst(lc)));
+ }
+ appendStringInfoChar(&s, ')');
+ }
+
+ res = walrcv_exec(wrconn, s.data, 3, slotRow);
+ pfree(s.data);
+
+ if (res->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errmsg("could not fetch slot info from primary: %s",
+ res->err)));
+
+ CommitTransactionCommand();
+ /* CommitTransactionCommand switches to TopMemoryContext */
+ MemoryContextSwitchTo(oldctx);
+
+ slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+ {
+ char *slot_name;
+ char *plugin_name;
+ XLogRecPtr confirmed_flush_lsn;
+ bool isnull;
+
+ slot_name = TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ plugin_name = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
+ Assert(!isnull);
+
+ confirmed_flush_lsn = DatumGetLSN(slot_getattr(slot, 3, &isnull));
+ Assert(!isnull);
+
+ synchronize_one_slot(wrconn, slot_name, database, plugin_name,
+ confirmed_flush_lsn);
+
+ ExecClearTuple(slot);
+ }
+
+ walrcv_clear_result(res);
+ pfree(database);
+
+ walrcv_disconnect(wrconn);
+}
+
+/*
+ * The main loop of our worker process.
+ */
+void
+ReplSlotSyncMain(Datum main_arg)
+{
+ int worker_slot = DatumGetInt32(main_arg);
+
+ /* Attach to slot */
+ logicalrep_worker_attach(worker_slot);
+
+ /* Establish signal handlers. */
+ BackgroundWorkerUnblockSignals();
+
+ /* Load the libpq-specific functions */
+ load_file("libpqwalreceiver", false);
+
+ /* Connect to our database. */
+ BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->userid,
+ 0);
+
+ StartTransactionCommand();
+ ereport(LOG,
+ (errmsg("replication slot synchronization worker for database \"%s\" has started",
+ get_database_name(MyLogicalRepWorker->dbid))));
+ CommitTransactionCommand();
+
+ /* Main wait loop. */
+ for (;;)
+ {
+ int rc;
+
+ CHECK_FOR_INTERRUPTS();
+
+ if (!RecoveryInProgress())
+ return;
+
+ if (strcmp(synchronize_slot_names, "") == 0)
+ return;
+
+ synchronize_slots();
+
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ wal_retrieve_retry_interval,
+ WAIT_EVENT_REPL_SLOT_SYNC_MAIN);
+
+ ResetLatch(MyLatch);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+ }
+}
+
+/*
+ * Routines for handling the GUC variable(s)
+ */
+
+bool
+check_synchronize_slot_names(char **newval, void **extra, GucSource source)
+{
+ /* Special handling for "*" which means all. */
+ if (strcmp(*newval, "*") == 0)
+ {
+ return true;
+ }
+ else
+ {
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Parse string into list of identifiers */
+ if (!SplitIdentifierString(rawname, ',', &namelist))
+ {
+ /* syntax error in name list */
+ GUC_check_errdetail("List syntax is invalid.");
+ pfree(rawname);
+ list_free(namelist);
+ return false;
+ }
+
+ foreach(lc, namelist)
+ {
+ char *curname = (char *) lfirst(lc);
+
+ ReplicationSlotValidateName(curname, ERROR);
+ }
+
+ pfree(rawname);
+ list_free(namelist);
+ }
+
+ return true;
+}
+
+
+bool
+check_standby_slot_names(char **newval, void **extra, GucSource source)
+{
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Parse string into list of identifiers */
+ if (!SplitIdentifierString(rawname, ',', &namelist))
+ {
+ /* syntax error in name list */
+ GUC_check_errdetail("List syntax is invalid.");
+ pfree(rawname);
+ list_free(namelist);
+ return false;
+ }
+
+ foreach(lc, namelist)
+ {
+ char *curname = (char *) lfirst(lc);
+
+ ReplicationSlotValidateName(curname, ERROR);
+ }
+
+ pfree(rawname);
+ list_free(namelist);
+
+ return true;
+}
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 6dce355633..2307d187e4 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -100,6 +100,7 @@
#include "catalog/pg_subscription_rel.h"
#include "catalog/pg_type.h"
#include "commands/copy.h"
+#include "commands/subscriptioncmds.h"
#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "parser/parse_relation.h"
@@ -155,7 +156,8 @@ finish_sync_worker(void)
CommitTransactionCommand();
/* Find the leader apply worker and signal it. */
- logicalrep_worker_wakeup(MyLogicalRepWorker->subid, InvalidOid);
+ logicalrep_worker_wakeup(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid, InvalidOid);
/* Stop gracefully */
proc_exit(0);
@@ -195,7 +197,8 @@ wait_for_relation_state_change(Oid relid, char expected_state)
/* Check if the sync worker is still running and bail if not. */
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- worker = logicalrep_worker_find(MyLogicalRepWorker->subid, relid,
+ worker = logicalrep_worker_find(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid, relid,
false);
LWLockRelease(LogicalRepWorkerLock);
if (!worker)
@@ -242,7 +245,8 @@ wait_for_worker_state_change(char expected_state)
* waiting.
*/
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- worker = logicalrep_worker_find(MyLogicalRepWorker->subid,
+ worker = logicalrep_worker_find(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid,
InvalidOid, false);
if (worker && worker->proc)
logicalrep_worker_wakeup_ptr(worker);
@@ -508,7 +512,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn)
*/
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- syncworker = logicalrep_worker_find(MyLogicalRepWorker->subid,
+ syncworker = logicalrep_worker_find(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid,
rstate->relid, false);
if (syncworker)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 3d58910c14..b9354bd023 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1600,7 +1600,8 @@ apply_handle_stream_start(StringInfo s)
* Signal the leader apply worker, as it may be waiting for
* us.
*/
- logicalrep_worker_wakeup(MyLogicalRepWorker->subid, InvalidOid);
+ logicalrep_worker_wakeup(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid, InvalidOid);
}
parallel_stream_nchanges = 0;
diff --git a/src/backend/replication/repl_gram.y b/src/backend/replication/repl_gram.y
index 0c874e33cf..12a4b74368 100644
--- a/src/backend/replication/repl_gram.y
+++ b/src/backend/replication/repl_gram.y
@@ -76,11 +76,12 @@ Node *replication_parse_result;
%token K_EXPORT_SNAPSHOT
%token K_NOEXPORT_SNAPSHOT
%token K_USE_SNAPSHOT
+%token K_LIST_SLOTS
%type <node> command
%type <node> base_backup start_replication start_logical_replication
create_replication_slot drop_replication_slot identify_system
- read_replication_slot timeline_history show
+ read_replication_slot timeline_history show list_slots
%type <list> generic_option_list
%type <defelt> generic_option
%type <uintval> opt_timeline
@@ -91,6 +92,7 @@ Node *replication_parse_result;
%type <boolval> opt_temporary
%type <list> create_slot_options create_slot_legacy_opt_list
%type <defelt> create_slot_legacy_opt
+%type <list> slot_name_list slot_name_list_opt
%%
@@ -114,6 +116,7 @@ command:
| read_replication_slot
| timeline_history
| show
+ | list_slots
;
/*
@@ -126,6 +129,33 @@ identify_system:
}
;
+slot_name_list:
+ IDENT
+ {
+ $$ = list_make1($1);
+ }
+ | slot_name_list ',' IDENT
+ {
+ $$ = lappend($1, $3);
+ }
+
+slot_name_list_opt:
+ slot_name_list { $$ = $1; }
+ | /* EMPTY */ { $$ = NIL; }
+ ;
+
+/*
+ * LIST_SLOTS
+ */
+list_slots:
+ K_LIST_SLOTS slot_name_list_opt
+ {
+ ListSlotsCmd *cmd = makeNode(ListSlotsCmd);
+ cmd->slot_names = $2;
+ $$ = (Node *) cmd;
+ }
+ ;
+
/*
* READ_REPLICATION_SLOT %s
*/
diff --git a/src/backend/replication/repl_scanner.l b/src/backend/replication/repl_scanner.l
index cb467ca46f..9501df38eb 100644
--- a/src/backend/replication/repl_scanner.l
+++ b/src/backend/replication/repl_scanner.l
@@ -128,6 +128,7 @@ DROP_REPLICATION_SLOT { return K_DROP_REPLICATION_SLOT; }
TIMELINE_HISTORY { return K_TIMELINE_HISTORY; }
PHYSICAL { return K_PHYSICAL; }
RESERVE_WAL { return K_RESERVE_WAL; }
+LIST_SLOTS { return K_LIST_SLOTS; }
LOGICAL { return K_LOGICAL; }
SLOT { return K_SLOT; }
TEMPORARY { return K_TEMPORARY; }
@@ -304,6 +305,7 @@ replication_scanner_is_replication_command(void)
case K_READ_REPLICATION_SLOT:
case K_TIMELINE_HISTORY:
case K_SHOW:
+ case K_LIST_SLOTS:
/* Yes; push back the first token so we can parse later. */
repl_pushed_back_token = first_token;
return true;
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 6035cf4816..83ada6db6a 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -467,7 +467,7 @@ pg_physical_replication_slot_advance(XLogRecPtr moveto)
* WAL and removal of old catalog tuples. As decoding is done in fast_forward
* mode, no changes are generated anyway.
*/
-static XLogRecPtr
+XLogRecPtr
pg_logical_replication_slot_advance(XLogRecPtr moveto)
{
LogicalDecodingContext *ctx;
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 45b8b3684f..0d01b8967a 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -473,6 +473,194 @@ IdentifySystem(void)
end_tup_output(tstate);
}
+static int
+pg_qsort_namecmp(const void *a, const void *b)
+{
+ return strncmp(NameStr(*(Name) a), NameStr(*(Name) b), NAMEDATALEN);
+}
+
+/*
+ * Handle the LIST_SLOTS command.
+ */
+static void
+ListSlots(ListSlotsCmd *cmd)
+{
+ DestReceiver *dest;
+ TupOutputState *tstate;
+ TupleDesc tupdesc;
+ NameData *slot_names;
+ int numslot_names;
+
+ numslot_names = list_length(cmd->slot_names);
+ if (numslot_names)
+ {
+ ListCell *lc;
+ int i = 0;
+
+ slot_names = palloc(numslot_names * sizeof(NameData));
+ foreach(lc, cmd->slot_names)
+ {
+ char *slot_name = lfirst(lc);
+
+ ReplicationSlotValidateName(slot_name, ERROR);
+ namestrcpy(&slot_names[i++], slot_name);
+ }
+
+ qsort(slot_names, numslot_names, sizeof(NameData), pg_qsort_namecmp);
+ }
+
+ dest = CreateDestReceiver(DestRemoteSimple);
+
+ /* need a tuple descriptor representing four columns */
+ tupdesc = CreateTemplateTupleDesc(10);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 1, "slot_name",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 2, "plugin",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 3, "slot_type",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 4, "datoid",
+ INT8OID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 5, "database",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 6, "temporary",
+ INT4OID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 7, "xmin",
+ INT8OID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 8, "catalog_xmin",
+ INT8OID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 9, "restart_lsn",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 10, "confirmed_flush",
+ TEXTOID, -1, 0);
+
+ /* prepare for projection of tuples */
+ tstate = begin_tup_output_tupdesc(dest, tupdesc, &TTSOpsVirtual);
+
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (int slotno = 0; slotno < max_replication_slots; slotno++)
+ {
+ ReplicationSlot *slot = &ReplicationSlotCtl->replication_slots[slotno];
+ char restart_lsn_str[MAXFNAMELEN];
+ char confirmed_flush_lsn_str[MAXFNAMELEN];
+ Datum values[10];
+ bool nulls[10];
+
+ ReplicationSlotPersistency persistency;
+ TransactionId xmin;
+ TransactionId catalog_xmin;
+ XLogRecPtr restart_lsn;
+ XLogRecPtr confirmed_flush_lsn;
+ Oid datoid;
+ NameData slot_name;
+ NameData plugin;
+ int i;
+ int64 tmpbigint;
+
+ if (!slot->in_use)
+ continue;
+
+ SpinLockAcquire(&slot->mutex);
+
+ xmin = slot->data.xmin;
+ catalog_xmin = slot->data.catalog_xmin;
+ datoid = slot->data.database;
+ restart_lsn = slot->data.restart_lsn;
+ confirmed_flush_lsn = slot->data.confirmed_flush;
+ namestrcpy(&slot_name, NameStr(slot->data.name));
+ namestrcpy(&plugin, NameStr(slot->data.plugin));
+ persistency = slot->data.persistency;
+
+ SpinLockRelease(&slot->mutex);
+
+ if (numslot_names &&
+ !bsearch((void *) &slot_name, (void *) slot_names,
+ numslot_names, sizeof(NameData), pg_qsort_namecmp))
+ continue;
+
+ memset(nulls, 0, sizeof(nulls));
+
+ i = 0;
+ values[i++] = CStringGetTextDatum(NameStr(slot_name));
+
+ if (datoid == InvalidOid)
+ nulls[i++] = true;
+ else
+ values[i++] = CStringGetTextDatum(NameStr(plugin));
+
+ if (datoid == InvalidOid)
+ values[i++] = CStringGetTextDatum("physical");
+ else
+ values[i++] = CStringGetTextDatum("logical");
+
+ if (datoid == InvalidOid)
+ nulls[i++] = true;
+ else
+ {
+ tmpbigint = datoid;
+ values[i++] = Int64GetDatum(tmpbigint);
+ }
+
+ if (datoid == InvalidOid)
+ nulls[i++] = true;
+ else
+ {
+ MemoryContext cur = CurrentMemoryContext;
+
+ /* syscache access needs a transaction env. */
+ StartTransactionCommand();
+ /* make dbname live outside TX context */
+ MemoryContextSwitchTo(cur);
+ values[i++] = CStringGetTextDatum(get_database_name(datoid));
+ CommitTransactionCommand();
+ /* CommitTransactionCommand switches to TopMemoryContext */
+ MemoryContextSwitchTo(cur);
+ }
+
+ values[i++] = Int32GetDatum(persistency == RS_TEMPORARY ? 1 : 0);
+
+ if (xmin != InvalidTransactionId)
+ {
+ tmpbigint = xmin;
+ values[i++] = Int64GetDatum(tmpbigint);
+ }
+ else
+ nulls[i++] = true;
+
+ if (catalog_xmin != InvalidTransactionId)
+ {
+ tmpbigint = catalog_xmin;
+ values[i++] = Int64GetDatum(tmpbigint);
+ }
+ else
+ nulls[i++] = true;
+
+ if (restart_lsn != InvalidXLogRecPtr)
+ {
+ snprintf(restart_lsn_str, sizeof(restart_lsn_str), "%X/%X",
+ LSN_FORMAT_ARGS(restart_lsn));
+ values[i++] = CStringGetTextDatum(restart_lsn_str);
+ }
+ else
+ nulls[i++] = true;
+
+ if (confirmed_flush_lsn != InvalidXLogRecPtr)
+ {
+ snprintf(confirmed_flush_lsn_str, sizeof(confirmed_flush_lsn_str),
+ "%X/%X", LSN_FORMAT_ARGS(confirmed_flush_lsn));
+ values[i++] = CStringGetTextDatum(confirmed_flush_lsn_str);
+ }
+ else
+ nulls[i++] = true;
+
+ /* send it to dest */
+ do_tup_output(tstate, values, nulls);
+ }
+ LWLockRelease(ReplicationSlotControlLock);
+
+ end_tup_output(tstate);
+}
+
/* Handle READ_REPLICATION_SLOT command */
static void
ReadReplicationSlot(ReadReplicationSlotCmd *cmd)
@@ -1820,6 +2008,13 @@ exec_replication_command(const char *cmd_string)
EndReplicationCommand(cmdtag);
break;
+ case T_ListSlotsCmd:
+ cmdtag = "LIST_SLOTS";
+ set_ps_display(cmdtag);
+ ListSlots((ListSlotsCmd *) cmd_node);
+ EndReplicationCommand(cmdtag);
+ break;
+
case T_StartReplicationCmd:
{
StartReplicationCmd *cmd = (StartReplicationCmd *) cmd_node;
diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c
index 7940d64639..f2a9517091 100644
--- a/src/backend/utils/activity/wait_event.c
+++ b/src/backend/utils/activity/wait_event.c
@@ -230,6 +230,9 @@ pgstat_get_wait_activity(WaitEventActivity w)
case WAIT_EVENT_LOGICAL_LAUNCHER_MAIN:
event_name = "LogicalLauncherMain";
break;
+ case WAIT_EVENT_REPL_SLOT_SYNC_MAIN:
+ event_name = "ReplSlotSyncMain";
+ break;
case WAIT_EVENT_LOGICAL_PARALLEL_APPLY_MAIN:
event_name = "LogicalParallelApplyMain";
break;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index cab3ddbe11..0ee7ad1348 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -63,8 +63,12 @@
#include "postmaster/syslogger.h"
#include "postmaster/walwriter.h"
#include "replication/logicallauncher.h"
+#include "replication/logicalworker.h"
+#include "replication/reorderbuffer.h"
#include "replication/slot.h"
#include "replication/syncrep.h"
+#include "replication/walreceiver.h"
+#include "replication/walsender.h"
#include "storage/bufmgr.h"
#include "storage/large_object.h"
#include "storage/pg_shmem.h"
@@ -4587,6 +4591,28 @@ struct config_string ConfigureNamesString[] =
check_io_direct, assign_io_direct, NULL
},
+ {
+ {"synchronize_slot_names", PGC_SIGHUP, REPLICATION_STANDBY,
+ gettext_noop("Sets the names of replication slots which to synchronize from primary to standby."),
+ gettext_noop("Value of \"*\" means all."),
+ GUC_LIST_INPUT | GUC_LIST_QUOTE
+ },
+ &synchronize_slot_names,
+ "",
+ check_synchronize_slot_names, NULL, NULL
+ },
+
+ {
+ {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY,
+ gettext_noop("List of physical slots that must confirm changes before changes are sent to logical replication consumers."),
+ NULL,
+ GUC_LIST_INPUT | GUC_LIST_QUOTE
+ },
+ &standby_slot_names,
+ "",
+ check_standby_slot_names, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index dce5049bc2..2ff2188c02 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -330,6 +330,7 @@
# and comma-separated list of application_name
# from standby(s); '*' = all
#vacuum_defer_cleanup_age = 0 # number of xacts by which cleanup is delayed
+#standby_slot_names = '' # physical standby slot names that logical replication waits for
# - Standby Servers -
@@ -357,6 +358,7 @@
#wal_retrieve_retry_interval = 5s # time to wait before retrying to
# retrieve WAL after a failed attempt
#recovery_min_apply_delay = 0 # minimum delay for applying changes during recovery
+#synchronize_slot_names = '' # logical replication slots to sync to standby
# - Subscribers -
diff --git a/src/include/commands/subscriptioncmds.h b/src/include/commands/subscriptioncmds.h
index 214dc6c29e..0e77f9ee5c 100644
--- a/src/include/commands/subscriptioncmds.h
+++ b/src/include/commands/subscriptioncmds.h
@@ -17,6 +17,7 @@
#include "catalog/objectaddress.h"
#include "parser/parse_node.h"
+#include "replication/walreceiver.h"
extern ObjectAddress CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
bool isTopLevel);
@@ -28,4 +29,6 @@ extern void AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId);
extern char defGetStreamingMode(DefElem *def);
+extern void ReplicationSlotDropAtPubNode(WalReceiverConn *wrconn, char *slotname, bool missing_ok);
+
#endif /* SUBSCRIPTIONCMDS_H */
diff --git a/src/include/nodes/replnodes.h b/src/include/nodes/replnodes.h
index 4321ba8f86..980e0b2ee2 100644
--- a/src/include/nodes/replnodes.h
+++ b/src/include/nodes/replnodes.h
@@ -33,6 +33,15 @@ typedef struct IdentifySystemCmd
NodeTag type;
} IdentifySystemCmd;
+/* ----------------------
+ * LIST_SLOTS command
+ * ----------------------
+ */
+typedef struct ListSlotsCmd
+{
+ NodeTag type;
+ List *slot_names;
+} ListSlotsCmd;
/* ----------------------
* BASE_BACKUP command
diff --git a/src/include/replication/logicallauncher.h b/src/include/replication/logicallauncher.h
index a07c9cb311..80fdbf9657 100644
--- a/src/include/replication/logicallauncher.h
+++ b/src/include/replication/logicallauncher.h
@@ -31,4 +31,6 @@ extern bool IsLogicalLauncher(void);
extern pid_t GetLeaderApplyWorkerPid(pid_t pid);
+extern PGDLLIMPORT char *PrimaryConnInfo;
+
#endif /* LOGICALLAUNCHER_H */
diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h
index 39588da79f..6408753557 100644
--- a/src/include/replication/logicalworker.h
+++ b/src/include/replication/logicalworker.h
@@ -14,10 +14,16 @@
#include <signal.h>
+#include "utils/guc.h"
+
+extern char *synchronize_slot_names;
+extern char *standby_slot_names;
+
extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending;
extern void ApplyWorkerMain(Datum main_arg);
extern void ParallelApplyWorkerMain(Datum main_arg);
+extern void ReplSlotSyncMain(Datum main_arg);
extern bool IsLogicalWorker(void);
extern bool IsLogicalParallelApplyWorker(void);
@@ -29,4 +35,7 @@ extern void LogicalRepWorkersWakeupAtCommit(Oid subid);
extern void AtEOXact_LogicalRepWorkers(bool isCommit);
+extern bool check_synchronize_slot_names(char **newval, void **extra, GucSource source);
+extern bool check_standby_slot_names(char **newval, void **extra, GucSource source);
+
#endif /* LOGICALWORKER_H */
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index a8a89dc784..5dc2e0d30d 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -15,7 +15,6 @@
#include "storage/lwlock.h"
#include "storage/shmem.h"
#include "storage/spin.h"
-#include "replication/walreceiver.h"
/*
* Behaviour of replication slots, upon release or crash.
@@ -238,7 +237,6 @@ extern ReplicationSlot *SearchNamedReplicationSlot(const char *name, bool need_l
extern int ReplicationSlotIndex(ReplicationSlot *slot);
extern bool ReplicationSlotName(int index, Name name);
extern void ReplicationSlotNameForTablesync(Oid suboid, Oid relid, char *syncslotname, Size szslot);
-extern void ReplicationSlotDropAtPubNode(WalReceiverConn *wrconn, char *slotname, bool missing_ok);
extern void StartupReplicationSlots(void);
extern void CheckPointReplicationSlots(void);
@@ -246,4 +244,7 @@ extern void CheckPointReplicationSlots(void);
extern void CheckSlotRequirements(void);
extern void CheckSlotPermissions(void);
+extern XLogRecPtr pg_logical_replication_slot_advance(XLogRecPtr moveto);
+
+
#endif /* SLOT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 281626fa6f..9e9d64faf2 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -20,6 +20,7 @@
#include "pgtime.h"
#include "port/atomics.h"
#include "replication/logicalproto.h"
+#include "replication/slot.h"
#include "replication/walsender.h"
#include "storage/condition_variable.h"
#include "storage/latch.h"
@@ -191,6 +192,17 @@ typedef struct
} proto;
} WalRcvStreamOptions;
+/*
+ * Slot information receiver from remote.
+ *
+ * Currently same as ReplicationSlotPersistentData except last_sync_time
+ */
+typedef struct WalRecvReplicationSlotData
+{
+ ReplicationSlotPersistentData persistent_data;
+ TimestampTz last_sync_time;
+} WalRecvReplicationSlotData;
+
struct WalReceiverConn;
typedef struct WalReceiverConn WalReceiverConn;
@@ -280,6 +292,11 @@ typedef void (*walrcv_get_senderinfo_fn) (WalReceiverConn *conn,
typedef char *(*walrcv_identify_system_fn) (WalReceiverConn *conn,
TimeLineID *primary_tli);
+/*
+ * TODO
+ */
+typedef List *(*walrcv_list_slots_fn) (WalReceiverConn *conn, const char *slots);
+
/*
* walrcv_server_version_fn
*
@@ -393,6 +410,7 @@ typedef struct WalReceiverFunctionsType
walrcv_get_conninfo_fn walrcv_get_conninfo;
walrcv_get_senderinfo_fn walrcv_get_senderinfo;
walrcv_identify_system_fn walrcv_identify_system;
+ walrcv_list_slots_fn walrcv_list_slots;
walrcv_server_version_fn walrcv_server_version;
walrcv_readtimelinehistoryfile_fn walrcv_readtimelinehistoryfile;
walrcv_startstreaming_fn walrcv_startstreaming;
@@ -417,6 +435,8 @@ extern PGDLLIMPORT WalReceiverFunctionsType *WalReceiverFunctions;
WalReceiverFunctions->walrcv_get_senderinfo(conn, sender_host, sender_port)
#define walrcv_identify_system(conn, primary_tli) \
WalReceiverFunctions->walrcv_identify_system(conn, primary_tli)
+#define walrcv_list_slots(conn, slots) \
+ WalReceiverFunctions->walrcv_list_slots(conn, slots)
#define walrcv_server_version(conn) \
WalReceiverFunctions->walrcv_server_version(conn)
#define walrcv_readtimelinehistoryfile(conn, tli, filename, content, size) \
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index dce71d2c50..5b4fda2fd9 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -65,7 +65,7 @@ typedef struct LogicalRepWorker
* would be created for each transaction which will be deleted after the
* transaction is finished.
*/
- FileSet *stream_fileset;
+ struct FileSet *stream_fileset;
/*
* PID of leader apply worker if this slot is used for a parallel apply
@@ -226,15 +226,15 @@ extern PGDLLIMPORT LogicalRepWorker *MyLogicalRepWorker;
extern PGDLLIMPORT bool in_remote_transaction;
extern void logicalrep_worker_attach(int slot);
-extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid,
+extern LogicalRepWorker *logicalrep_worker_find(Oid dbid, Oid subid, Oid relid,
bool only_running);
extern List *logicalrep_workers_find(Oid subid, bool only_running);
extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname,
Oid userid, Oid relid,
dsm_handle subworker_dsm);
-extern void logicalrep_worker_stop(Oid subid, Oid relid);
+extern void logicalrep_worker_stop(Oid dbid, Oid subid, Oid relid);
extern void logicalrep_pa_worker_stop(int slot_no, uint16 generation);
-extern void logicalrep_worker_wakeup(Oid subid, Oid relid);
+extern void logicalrep_worker_wakeup(Oid dbid, Oid subid, Oid relid);
extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker);
extern int logicalrep_sync_worker_count(Oid subid);
diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h
index 518d3b0a1f..cccd4d9d32 100644
--- a/src/include/utils/wait_event.h
+++ b/src/include/utils/wait_event.h
@@ -43,6 +43,7 @@ typedef enum
WAIT_EVENT_LOGICAL_APPLY_MAIN,
WAIT_EVENT_LOGICAL_LAUNCHER_MAIN,
WAIT_EVENT_LOGICAL_PARALLEL_APPLY_MAIN,
+ WAIT_EVENT_REPL_SLOT_SYNC_MAIN,
WAIT_EVENT_RECOVERY_WAL_STREAM,
WAIT_EVENT_SYSLOGGER_MAIN,
WAIT_EVENT_WAL_RECEIVER_MAIN,
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index 2008958010..b6fcc8704e 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -42,6 +42,7 @@ tests += {
't/034_create_database.pl',
't/035_standby_logical_decoding.pl',
't/036_truncated_dropped.pl',
+ 't/037_slot_sync.pl',
],
},
}
diff --git a/src/test/recovery/t/037_slot_sync.pl b/src/test/recovery/t/037_slot_sync.pl
new file mode 100644
index 0000000000..0520042d96
--- /dev/null
+++ b/src/test/recovery/t/037_slot_sync.pl
@@ -0,0 +1,130 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+my $node_phys_standby = PostgreSQL::Test::Cluster->new('phys_standby');
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+
+# find $pat in logfile of $node after $off-th byte
+sub find_in_log
+{
+ my ($node, $pat, $off) = @_;
+
+ $off = 0 unless defined $off;
+ my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile);
+ return 0 if (length($log) <= $off);
+
+ $log = substr($log, $off);
+
+ return $log =~ m/$pat/;
+}
+
+# Check invalidation in the logfile
+sub check_for_invalidation
+{
+ my ($log_start, $test_name) = @_;
+
+ # message should be issued
+ ok( find_in_log(
+ $node_phys_standby,
+ "invalidating obsolete replication slot \"sub1\"", $log_start),
+ "sub1 slot invalidation is logged $test_name");
+}
+
+# Check conflicting status in pg_replication_slots.
+sub check_slots_conflicting_status
+{
+ my $res = $node_phys_standby->safe_psql(
+ 'postgres', qq(
+ select bool_and(conflicting) from pg_replication_slots;));
+
+ is($res, 't',
+ "Logical slot is reported as conflicting");
+}
+
+$node_primary->init(allows_streaming => 'logical');
+$node_primary->append_conf('postgresql.conf', "standby_slot_names = 'pslot1'");
+$node_primary->start;
+$node_primary->psql('postgres', q{SELECT pg_create_physical_replication_slot('pslot1');});
+
+$node_primary->backup('backup');
+
+$node_phys_standby->init_from_backup($node_primary, 'backup', has_streaming => 1);
+$node_phys_standby->append_conf('postgresql.conf', q{
+synchronize_slot_names = '*'
+primary_slot_name = 'pslot1'
+hot_standby_feedback = off
+});
+
+$node_phys_standby->start;
+
+$node_primary->safe_psql('postgres', "CREATE TABLE t1 (a int PRIMARY KEY)");
+$node_primary->safe_psql('postgres', "INSERT INTO t1 VALUES (1), (2), (3)");
+
+# Some tests need to wait for VACUUM to be replayed. But vacuum does not flush
+# WAL. An insert into flush_wal outside transaction does guarantee a flush.
+$node_primary->psql('postgres', q[CREATE TABLE flush_wal();]);
+
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+$node_subscriber->safe_psql('postgres', "CREATE TABLE t1 (a int PRIMARY KEY)");
+
+$node_primary->safe_psql('postgres', "CREATE PUBLICATION pub1 FOR TABLE t1");
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1 CONNECTION '" . ($node_primary->connstr . ' dbname=postgres') . "' PUBLICATION pub1");
+
+# Wait for initial sync of all subscriptions
+my $synced_query =
+ "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result = $node_primary->safe_psql('postgres',
+ "SELECT slot_name, plugin, database FROM pg_replication_slots WHERE slot_type = 'logical'");
+
+is($result, qq(sub1|pgoutput|postgres), 'logical slot on primary');
+
+# FIXME: standby needs restart to pick up new slots
+$node_phys_standby->restart;
+sleep 3;
+
+$result = $node_phys_standby->safe_psql('postgres',
+ "SELECT slot_name, plugin, database FROM pg_replication_slots");
+
+is($result, qq(sub1|pgoutput|postgres), 'logical slot on standby');
+
+$node_primary->safe_psql('postgres', "INSERT INTO t1 VALUES (4), (5), (6)");
+$node_primary->wait_for_catchup('sub1');
+
+$node_primary->wait_for_catchup($node_phys_standby->name);
+
+# Logical subscriber and physical replica are caught up at this point.
+
+# Drop the subscription so that catalog_xmin is unknown on the primary
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub1");
+
+# This should trigger a conflict as hot_standby_feedback is off on the standby
+$node_primary->safe_psql('postgres', qq[
+ CREATE TABLE conflict_test(x integer, y text);
+ DROP TABLE conflict_test;
+ VACUUM full pg_class;
+ INSERT INTO flush_wal DEFAULT VALUES; -- see create table flush_wal
+]);
+
+# Ensure physical replay catches up
+$node_primary->wait_for_catchup($node_phys_standby);
+
+# Check invalidation in the logfile
+check_for_invalidation(1, 'with vacuum FULL on pg_class');
+
+# Check conflicting status in pg_replication_slots.
+check_slots_conflicting_status();
+
+done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
@ 2023-06-16 09:56 ` Amit Kapila <[email protected]>
2023-06-19 06:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-07-09 07:30 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-20 11:34 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
1 sibling, 3 replies; 59+ messages in thread
From: Amit Kapila @ 2023-06-16 09:56 UTC (permalink / raw)
To: Drouvot, Bertrand <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
On Mon, Apr 17, 2023 at 7:37 PM Drouvot, Bertrand
<[email protected]> wrote:
>
> Please find attached V5 (a rebase of V4 posted up-thread).
>
> In addition to the "rebasing" work, the TAP test adds a test about conflict handling (logical slot invalidation)
> relying on the work done in the "Minimal logical decoding on standby" patch series.
>
> I did not look more at the patch (than what's was needed for the rebase) but plan to do so.
>
Are you still planning to continue working on this? Some miscellaneous
comments while going through this patch are as follows?
1. Can you please try to explain the functionality of the overall
patch somewhere in the form of comments and or commit message?
2. It seems that the initially synchronized list of slots is only used
to launch a per-database worker to synchronize all the slots
corresponding to that database. If so, then why do we need to fetch
all the slot-related information via LIST_SLOTS command?
3. As mentioned in the initial email, I think it would be better to
replace LIST_SLOTS command with a SELECT query.
4. How the limit of sync_slot workers is decided? Can we document such
a piece of information? Do we need a new GUC to decide the number of
workers? Ideally, it would be better to avoid GUC, can we use any
existing logical replication workers related GUC?
5. Can we separate out the functionality related to standby_slot_names
in a separate patch, probably the first one? I think that will patch
easier to review.
6. In libpqrcv_list_slots(), two-phase related slot information is not
retrieved. Is there a reason for the same?
7.
+static void
+wait_for_standby_confirmation(XLogRecPtr commit_lsn)
Some comments atop this function would make it easier to review.
8.
+/*-------------------------------------------------------------------------
+ * slotsync.c
+ * PostgreSQL worker for synchronizing slots to a standby from primary
+ *
+ * Copyright (c) 2016-2018, PostgreSQL Global Development Group
+ *
The copyright notice is out-of-date.
9. Why synchronize_one_slot() compares
MyReplicationSlot->data.restart_lsn with the value of
confirmed_flush_lsn passed to it? Also, why it does only for new slots
but not existing slots?
10. Can we somehow test if the restart_lsn is advanced properly after
sync? I think it is important to ensure that because otherwise after
standby's promotion, the subscriber can start syncing from the wrong
position.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
@ 2023-06-19 06:02 ` Drouvot, Bertrand <[email protected]>
2023-06-19 10:03 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2 siblings, 1 reply; 59+ messages in thread
From: Drouvot, Bertrand @ 2023-06-19 06:02 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
Hi,
On 6/16/23 11:56 AM, Amit Kapila wrote:
> On Mon, Apr 17, 2023 at 7:37 PM Drouvot, Bertrand
> <[email protected]> wrote:
>>
>> Please find attached V5 (a rebase of V4 posted up-thread).
>>
>> In addition to the "rebasing" work, the TAP test adds a test about conflict handling (logical slot invalidation)
>> relying on the work done in the "Minimal logical decoding on standby" patch series.
>>
>> I did not look more at the patch (than what's was needed for the rebase) but plan to do so.
>>
>
> Are you still planning to continue working on this?
Yes, I think it would be great to have such a feature in core.
> Some miscellaneous
> comments while going through this patch are as follows?
Thanks! I'll look at them and will try to come back to you by
mid of next week.
Also I think we need to handle the case of invalidated replication slot(s): should
we drop/recreate it/them? (as the main goal is to have sync slot(s) on the standby).
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-06-19 06:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
@ 2023-06-19 10:03 ` Amit Kapila <[email protected]>
2023-06-19 16:26 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Amit Kapila @ 2023-06-19 10:03 UTC (permalink / raw)
To: Drouvot, Bertrand <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
On Mon, Jun 19, 2023 at 11:34 AM Drouvot, Bertrand
<[email protected]> wrote:
>
> Also I think we need to handle the case of invalidated replication slot(s): should
> we drop/recreate it/them? (as the main goal is to have sync slot(s) on the standby).
>
Do you intend to ask what happens to logical slots invalidated (due to
say max_slot_wal_keep_size) on publisher? I think those should be
invalidated on standby too. Another thought whether there is chance
that the slot on standby gets invalidated due to conflict (say
required rows removed on primary)? I think in such cases the slot on
primary/publisher should have been dropped/invalidated by that time.
BTW, does the patch handles drop of logical slots on standby when the
same slot is dropped on publisher/primary?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-06-19 06:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-19 10:03 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
@ 2023-06-19 16:26 ` Drouvot, Bertrand <[email protected]>
2023-06-20 10:22 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Drouvot, Bertrand @ 2023-06-19 16:26 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
Hi,
On 6/19/23 12:03 PM, Amit Kapila wrote:
> On Mon, Jun 19, 2023 at 11:34 AM Drouvot, Bertrand
> <[email protected]> wrote:
>>
>> Also I think we need to handle the case of invalidated replication slot(s): should
>> we drop/recreate it/them? (as the main goal is to have sync slot(s) on the standby).
>>
>
> Do you intend to ask what happens to logical slots invalidated (due to
> say max_slot_wal_keep_size) on publisher? I think those should be
> invalidated on standby too.
Agree that it should behave that way.
> Another thought whether there is chance
> that the slot on standby gets invalidated due to conflict (say
> required rows removed on primary)?
That's the scenario I had in mind when asking the question above.
> I think in such cases the slot on
> primary/publisher should have been dropped/invalidated by that time.
I don't think so.
For example, such a scenario could occur:
- there is no physical slot between the standby and the primary
- the standby is shut down
- logical decoding on the primary is moving forward and now there is vacuum
operations that will conflict on the standby
- the standby starts and reports the logical slot being invalidated (while it is
not on the primary)
In such a case (slot valid on the primary but invalidated on the standby) then I think we
could drop and recreate the invalidated slot on the standby.
> BTW, does the patch handles drop of logical slots on standby when the
> same slot is dropped on publisher/primary?
>
from what I've seen, yes it looks like it behaves that way (will look closer).
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-06-19 06:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-19 10:03 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-06-19 16:26 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
@ 2023-06-20 10:22 ` Amit Kapila <[email protected]>
2023-06-26 05:45 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Amit Kapila @ 2023-06-20 10:22 UTC (permalink / raw)
To: Drouvot, Bertrand <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
On Mon, Jun 19, 2023 at 9:56 PM Drouvot, Bertrand
<[email protected]> wrote:
>
> On 6/19/23 12:03 PM, Amit Kapila wrote:
> > On Mon, Jun 19, 2023 at 11:34 AM Drouvot, Bertrand
> > <[email protected]> wrote:
> >>
> >> Also I think we need to handle the case of invalidated replication slot(s): should
> >> we drop/recreate it/them? (as the main goal is to have sync slot(s) on the standby).
> >>
> >
> > Do you intend to ask what happens to logical slots invalidated (due to
> > say max_slot_wal_keep_size) on publisher? I think those should be
> > invalidated on standby too.
>
> Agree that it should behave that way.
>
> > Another thought whether there is chance
> > that the slot on standby gets invalidated due to conflict (say
> > required rows removed on primary)?
>
> That's the scenario I had in mind when asking the question above.
>
> > I think in such cases the slot on
> > primary/publisher should have been dropped/invalidated by that time.
>
> I don't think so.
>
> For example, such a scenario could occur:
>
> - there is no physical slot between the standby and the primary
> - the standby is shut down
> - logical decoding on the primary is moving forward and now there is vacuum
> operations that will conflict on the standby
> - the standby starts and reports the logical slot being invalidated (while it is
> not on the primary)
>
> In such a case (slot valid on the primary but invalidated on the standby) then I think we
> could drop and recreate the invalidated slot on the standby.
>
Will it be safe? Because after recreating the slot, it will reserve
the new WAL location and build the snapshot based on that which might
miss some important information in the snapshot. For example, to
update the slot's position with new information from the primary, the
patch uses pg_logical_replication_slot_advance() which means it will
process all records and update the snapshot via
DecodeCommit->SnapBuildCommitTxn().
The other related thing is that do we somehow need to ensure that WAL
is replayed on standby before moving the slot's position to the target
location received from the primary?
> > BTW, does the patch handles drop of logical slots on standby when the
> > same slot is dropped on publisher/primary?
> >
>
> from what I've seen, yes it looks like it behaves that way (will look closer).
>
Okay, I have asked because I don't see a call to ReplicationSlotDrop()
in the patch.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-06-19 06:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-19 10:03 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-06-19 16:26 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-20 10:22 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
@ 2023-06-26 05:45 ` Drouvot, Bertrand <[email protected]>
2023-06-26 10:34 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Drouvot, Bertrand @ 2023-06-26 05:45 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
Hi,
On 6/20/23 12:22 PM, Amit Kapila wrote:
> On Mon, Jun 19, 2023 at 9:56 PM Drouvot, Bertrand
> <[email protected]> wrote:
>> In such a case (slot valid on the primary but invalidated on the standby) then I think we
>> could drop and recreate the invalidated slot on the standby.
>>
>
> Will it be safe? Because after recreating the slot, it will reserve
> the new WAL location and build the snapshot based on that which might
> miss some important information in the snapshot. For example, to
> update the slot's position with new information from the primary, the
> patch uses pg_logical_replication_slot_advance() which means it will
> process all records and update the snapshot via
> DecodeCommit->SnapBuildCommitTxn().
Your concern is that the slot could have been consumed on the standby?
I mean, if we suppose the "synchronized" slot can't be consumed on the standby then
drop/recreate such an invalidated slot would be ok?
Asking, because I'm not sure we should allow consumption of a "synchronized" slot
until the standby gets promoted.
When the patch has been initially proposed, logical decoding from a standby
was not implemented yet.
> The other related thing is that do we somehow need to ensure that WAL
> is replayed on standby before moving the slot's position to the target
> location received from the primary?
Yeah, will check if this is currently done that way in the patch proposal.
>>> BTW, does the patch handles drop of logical slots on standby when the
>>> same slot is dropped on publisher/primary?
>>>
>>
>> from what I've seen, yes it looks like it behaves that way (will look closer).
>>
>
> Okay, I have asked because I don't see a call to ReplicationSlotDrop()
> in the patch.
>
Right. I'd need to look closer to understand how it works (for the moment
the "only" thing I've done was the re-base shared up-thread).
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-06-19 06:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-19 10:03 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-06-19 16:26 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-20 10:22 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-06-26 05:45 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
@ 2023-06-26 10:34 ` Amit Kapila <[email protected]>
2023-06-28 06:49 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Amit Kapila @ 2023-06-26 10:34 UTC (permalink / raw)
To: Drouvot, Bertrand <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
On Mon, Jun 26, 2023 at 11:15 AM Drouvot, Bertrand
<[email protected]> wrote:
>
> On 6/20/23 12:22 PM, Amit Kapila wrote:
> > On Mon, Jun 19, 2023 at 9:56 PM Drouvot, Bertrand
> > <[email protected]> wrote:
>
> >> In such a case (slot valid on the primary but invalidated on the standby) then I think we
> >> could drop and recreate the invalidated slot on the standby.
> >>
> >
> > Will it be safe? Because after recreating the slot, it will reserve
> > the new WAL location and build the snapshot based on that which might
> > miss some important information in the snapshot. For example, to
> > update the slot's position with new information from the primary, the
> > patch uses pg_logical_replication_slot_advance() which means it will
> > process all records and update the snapshot via
> > DecodeCommit->SnapBuildCommitTxn().
>
> Your concern is that the slot could have been consumed on the standby?
>
> I mean, if we suppose the "synchronized" slot can't be consumed on the standby then
> drop/recreate such an invalidated slot would be ok?
>
That also may not be sufficient because as soon as the slot is
invalidated/dropped, the required WAL could be removed on standby.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-06-19 06:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-19 10:03 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-06-19 16:26 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-20 10:22 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-06-26 05:45 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-26 10:34 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
@ 2023-06-28 06:49 ` Drouvot, Bertrand <[email protected]>
2023-06-29 10:36 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Drouvot, Bertrand @ 2023-06-28 06:49 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
Hi,
On 6/26/23 12:34 PM, Amit Kapila wrote:
> On Mon, Jun 26, 2023 at 11:15 AM Drouvot, Bertrand
> <[email protected]> wrote:
>>
>> On 6/20/23 12:22 PM, Amit Kapila wrote:
>>> On Mon, Jun 19, 2023 at 9:56 PM Drouvot, Bertrand
>>> <[email protected]> wrote:
>>
>>>> In such a case (slot valid on the primary but invalidated on the standby) then I think we
>>>> could drop and recreate the invalidated slot on the standby.
>>>>
>>>
>>> Will it be safe? Because after recreating the slot, it will reserve
>>> the new WAL location and build the snapshot based on that which might
>>> miss some important information in the snapshot. For example, to
>>> update the slot's position with new information from the primary, the
>>> patch uses pg_logical_replication_slot_advance() which means it will
>>> process all records and update the snapshot via
>>> DecodeCommit->SnapBuildCommitTxn().
>>
>> Your concern is that the slot could have been consumed on the standby?
>>
>> I mean, if we suppose the "synchronized" slot can't be consumed on the standby then
>> drop/recreate such an invalidated slot would be ok?
>>
>
> That also may not be sufficient because as soon as the slot is
> invalidated/dropped, the required WAL could be removed on standby.
>
Yeah, I think once the slot is dropped we just have to wait for the slot to
be re-created on the standby according to the new synchronize_slot_names GUC.
Assuming the initial slot "creation" on the standby (coming from the synchronize_slot_names usage)
is working "correctly" then it should also work "correctly" once the slot is dropped.
If we agree that a synchronized slot can not/should not be consumed (will implement this behavior) then
I think the proposed scenario above should make sense, do you agree?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-06-19 06:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-19 10:03 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-06-19 16:26 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-20 10:22 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-06-26 05:45 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-26 10:34 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-06-28 06:49 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
@ 2023-06-29 10:36 ` Amit Kapila <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Amit Kapila @ 2023-06-29 10:36 UTC (permalink / raw)
To: Drouvot, Bertrand <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
On Wed, Jun 28, 2023 at 12:19 PM Drouvot, Bertrand
<[email protected]> wrote:
>
> On 6/26/23 12:34 PM, Amit Kapila wrote:
> > On Mon, Jun 26, 2023 at 11:15 AM Drouvot, Bertrand
> > <[email protected]> wrote:
> >>
> >> On 6/20/23 12:22 PM, Amit Kapila wrote:
> >>> On Mon, Jun 19, 2023 at 9:56 PM Drouvot, Bertrand
> >>> <[email protected]> wrote:
> >>
> >>>> In such a case (slot valid on the primary but invalidated on the standby) then I think we
> >>>> could drop and recreate the invalidated slot on the standby.
> >>>>
> >>>
> >>> Will it be safe? Because after recreating the slot, it will reserve
> >>> the new WAL location and build the snapshot based on that which might
> >>> miss some important information in the snapshot. For example, to
> >>> update the slot's position with new information from the primary, the
> >>> patch uses pg_logical_replication_slot_advance() which means it will
> >>> process all records and update the snapshot via
> >>> DecodeCommit->SnapBuildCommitTxn().
> >>
> >> Your concern is that the slot could have been consumed on the standby?
> >>
> >> I mean, if we suppose the "synchronized" slot can't be consumed on the standby then
> >> drop/recreate such an invalidated slot would be ok?
> >>
> >
> > That also may not be sufficient because as soon as the slot is
> > invalidated/dropped, the required WAL could be removed on standby.
> >
>
> Yeah, I think once the slot is dropped we just have to wait for the slot to
> be re-created on the standby according to the new synchronize_slot_names GUC.
>
> Assuming the initial slot "creation" on the standby (coming from the synchronize_slot_names usage)
> is working "correctly" then it should also work "correctly" once the slot is dropped.
>
I also think so.
> If we agree that a synchronized slot can not/should not be consumed (will implement this behavior) then
> I think the proposed scenario above should make sense, do you agree?
>
Yeah, I also can't think of a use case for this. So, we can probably
disallow it and document the same. I guess if we came across a use
case for this, we can rethink allowing to consume the changes from
synchronized slots.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
@ 2023-07-09 07:30 ` Bharath Rupireddy <[email protected]>
2023-07-10 03:36 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2 siblings, 1 reply; 59+ messages in thread
From: Bharath Rupireddy @ 2023-07-09 07:30 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Drouvot, Bertrand <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
On Fri, Jun 16, 2023 at 3:26 PM Amit Kapila <[email protected]> wrote:
>
> On Mon, Apr 17, 2023 at 7:37 PM Drouvot, Bertrand
> <[email protected]> wrote:
> >
> > Please find attached V5 (a rebase of V4 posted up-thread).
> >
> > In addition to the "rebasing" work, the TAP test adds a test about conflict handling (logical slot invalidation)
> > relying on the work done in the "Minimal logical decoding on standby" patch series.
> >
> > I did not look more at the patch (than what's was needed for the rebase) but plan to do so.
> >
>
> Are you still planning to continue working on this? Some miscellaneous
> comments while going through this patch are as follows?
>
> 1. Can you please try to explain the functionality of the overall
> patch somewhere in the form of comments and or commit message?
IIUC, there are 2 core ideas of the feature:
1) It will never let the logical replication subscribers go ahead of
physical replication standbys specified in standby_slot_names. It
implements this by delaying decoding of commit records on the
walsenders corresponding to logical replication subscribers on the
primary until all the specified standbys confirm receiving the commit
LSN.
2) The physical replication standbys will synchronize data of the
specified logical replication slots (in synchronize_slot_names) from
the primary, creating the logical replication slots if necessary.
Since the logical replication subscribers will never go out of
physical replication standbys, the standbys can safely synchronize the
slots and keep the data necessary for subscribers to connect to it and
work seamlessly even after a failover.
If my understanding is right, I have few thoughts here:
1. All the logical walsenders are delayed on the primary - per
wait_for_standby_confirmation() despite the user being interested in
only a few of them via synchronize_slot_names. Shouldn't the delay be
for just the slots specified in synchronize_slot_names?
2. I think we can split the patch like this - 0001 can be the logical
walsenders delaying decoding on the primary unless standbys confirm,
0002 standby synchronizing the logical slots.
3. I think we need to change the GUC standby_slot_names to better
reflect what it is used for - wait_for_replication_slot_names or
wait_for_
4. It allows specifying logical slots in standby_slot_names, meaning,
it can disallow logical slots getting ahead of other logical slots
specified in standby_slot_names. Should we allow this case with the
thinking that if there's anyone using logical replication for failover
(well, will anybody do that in production?).
5. Similar to above, it allows specifying physical slots in
synchronize_slot_names. Should we disallow?
I'm attaching the v6 patch, a rebased version of v5.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] v6-0001-Synchronize-logical-replication-slots-from-primar.patch (62.6K, ../../CALj2ACW5z=QYNabSEW-iqJ0oZUyTNztPBXzCoAiDtWFWYMcuag@mail.gmail.com/2-v6-0001-Synchronize-logical-replication-slots-from-primar.patch)
download | inline diff:
From b7a9ab4ed1e6da6cebf9d8fa48f834369b583f14 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Sun, 9 Jul 2023 03:56:16 +0000
Subject: [PATCH v6] Synchronize logical replication slots from primary to
standby
---
doc/src/sgml/config.sgml | 34 ++
src/backend/commands/subscriptioncmds.c | 4 +-
src/backend/postmaster/bgworker.c | 3 +
.../libpqwalreceiver/libpqwalreceiver.c | 95 ++++
src/backend/replication/logical/Makefile | 1 +
src/backend/replication/logical/launcher.c | 263 +++++++----
src/backend/replication/logical/meson.build | 1 +
.../replication/logical/reorderbuffer.c | 86 ++++
src/backend/replication/logical/slotsync.c | 413 ++++++++++++++++++
src/backend/replication/logical/tablesync.c | 13 +-
src/backend/replication/logical/worker.c | 3 +-
src/backend/replication/repl_gram.y | 32 +-
src/backend/replication/repl_scanner.l | 2 +
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 195 +++++++++
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 26 ++
src/backend/utils/misc/postgresql.conf.sample | 3 +
src/include/commands/subscriptioncmds.h | 3 +
src/include/nodes/replnodes.h | 9 +
src/include/replication/logicallauncher.h | 2 +
src/include/replication/logicalworker.h | 9 +
src/include/replication/slot.h | 5 +-
src/include/replication/walreceiver.h | 20 +
src/include/replication/worker_internal.h | 8 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/037_slot_sync.pl | 130 ++++++
27 files changed, 1270 insertions(+), 94 deletions(-)
create mode 100644 src/backend/replication/logical/slotsync.c
create mode 100644 src/test/recovery/t/037_slot_sync.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index c9fa6cd9c7..756db04886 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4450,6 +4450,23 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-standby-slot-names" xreflabel="standby_slot_names">
+ <term><varname>standby_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>standby_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ List of physical replication slots that logical replication waits for.
+ If a logical replication connection is meant to switch to a physical
+ standby after the standby is promoted, the physical replication slot
+ for the standby should be listed here. This ensures that logical
+ replication is not ahead of the physical standby.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
@@ -4598,6 +4615,23 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
</listitem>
</varlistentry>
+ <varlistentry id="guc-synchronize_slot_names" xreflabel="synchronize_slot_names">
+ <term><varname>synchronize_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>synchronize_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies a list of logical replication slots that a physical standby
+ should synchronize from the primary server. This is necessary to be
+ able to retarget those logical replication connections to this standby
+ if it gets promoted. Specify <literal>*</literal> to synchronize all
+ logical replication slots. The default is empty.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index d4e798baeb..42e9b1056c 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -993,7 +993,7 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data,
RemoveSubscriptionRel(sub->oid, relid);
- logicalrep_worker_stop(sub->oid, relid);
+ logicalrep_worker_stop(MyDatabaseId, sub->oid, relid);
/*
* For READY state, we would have already dropped the
@@ -1591,7 +1591,7 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel)
{
LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc);
- logicalrep_worker_stop(w->subid, w->relid);
+ logicalrep_worker_stop(w->dbid, w->subid, w->relid);
}
list_free(subworkers);
diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c
index 5b4bd71694..f2f4475c3b 100644
--- a/src/backend/postmaster/bgworker.c
+++ b/src/backend/postmaster/bgworker.c
@@ -129,6 +129,9 @@ static const struct
{
"ApplyWorkerMain", ApplyWorkerMain
},
+ {
+ "ReplSlotSyncMain", ReplSlotSyncMain
+ },
{
"ParallelApplyWorkerMain", ParallelApplyWorkerMain
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index dc9c5c82d9..e7ecfa947b 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -34,6 +34,7 @@
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
#include "utils/tuplestore.h"
+#include "utils/varlena.h"
PG_MODULE_MAGIC;
@@ -58,6 +59,7 @@ static void libpqrcv_get_senderinfo(WalReceiverConn *conn,
char **sender_host, int *sender_port);
static char *libpqrcv_identify_system(WalReceiverConn *conn,
TimeLineID *primary_tli);
+static List *libpqrcv_list_slots(WalReceiverConn *conn, const char *slot_names);
static int libpqrcv_server_version(WalReceiverConn *conn);
static void libpqrcv_readtimelinehistoryfile(WalReceiverConn *conn,
TimeLineID tli, char **filename,
@@ -96,6 +98,7 @@ static WalReceiverFunctionsType PQWalReceiverFunctions = {
.walrcv_receive = libpqrcv_receive,
.walrcv_send = libpqrcv_send,
.walrcv_create_slot = libpqrcv_create_slot,
+ .walrcv_list_slots = libpqrcv_list_slots,
.walrcv_get_backend_pid = libpqrcv_get_backend_pid,
.walrcv_exec = libpqrcv_exec,
.walrcv_disconnect = libpqrcv_disconnect
@@ -409,6 +412,98 @@ libpqrcv_server_version(WalReceiverConn *conn)
return PQserverVersion(conn->streamConn);
}
+/*
+ * Get list of slots from primary.
+ */
+static List *
+libpqrcv_list_slots(WalReceiverConn *conn, const char *slot_names)
+{
+ PGresult *res;
+ List *slotlist = NIL;
+ int ntuples;
+ StringInfoData s;
+ WalRecvReplicationSlotData *slot_data;
+
+ initStringInfo(&s);
+ appendStringInfoString(&s, "LIST_SLOTS");
+
+ if (strcmp(slot_names, "") != 0 && strcmp(slot_names, "*") != 0)
+ {
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+
+ appendStringInfoChar(&s, ' ');
+ rawname = pstrdup(slot_names);
+ SplitIdentifierString(rawname, ',', &namelist);
+ foreach (lc, namelist)
+ {
+ if (lc != list_head(namelist))
+ appendStringInfoChar(&s, ',');
+ appendStringInfo(&s, "%s",
+ quote_identifier(lfirst(lc)));
+ }
+ }
+
+ res = libpqrcv_PQexec(conn->streamConn, s.data);
+ pfree(s.data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ PQclear(res);
+ ereport(ERROR,
+ (errmsg("could not receive list of slots the primary server: %s",
+ pchomp(PQerrorMessage(conn->streamConn)))));
+ }
+ if (PQnfields(res) < 10)
+ {
+ int nfields = PQnfields(res);
+
+ PQclear(res);
+ ereport(ERROR,
+ (errmsg("invalid response from primary server"),
+ errdetail("Could not get list of slots: got %d fields, expected %d or more fields.",
+ nfields, 10)));
+ }
+
+ ntuples = PQntuples(res);
+ for (int i = 0; i < ntuples; i++)
+ {
+ char *slot_type;
+
+ slot_data = palloc0(sizeof(WalRecvReplicationSlotData));
+ namestrcpy(&slot_data->persistent_data.name, PQgetvalue(res, i, 0));
+ if (!PQgetisnull(res, i, 1))
+ namestrcpy(&slot_data->persistent_data.plugin, PQgetvalue(res, i, 1));
+ slot_type = PQgetvalue(res, i, 2);
+ if (!PQgetisnull(res, i, 3))
+ slot_data->persistent_data.database = atooid(PQgetvalue(res, i, 3));
+ if (strcmp(slot_type, "physical") == 0)
+ {
+ if (OidIsValid(slot_data->persistent_data.database))
+ elog(ERROR, "unexpected physical replication slot with database set");
+ }
+ if (pg_strtoint32(PQgetvalue(res, i, 5)) == 1)
+ slot_data->persistent_data.persistency = RS_TEMPORARY;
+ else
+ slot_data->persistent_data.persistency = RS_PERSISTENT;
+ if (!PQgetisnull(res, i, 6))
+ slot_data->persistent_data.xmin = atooid(PQgetvalue(res, i, 6));
+ if (!PQgetisnull(res, i, 7))
+ slot_data->persistent_data.catalog_xmin = atooid(PQgetvalue(res, i, 7));
+ if (!PQgetisnull(res, i, 8))
+ slot_data->persistent_data.restart_lsn = strtou64(PQgetvalue(res, i, 8), NULL, 10);
+ if (!PQgetisnull(res, i, 9))
+ slot_data->persistent_data.confirmed_flush = strtou64(PQgetvalue(res, i, 9), NULL, 10);
+
+ slot_data->last_sync_time = 0;
+ slotlist = lappend(slotlist, slot_data);
+ }
+
+ PQclear(res);
+
+ return slotlist;
+}
+
/*
* Start streaming WAL data from given streaming options.
*
diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile
index 2dc25e37bb..ba03eeff1c 100644
--- a/src/backend/replication/logical/Makefile
+++ b/src/backend/replication/logical/Makefile
@@ -25,6 +25,7 @@ OBJS = \
proto.o \
relation.o \
reorderbuffer.o \
+ slotsync.o \
snapbuild.o \
tablesync.o \
worker.o
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 542af7d863..640f7647cc 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -22,6 +22,7 @@
#include "access/htup_details.h"
#include "access/tableam.h"
#include "access/xact.h"
+#include "catalog/pg_authid.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
#include "funcapi.h"
@@ -246,7 +247,7 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker,
* We are only interested in the leader apply worker or table sync worker.
*/
LogicalRepWorker *
-logicalrep_worker_find(Oid subid, Oid relid, bool only_running)
+logicalrep_worker_find(Oid dbid, Oid subid, Oid relid, bool only_running)
{
int i;
LogicalRepWorker *res = NULL;
@@ -262,8 +263,8 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running)
if (isParallelApplyWorker(w))
continue;
- if (w->in_use && w->subid == subid && w->relid == relid &&
- (!only_running || w->proc))
+ if (w->in_use && w->dbid == dbid && w->subid == subid &&
+ w->relid == relid && (!only_running || w->proc))
{
res = w;
break;
@@ -320,9 +321,13 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid,
/* Sanity check - tablesync worker cannot be a subworker */
Assert(!(is_parallel_apply_worker && OidIsValid(relid)));
- ereport(DEBUG1,
- (errmsg_internal("starting logical replication worker for subscription \"%s\"",
- subname)));
+ if (OidIsValid(subid))
+ ereport(DEBUG1,
+ (errmsg_internal("starting logical replication worker for subscription \"%s\"",
+ subname)));
+ else
+ ereport(DEBUG1,
+ (errmsg_internal("starting replication slot synchronization worker")));
/* Report this after the initial starting message for consistency. */
if (max_replication_slots == 0)
@@ -359,7 +364,9 @@ retry:
* reason we do this is because if some worker failed to start up and its
* parent has crashed while waiting, the in_use state was never cleared.
*/
- if (worker == NULL || nsyncworkers >= max_sync_workers_per_subscription)
+ if (worker == NULL ||
+ (OidIsValid(relid) &&
+ nsyncworkers >= max_sync_workers_per_subscription))
{
bool did_cleanup = false;
@@ -455,15 +462,20 @@ retry:
memset(&bgw, 0, sizeof(bgw));
bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
BGWORKER_BACKEND_DATABASE_CONNECTION;
- bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ bgw.bgw_start_time = BgWorkerStart_ConsistentState;
snprintf(bgw.bgw_library_name, MAXPGPATH, "postgres");
- if (is_parallel_apply_worker)
+ if (!OidIsValid(subid))
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ReplSlotSyncMain");
+ else if (is_parallel_apply_worker)
snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ParallelApplyWorkerMain");
else
snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain");
- if (OidIsValid(relid))
+ if (!OidIsValid(subid))
+ snprintf(bgw.bgw_name, BGW_MAXLEN,
+ "replication slot synchronization worker");
+ else if (OidIsValid(relid))
snprintf(bgw.bgw_name, BGW_MAXLEN,
"logical replication worker for subscription %u sync %u", subid, relid);
else if (is_parallel_apply_worker)
@@ -591,13 +603,13 @@ logicalrep_worker_stop_internal(LogicalRepWorker *worker, int signo)
* Stop the logical replication worker for subid/relid, if any.
*/
void
-logicalrep_worker_stop(Oid subid, Oid relid)
+logicalrep_worker_stop(Oid dbid, Oid subid, Oid relid)
{
LogicalRepWorker *worker;
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- worker = logicalrep_worker_find(subid, relid, false);
+ worker = logicalrep_worker_find(dbid, subid, relid, false);
if (worker)
{
@@ -658,13 +670,13 @@ logicalrep_pa_worker_stop(ParallelApplyWorkerInfo *winfo)
* Wake up (using latch) any logical replication worker for specified sub/rel.
*/
void
-logicalrep_worker_wakeup(Oid subid, Oid relid)
+logicalrep_worker_wakeup(Oid dbid, Oid subid, Oid relid)
{
LogicalRepWorker *worker;
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- worker = logicalrep_worker_find(subid, relid, true);
+ worker = logicalrep_worker_find(dbid, subid, relid, true);
if (worker)
logicalrep_worker_wakeup_ptr(worker);
@@ -909,7 +921,7 @@ ApplyLauncherRegister(void)
memset(&bgw, 0, sizeof(bgw));
bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
BGWORKER_BACKEND_DATABASE_CONNECTION;
- bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ bgw.bgw_start_time = BgWorkerStart_ConsistentState;
snprintf(bgw.bgw_library_name, MAXPGPATH, "postgres");
snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyLauncherMain");
snprintf(bgw.bgw_name, BGW_MAXLEN,
@@ -1092,6 +1104,157 @@ ApplyLauncherWakeup(void)
kill(LogicalRepCtx->launcher_pid, SIGUSR1);
}
+static void
+ApplyLauncherStartSlotSync(long *wait_time)
+{
+ WalReceiverConn *wrconn;
+ char *err;
+ List *slots;
+ ListCell *lc;
+ MemoryContext tmpctx;
+ MemoryContext oldctx;
+
+ if (strcmp(synchronize_slot_names, "") == 0)
+ return;
+
+ wrconn = walrcv_connect(PrimaryConnInfo, false, false,
+ "Logical Replication Launcher", &err);
+ if (!wrconn)
+ ereport(ERROR,
+ (errmsg("could not connect to the primary server: %s", err)));
+
+ /* Use temporary context for the slot list and worker info. */
+ tmpctx = AllocSetContextCreate(TopMemoryContext,
+ "Logical Replication Launcher slot sync ctx",
+ ALLOCSET_DEFAULT_SIZES);
+ oldctx = MemoryContextSwitchTo(tmpctx);
+
+ slots = walrcv_list_slots(wrconn, synchronize_slot_names);
+
+ foreach(lc, slots)
+ {
+ WalRecvReplicationSlotData *slot_data = lfirst(lc);
+ LogicalRepWorker *w;
+ TimestampTz last_sync;
+ TimestampTz now;
+ long elapsed;
+
+ if (!OidIsValid(slot_data->persistent_data.database))
+ continue;
+
+ LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
+ w = logicalrep_worker_find(slot_data->persistent_data.database, InvalidOid,
+ InvalidOid, false);
+ LWLockRelease(LogicalRepWorkerLock);
+
+ if (w != NULL)
+ continue; /* worker is running already */
+
+ /*
+ * If the worker is eligible to start now, launch it. Otherwise,
+ * adjust wait_time so that we'll wake up as soon as it can be
+ * started.
+ *
+ * Each apply worker can only be restarted once per
+ * wal_retrieve_retry_interval, so that errors do not cause us to
+ * repeatedly restart the worker as fast as possible.
+ */
+ last_sync = slot_data->last_sync_time;
+ now = GetCurrentTimestamp();
+ if (last_sync == 0 ||
+ (elapsed = TimestampDifferenceMilliseconds(last_sync, now)) >= wal_retrieve_retry_interval)
+ {
+ slot_data->last_sync_time = now;
+ logicalrep_worker_launch(slot_data->persistent_data.database,
+ InvalidOid, NULL,
+ BOOTSTRAP_SUPERUSERID, InvalidOid,
+ DSM_HANDLE_INVALID);
+ }
+ else
+ {
+ *wait_time = Min(*wait_time,
+ wal_retrieve_retry_interval - elapsed);
+ }
+ }
+
+ /* Switch back to original memory context. */
+ MemoryContextSwitchTo(oldctx);
+ /* Clean the temporary memory. */
+ MemoryContextDelete(tmpctx);
+
+ walrcv_disconnect(wrconn);
+}
+
+static void
+ApplyLauncherStartSubs(long *wait_time)
+{
+ List *sublist;
+ ListCell *lc;
+ MemoryContext subctx;
+ MemoryContext oldctx;
+
+ /* Use temporary context to avoid leaking memory across cycles. */
+ subctx = AllocSetContextCreate(TopMemoryContext,
+ "Logical Replication Launcher sublist",
+ ALLOCSET_DEFAULT_SIZES);
+ oldctx = MemoryContextSwitchTo(subctx);
+
+ /* Start the missing workers for enabled subscriptions. */
+ sublist = get_subscription_list();
+ foreach(lc, sublist)
+ {
+ Subscription *sub = (Subscription *) lfirst(lc);
+ LogicalRepWorker *w;
+ TimestampTz last_start;
+ TimestampTz now;
+ long elapsed;
+
+ if (!sub->enabled)
+ continue;
+
+ LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
+ w = logicalrep_worker_find(sub->dbid, sub->oid, InvalidOid, false);
+ LWLockRelease(LogicalRepWorkerLock);
+
+ if (w != NULL)
+ continue; /* worker is running already */
+
+ /*
+ * If the worker is eligible to start now, launch it. Otherwise,
+ * adjust wait_time so that we'll wake up as soon as it can be
+ * started.
+ *
+ * Each subscription's apply worker can only be restarted once per
+ * wal_retrieve_retry_interval, so that errors do not cause us to
+ * repeatedly restart the worker as fast as possible. In cases
+ * where a restart is expected (e.g., subscription parameter
+ * changes), another process should remove the last-start entry
+ * for the subscription so that the worker can be restarted
+ * without waiting for wal_retrieve_retry_interval to elapse.
+ */
+ last_start = ApplyLauncherGetWorkerStartTime(sub->oid);
+ now = GetCurrentTimestamp();
+ if (last_start == 0 ||
+ (elapsed = TimestampDifferenceMilliseconds(last_start, now)) >= wal_retrieve_retry_interval)
+ {
+ ApplyLauncherSetWorkerStartTime(sub->oid, now);
+ logicalrep_worker_launch(sub->dbid, sub->oid, sub->name,
+ sub->owner, InvalidOid,
+ DSM_HANDLE_INVALID);
+ }
+ else
+ {
+ *wait_time = Min(*wait_time,
+ wal_retrieve_retry_interval - elapsed);
+ }
+ }
+
+ /* Switch back to original memory context. */
+ MemoryContextSwitchTo(oldctx);
+ /* Clean the temporary memory. */
+ MemoryContextDelete(subctx);
+}
+
/*
* Main loop for the apply launcher process.
*/
@@ -1117,78 +1280,20 @@ ApplyLauncherMain(Datum main_arg)
*/
BackgroundWorkerInitializeConnection(NULL, NULL, 0);
+ load_file("libpqwalreceiver", false);
+
/* Enter main loop */
for (;;)
{
int rc;
- List *sublist;
- ListCell *lc;
- MemoryContext subctx;
- MemoryContext oldctx;
long wait_time = DEFAULT_NAPTIME_PER_CYCLE;
CHECK_FOR_INTERRUPTS();
- /* Use temporary context to avoid leaking memory across cycles. */
- subctx = AllocSetContextCreate(TopMemoryContext,
- "Logical Replication Launcher sublist",
- ALLOCSET_DEFAULT_SIZES);
- oldctx = MemoryContextSwitchTo(subctx);
-
- /* Start any missing workers for enabled subscriptions. */
- sublist = get_subscription_list();
- foreach(lc, sublist)
- {
- Subscription *sub = (Subscription *) lfirst(lc);
- LogicalRepWorker *w;
- TimestampTz last_start;
- TimestampTz now;
- long elapsed;
-
- if (!sub->enabled)
- continue;
-
- LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- w = logicalrep_worker_find(sub->oid, InvalidOid, false);
- LWLockRelease(LogicalRepWorkerLock);
-
- if (w != NULL)
- continue; /* worker is running already */
-
- /*
- * If the worker is eligible to start now, launch it. Otherwise,
- * adjust wait_time so that we'll wake up as soon as it can be
- * started.
- *
- * Each subscription's apply worker can only be restarted once per
- * wal_retrieve_retry_interval, so that errors do not cause us to
- * repeatedly restart the worker as fast as possible. In cases
- * where a restart is expected (e.g., subscription parameter
- * changes), another process should remove the last-start entry
- * for the subscription so that the worker can be restarted
- * without waiting for wal_retrieve_retry_interval to elapse.
- */
- last_start = ApplyLauncherGetWorkerStartTime(sub->oid);
- now = GetCurrentTimestamp();
- if (last_start == 0 ||
- (elapsed = TimestampDifferenceMilliseconds(last_start, now)) >= wal_retrieve_retry_interval)
- {
- ApplyLauncherSetWorkerStartTime(sub->oid, now);
- logicalrep_worker_launch(sub->dbid, sub->oid, sub->name,
- sub->owner, InvalidOid,
- DSM_HANDLE_INVALID);
- }
- else
- {
- wait_time = Min(wait_time,
- wal_retrieve_retry_interval - elapsed);
- }
- }
-
- /* Switch back to original memory context. */
- MemoryContextSwitchTo(oldctx);
- /* Clean the temporary memory. */
- MemoryContextDelete(subctx);
+ if (!RecoveryInProgress())
+ ApplyLauncherStartSubs(&wait_time);
+ else
+ ApplyLauncherStartSlotSync(&wait_time);
/* Wait for more work. */
rc = WaitLatch(MyLatch,
diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build
index d48cd4c590..9e52ec421f 100644
--- a/src/backend/replication/logical/meson.build
+++ b/src/backend/replication/logical/meson.build
@@ -11,6 +11,7 @@ backend_sources += files(
'proto.c',
'relation.c',
'reorderbuffer.c',
+ 'slotsync.c',
'snapbuild.c',
'tablesync.c',
'worker.c',
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 26d252bd87..98c93712ef 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -95,11 +95,14 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "replication/logical.h"
+#include "replication/logicalworker.h"
#include "replication/reorderbuffer.h"
#include "replication/slot.h"
#include "replication/snapbuild.h" /* just for SnapBuildSnapDecRefcount */
#include "storage/bufmgr.h"
#include "storage/fd.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
@@ -107,6 +110,7 @@
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
+#include "utils/varlena.h"
/* entry for a hash table we use to map from xid to our transaction state */
@@ -2053,6 +2057,85 @@ ReorderBufferResetTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
}
+static void
+wait_for_standby_confirmation(XLogRecPtr commit_lsn)
+{
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+ XLogRecPtr flush_pos = InvalidXLogRecPtr;
+
+ if (strcmp(standby_slot_names, "") == 0)
+ return;
+
+ rawname = pstrdup(standby_slot_names);
+ SplitIdentifierString(rawname, ',', &namelist);
+
+ while (true)
+ {
+ int wait_slots_remaining;
+ XLogRecPtr oldest_flush_pos = InvalidXLogRecPtr;
+ int rc;
+
+ wait_slots_remaining = list_length(namelist);
+
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (int i = 0; i < max_replication_slots; i++)
+ {
+ ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i];
+ bool inlist;
+
+ if (!s->in_use)
+ continue;
+
+ inlist = false;
+ foreach (lc, namelist)
+ {
+ char *name = lfirst(lc);
+ if (strcmp(name, NameStr(s->data.name)) == 0)
+ {
+ inlist = true;
+ break;
+ }
+ }
+ if (!inlist)
+ continue;
+
+ SpinLockAcquire(&s->mutex);
+
+ if (s->data.database == InvalidOid)
+ /* Physical slots advance restart_lsn on flush and ignore confirmed_flush_lsn */
+ flush_pos = s->data.restart_lsn;
+ else
+ /* For logical slots we must wait for commit and flush */
+ flush_pos = s->data.confirmed_flush;
+
+ SpinLockRelease(&s->mutex);
+
+ /* We want to find out the min(flush pos) over all named slots */
+ if (oldest_flush_pos == InvalidXLogRecPtr
+ || oldest_flush_pos > flush_pos)
+ oldest_flush_pos = flush_pos;
+
+ if (flush_pos >= commit_lsn && wait_slots_remaining > 0)
+ wait_slots_remaining --;
+ }
+ LWLockRelease(ReplicationSlotControlLock);
+
+ if (wait_slots_remaining == 0)
+ return;
+
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ 1000L, PG_WAIT_EXTENSION);
+
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+
+ CHECK_FOR_INTERRUPTS();
+ }
+}
+
/*
* Helper function for ReorderBufferReplay and ReorderBufferStreamTXN.
*
@@ -2502,6 +2585,9 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
* Call either PREPARE (for two-phase transactions) or COMMIT (for
* regular ones).
*/
+
+ wait_for_standby_confirmation(commit_lsn);
+
if (rbtxn_prepared(txn))
rb->prepare(rb, txn, commit_lsn);
else
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
new file mode 100644
index 0000000000..529ddb21ae
--- /dev/null
+++ b/src/backend/replication/logical/slotsync.c
@@ -0,0 +1,413 @@
+/*-------------------------------------------------------------------------
+ * slotsync.c
+ * PostgreSQL worker for synchronizing slots to a standby from primary
+ *
+ * Copyright (c) 2016-2018, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/replication/logical/slotsync.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "commands/dbcommands.h"
+#include "pgstat.h"
+#include "postmaster/bgworker.h"
+#include "replication/logicallauncher.h"
+#include "replication/logicalworker.h"
+#include "replication/walreceiver.h"
+#include "replication/worker_internal.h"
+#include "storage/ipc.h"
+#include "storage/procarray.h"
+#include "utils/builtins.h"
+#include "utils/pg_lsn.h"
+#include "utils/varlena.h"
+
+char *synchronize_slot_names;
+char *standby_slot_names;
+
+/*
+ * Wait for remote slot to pass localy reserved position.
+ */
+static void
+wait_for_primary_slot_catchup(WalReceiverConn *wrconn, char *slot_name,
+ XLogRecPtr min_lsn)
+{
+ WalRcvExecResult *res;
+ TupleTableSlot *slot;
+ Oid slotRow[1] = {LSNOID};
+ StringInfoData cmd;
+ bool isnull;
+ XLogRecPtr restart_lsn;
+
+ for (;;)
+ {
+ int rc;
+
+ CHECK_FOR_INTERRUPTS();
+
+ initStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT restart_lsn"
+ " FROM pg_catalog.pg_replication_slots"
+ " WHERE slot_name = %s",
+ quote_literal_cstr(slot_name));
+ res = walrcv_exec(wrconn, cmd.data, 1, slotRow);
+
+ if (res->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errmsg("could not fetch slot info for slot \"%s\" from primary: %s",
+ slot_name, res->err)));
+
+ slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+ if (!tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+ ereport(ERROR,
+ (errmsg("slot \"%s\" disapeared from provider",
+ slot_name)));
+
+ restart_lsn = DatumGetLSN(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ ExecClearTuple(slot);
+ walrcv_clear_result(res);
+
+ if (restart_lsn >= min_lsn)
+ break;
+
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ wal_retrieve_retry_interval,
+ WAIT_EVENT_REPL_SLOT_SYNC_MAIN);
+
+ ResetLatch(MyLatch);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+ }
+}
+
+/*
+ * Synchronize single slot to given position.
+ *
+ * This optionally creates new slot if there is no existing one.
+ */
+static void
+synchronize_one_slot(WalReceiverConn *wrconn, char *slot_name, char *database,
+ char *plugin_name, XLogRecPtr target_lsn)
+{
+ bool found = false;
+ XLogRecPtr endlsn;
+
+ /* Search for the named slot and mark it active if we find it. */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (int i = 0; i < max_replication_slots; i++)
+ {
+ ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i];
+
+ if (!s->in_use)
+ continue;
+
+ if (strcmp(NameStr(s->data.name), slot_name) == 0)
+ {
+ found = true;
+ break;
+ }
+ }
+ LWLockRelease(ReplicationSlotControlLock);
+
+ StartTransactionCommand();
+
+ /* Already existing slot, acquire */
+ if (found)
+ {
+ ReplicationSlotAcquire(slot_name, true);
+
+ if (target_lsn < MyReplicationSlot->data.confirmed_flush)
+ {
+ elog(DEBUG1,
+ "not synchronizing slot %s; synchronization would move it backward",
+ slot_name);
+
+ ReplicationSlotRelease();
+ CommitTransactionCommand();
+ return;
+ }
+ }
+ /* Otherwise create the slot first. */
+ else
+ {
+ TransactionId xmin_horizon = InvalidTransactionId;
+ ReplicationSlot *slot;
+
+ ReplicationSlotCreate(slot_name, true, RS_EPHEMERAL, false);
+ slot = MyReplicationSlot;
+
+ SpinLockAcquire(&slot->mutex);
+ slot->data.database = get_database_oid(database, false);
+ namestrcpy(&slot->data.plugin, plugin_name);
+ SpinLockRelease(&slot->mutex);
+
+ ReplicationSlotReserveWal();
+
+ LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
+ xmin_horizon = GetOldestSafeDecodingTransactionId(true);
+ slot->effective_catalog_xmin = xmin_horizon;
+ slot->data.catalog_xmin = xmin_horizon;
+ ReplicationSlotsComputeRequiredXmin(true);
+ LWLockRelease(ProcArrayLock);
+
+ if (target_lsn < MyReplicationSlot->data.restart_lsn)
+ {
+ ereport(LOG,
+ errmsg("waiting for remote slot \"%s\" LSN (%X/%X) to pass local slot LSN (%X/%X)",
+ slot_name,
+ LSN_FORMAT_ARGS(target_lsn), LSN_FORMAT_ARGS(MyReplicationSlot->data.restart_lsn)));
+
+ wait_for_primary_slot_catchup(wrconn, slot_name,
+ MyReplicationSlot->data.restart_lsn);
+ }
+
+ ReplicationSlotPersist();
+ }
+
+ endlsn = pg_logical_replication_slot_advance(target_lsn);
+
+ elog(DEBUG3, "synchronized slot %s to lsn (%X/%X)",
+ slot_name, LSN_FORMAT_ARGS(endlsn));
+
+ ReplicationSlotRelease();
+ CommitTransactionCommand();
+}
+
+static void
+synchronize_slots(void)
+{
+ WalRcvExecResult *res;
+ WalReceiverConn *wrconn = NULL;
+ TupleTableSlot *slot;
+ Oid slotRow[3] = {TEXTOID, TEXTOID, LSNOID};
+ StringInfoData s;
+ char *database;
+ char *err;
+ MemoryContext oldctx = CurrentMemoryContext;
+
+ if (!WalRcv)
+ return;
+
+ /* syscache access needs a transaction env. */
+ StartTransactionCommand();
+ /* make dbname live outside TX context */
+ MemoryContextSwitchTo(oldctx);
+
+ database = get_database_name(MyDatabaseId);
+ initStringInfo(&s);
+ appendStringInfo(&s, "%s dbname=%s", PrimaryConnInfo, database);
+ wrconn = walrcv_connect(s.data, true, false, "slot_sync", &err);
+
+ if (wrconn == NULL)
+ ereport(ERROR,
+ (errmsg("could not connect to the primary server: %s", err)));
+
+ resetStringInfo(&s);
+ appendStringInfo(&s,
+ "SELECT slot_name, plugin, confirmed_flush_lsn"
+ " FROM pg_catalog.pg_replication_slots"
+ " WHERE database = %s",
+ quote_literal_cstr(database));
+ if (strcmp(synchronize_slot_names, "") != 0 && strcmp(synchronize_slot_names, "*") != 0)
+ {
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+
+ rawname = pstrdup(synchronize_slot_names);
+ SplitIdentifierString(rawname, ',', &namelist);
+
+ appendStringInfoString(&s, " AND slot_name IN (");
+ foreach (lc, namelist)
+ {
+ if (lc != list_head(namelist))
+ appendStringInfoChar(&s, ',');
+ appendStringInfo(&s, "%s",
+ quote_literal_cstr(lfirst(lc)));
+ }
+ appendStringInfoChar(&s, ')');
+ }
+
+ res = walrcv_exec(wrconn, s.data, 3, slotRow);
+ pfree(s.data);
+
+ if (res->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errmsg("could not fetch slot info from primary: %s",
+ res->err)));
+
+ CommitTransactionCommand();
+ /* CommitTransactionCommand switches to TopMemoryContext */
+ MemoryContextSwitchTo(oldctx);
+
+ slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+ {
+ char *slot_name;
+ char *plugin_name;
+ XLogRecPtr confirmed_flush_lsn;
+ bool isnull;
+
+ slot_name = TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ plugin_name = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
+ Assert(!isnull);
+
+ confirmed_flush_lsn = DatumGetLSN(slot_getattr(slot, 3, &isnull));
+ Assert(!isnull);
+
+ synchronize_one_slot(wrconn, slot_name, database, plugin_name,
+ confirmed_flush_lsn);
+
+ ExecClearTuple(slot);
+ }
+
+ walrcv_clear_result(res);
+ pfree(database);
+
+ walrcv_disconnect(wrconn);
+}
+
+/*
+ * The main loop of our worker process.
+ */
+void
+ReplSlotSyncMain(Datum main_arg)
+{
+ int worker_slot = DatumGetInt32(main_arg);
+
+ /* Attach to slot */
+ logicalrep_worker_attach(worker_slot);
+
+ /* Establish signal handlers. */
+ BackgroundWorkerUnblockSignals();
+
+ /* Load the libpq-specific functions */
+ load_file("libpqwalreceiver", false);
+
+ /* Connect to our database. */
+ BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->userid,
+ 0);
+
+ StartTransactionCommand();
+ ereport(LOG,
+ (errmsg("replication slot synchronization worker for database \"%s\" has started",
+ get_database_name(MyLogicalRepWorker->dbid))));
+ CommitTransactionCommand();
+
+ /* Main wait loop. */
+ for (;;)
+ {
+ int rc;
+
+ CHECK_FOR_INTERRUPTS();
+
+ if (!RecoveryInProgress())
+ return;
+
+ if (strcmp(synchronize_slot_names, "") == 0)
+ return;
+
+ synchronize_slots();
+
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ wal_retrieve_retry_interval,
+ WAIT_EVENT_REPL_SLOT_SYNC_MAIN);
+
+ ResetLatch(MyLatch);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+ }
+}
+
+/*
+ * Routines for handling the GUC variable(s)
+ */
+
+bool
+check_synchronize_slot_names(char **newval, void **extra, GucSource source)
+{
+ /* Special handling for "*" which means all. */
+ if (strcmp(*newval, "*") == 0)
+ {
+ return true;
+ }
+ else
+ {
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Parse string into list of identifiers */
+ if (!SplitIdentifierString(rawname, ',', &namelist))
+ {
+ /* syntax error in name list */
+ GUC_check_errdetail("List syntax is invalid.");
+ pfree(rawname);
+ list_free(namelist);
+ return false;
+ }
+
+ foreach(lc, namelist)
+ {
+ char *curname = (char *) lfirst(lc);
+
+ ReplicationSlotValidateName(curname, ERROR);
+ }
+
+ pfree(rawname);
+ list_free(namelist);
+ }
+
+ return true;
+}
+
+
+bool
+check_standby_slot_names(char **newval, void **extra, GucSource source)
+{
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Parse string into list of identifiers */
+ if (!SplitIdentifierString(rawname, ',', &namelist))
+ {
+ /* syntax error in name list */
+ GUC_check_errdetail("List syntax is invalid.");
+ pfree(rawname);
+ list_free(namelist);
+ return false;
+ }
+
+ foreach(lc, namelist)
+ {
+ char *curname = (char *) lfirst(lc);
+
+ ReplicationSlotValidateName(curname, ERROR);
+ }
+
+ pfree(rawname);
+ list_free(namelist);
+
+ return true;
+}
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 6d461654ab..5a98d2b699 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -100,6 +100,7 @@
#include "catalog/pg_subscription_rel.h"
#include "catalog/pg_type.h"
#include "commands/copy.h"
+#include "commands/subscriptioncmds.h"
#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "parser/parse_relation.h"
@@ -156,7 +157,8 @@ finish_sync_worker(void)
CommitTransactionCommand();
/* Find the leader apply worker and signal it. */
- logicalrep_worker_wakeup(MyLogicalRepWorker->subid, InvalidOid);
+ logicalrep_worker_wakeup(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid, InvalidOid);
/* Stop gracefully */
proc_exit(0);
@@ -196,7 +198,8 @@ wait_for_relation_state_change(Oid relid, char expected_state)
/* Check if the sync worker is still running and bail if not. */
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- worker = logicalrep_worker_find(MyLogicalRepWorker->subid, relid,
+ worker = logicalrep_worker_find(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid, relid,
false);
LWLockRelease(LogicalRepWorkerLock);
if (!worker)
@@ -243,7 +246,8 @@ wait_for_worker_state_change(char expected_state)
* waiting.
*/
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- worker = logicalrep_worker_find(MyLogicalRepWorker->subid,
+ worker = logicalrep_worker_find(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid,
InvalidOid, false);
if (worker && worker->proc)
logicalrep_worker_wakeup_ptr(worker);
@@ -509,7 +513,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn)
*/
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- syncworker = logicalrep_worker_find(MyLogicalRepWorker->subid,
+ syncworker = logicalrep_worker_find(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid,
rstate->relid, false);
if (syncworker)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 0ee764d68f..651f39c616 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1603,7 +1603,8 @@ apply_handle_stream_start(StringInfo s)
* Signal the leader apply worker, as it may be waiting for
* us.
*/
- logicalrep_worker_wakeup(MyLogicalRepWorker->subid, InvalidOid);
+ logicalrep_worker_wakeup(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid, InvalidOid);
}
parallel_stream_nchanges = 0;
diff --git a/src/backend/replication/repl_gram.y b/src/backend/replication/repl_gram.y
index 0c874e33cf..12a4b74368 100644
--- a/src/backend/replication/repl_gram.y
+++ b/src/backend/replication/repl_gram.y
@@ -76,11 +76,12 @@ Node *replication_parse_result;
%token K_EXPORT_SNAPSHOT
%token K_NOEXPORT_SNAPSHOT
%token K_USE_SNAPSHOT
+%token K_LIST_SLOTS
%type <node> command
%type <node> base_backup start_replication start_logical_replication
create_replication_slot drop_replication_slot identify_system
- read_replication_slot timeline_history show
+ read_replication_slot timeline_history show list_slots
%type <list> generic_option_list
%type <defelt> generic_option
%type <uintval> opt_timeline
@@ -91,6 +92,7 @@ Node *replication_parse_result;
%type <boolval> opt_temporary
%type <list> create_slot_options create_slot_legacy_opt_list
%type <defelt> create_slot_legacy_opt
+%type <list> slot_name_list slot_name_list_opt
%%
@@ -114,6 +116,7 @@ command:
| read_replication_slot
| timeline_history
| show
+ | list_slots
;
/*
@@ -126,6 +129,33 @@ identify_system:
}
;
+slot_name_list:
+ IDENT
+ {
+ $$ = list_make1($1);
+ }
+ | slot_name_list ',' IDENT
+ {
+ $$ = lappend($1, $3);
+ }
+
+slot_name_list_opt:
+ slot_name_list { $$ = $1; }
+ | /* EMPTY */ { $$ = NIL; }
+ ;
+
+/*
+ * LIST_SLOTS
+ */
+list_slots:
+ K_LIST_SLOTS slot_name_list_opt
+ {
+ ListSlotsCmd *cmd = makeNode(ListSlotsCmd);
+ cmd->slot_names = $2;
+ $$ = (Node *) cmd;
+ }
+ ;
+
/*
* READ_REPLICATION_SLOT %s
*/
diff --git a/src/backend/replication/repl_scanner.l b/src/backend/replication/repl_scanner.l
index 1cc7fb858c..11064feb86 100644
--- a/src/backend/replication/repl_scanner.l
+++ b/src/backend/replication/repl_scanner.l
@@ -128,6 +128,7 @@ DROP_REPLICATION_SLOT { return K_DROP_REPLICATION_SLOT; }
TIMELINE_HISTORY { return K_TIMELINE_HISTORY; }
PHYSICAL { return K_PHYSICAL; }
RESERVE_WAL { return K_RESERVE_WAL; }
+LIST_SLOTS { return K_LIST_SLOTS; }
LOGICAL { return K_LOGICAL; }
SLOT { return K_SLOT; }
TEMPORARY { return K_TEMPORARY; }
@@ -304,6 +305,7 @@ replication_scanner_is_replication_command(void)
case K_READ_REPLICATION_SLOT:
case K_TIMELINE_HISTORY:
case K_SHOW:
+ case K_LIST_SLOTS:
/* Yes; push back the first token so we can parse later. */
repl_pushed_back_token = first_token;
return true;
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 6035cf4816..83ada6db6a 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -467,7 +467,7 @@ pg_physical_replication_slot_advance(XLogRecPtr moveto)
* WAL and removal of old catalog tuples. As decoding is done in fast_forward
* mode, no changes are generated anyway.
*/
-static XLogRecPtr
+XLogRecPtr
pg_logical_replication_slot_advance(XLogRecPtr moveto)
{
LogicalDecodingContext *ctx;
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index cedadb0036..f905ae5722 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -473,6 +473,194 @@ IdentifySystem(void)
end_tup_output(tstate);
}
+static int
+pg_qsort_namecmp(const void *a, const void *b)
+{
+ return strncmp(NameStr(*(Name) a), NameStr(*(Name) b), NAMEDATALEN);
+}
+
+/*
+ * Handle the LIST_SLOTS command.
+ */
+static void
+ListSlots(ListSlotsCmd *cmd)
+{
+ DestReceiver *dest;
+ TupOutputState *tstate;
+ TupleDesc tupdesc;
+ NameData *slot_names;
+ int numslot_names;
+
+ numslot_names = list_length(cmd->slot_names);
+ if (numslot_names)
+ {
+ ListCell *lc;
+ int i = 0;
+
+ slot_names = palloc(numslot_names * sizeof(NameData));
+ foreach(lc, cmd->slot_names)
+ {
+ char *slot_name = lfirst(lc);
+
+ ReplicationSlotValidateName(slot_name, ERROR);
+ namestrcpy(&slot_names[i++], slot_name);
+ }
+
+ qsort(slot_names, numslot_names, sizeof(NameData), pg_qsort_namecmp);
+ }
+
+ dest = CreateDestReceiver(DestRemoteSimple);
+
+ /* need a tuple descriptor representing four columns */
+ tupdesc = CreateTemplateTupleDesc(10);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 1, "slot_name",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 2, "plugin",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 3, "slot_type",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 4, "datoid",
+ INT8OID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 5, "database",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 6, "temporary",
+ INT4OID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 7, "xmin",
+ INT8OID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 8, "catalog_xmin",
+ INT8OID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 9, "restart_lsn",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 10, "confirmed_flush",
+ TEXTOID, -1, 0);
+
+ /* prepare for projection of tuples */
+ tstate = begin_tup_output_tupdesc(dest, tupdesc, &TTSOpsVirtual);
+
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (int slotno = 0; slotno < max_replication_slots; slotno++)
+ {
+ ReplicationSlot *slot = &ReplicationSlotCtl->replication_slots[slotno];
+ char restart_lsn_str[MAXFNAMELEN];
+ char confirmed_flush_lsn_str[MAXFNAMELEN];
+ Datum values[10];
+ bool nulls[10];
+
+ ReplicationSlotPersistency persistency;
+ TransactionId xmin;
+ TransactionId catalog_xmin;
+ XLogRecPtr restart_lsn;
+ XLogRecPtr confirmed_flush_lsn;
+ Oid datoid;
+ NameData slot_name;
+ NameData plugin;
+ int i;
+ int64 tmpbigint;
+
+ if (!slot->in_use)
+ continue;
+
+ SpinLockAcquire(&slot->mutex);
+
+ xmin = slot->data.xmin;
+ catalog_xmin = slot->data.catalog_xmin;
+ datoid = slot->data.database;
+ restart_lsn = slot->data.restart_lsn;
+ confirmed_flush_lsn = slot->data.confirmed_flush;
+ namestrcpy(&slot_name, NameStr(slot->data.name));
+ namestrcpy(&plugin, NameStr(slot->data.plugin));
+ persistency = slot->data.persistency;
+
+ SpinLockRelease(&slot->mutex);
+
+ if (numslot_names &&
+ !bsearch((void *) &slot_name, (void *) slot_names,
+ numslot_names, sizeof(NameData), pg_qsort_namecmp))
+ continue;
+
+ memset(nulls, 0, sizeof(nulls));
+
+ i = 0;
+ values[i++] = CStringGetTextDatum(NameStr(slot_name));
+
+ if (datoid == InvalidOid)
+ nulls[i++] = true;
+ else
+ values[i++] = CStringGetTextDatum(NameStr(plugin));
+
+ if (datoid == InvalidOid)
+ values[i++] = CStringGetTextDatum("physical");
+ else
+ values[i++] = CStringGetTextDatum("logical");
+
+ if (datoid == InvalidOid)
+ nulls[i++] = true;
+ else
+ {
+ tmpbigint = datoid;
+ values[i++] = Int64GetDatum(tmpbigint);
+ }
+
+ if (datoid == InvalidOid)
+ nulls[i++] = true;
+ else
+ {
+ MemoryContext cur = CurrentMemoryContext;
+
+ /* syscache access needs a transaction env. */
+ StartTransactionCommand();
+ /* make dbname live outside TX context */
+ MemoryContextSwitchTo(cur);
+ values[i++] = CStringGetTextDatum(get_database_name(datoid));
+ CommitTransactionCommand();
+ /* CommitTransactionCommand switches to TopMemoryContext */
+ MemoryContextSwitchTo(cur);
+ }
+
+ values[i++] = Int32GetDatum(persistency == RS_TEMPORARY ? 1 : 0);
+
+ if (xmin != InvalidTransactionId)
+ {
+ tmpbigint = xmin;
+ values[i++] = Int64GetDatum(tmpbigint);
+ }
+ else
+ nulls[i++] = true;
+
+ if (catalog_xmin != InvalidTransactionId)
+ {
+ tmpbigint = catalog_xmin;
+ values[i++] = Int64GetDatum(tmpbigint);
+ }
+ else
+ nulls[i++] = true;
+
+ if (restart_lsn != InvalidXLogRecPtr)
+ {
+ snprintf(restart_lsn_str, sizeof(restart_lsn_str), "%X/%X",
+ LSN_FORMAT_ARGS(restart_lsn));
+ values[i++] = CStringGetTextDatum(restart_lsn_str);
+ }
+ else
+ nulls[i++] = true;
+
+ if (confirmed_flush_lsn != InvalidXLogRecPtr)
+ {
+ snprintf(confirmed_flush_lsn_str, sizeof(confirmed_flush_lsn_str),
+ "%X/%X", LSN_FORMAT_ARGS(confirmed_flush_lsn));
+ values[i++] = CStringGetTextDatum(confirmed_flush_lsn_str);
+ }
+ else
+ nulls[i++] = true;
+
+ /* send it to dest */
+ do_tup_output(tstate, values, nulls);
+ }
+ LWLockRelease(ReplicationSlotControlLock);
+
+ end_tup_output(tstate);
+}
+
/* Handle READ_REPLICATION_SLOT command */
static void
ReadReplicationSlot(ReadReplicationSlotCmd *cmd)
@@ -1819,6 +2007,13 @@ exec_replication_command(const char *cmd_string)
EndReplicationCommand(cmdtag);
break;
+ case T_ListSlotsCmd:
+ cmdtag = "LIST_SLOTS";
+ set_ps_display(cmdtag);
+ ListSlots((ListSlotsCmd *) cmd_node);
+ EndReplicationCommand(cmdtag);
+ break;
+
case T_StartReplicationCmd:
{
StartReplicationCmd *cmd = (StartReplicationCmd *) cmd_node;
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 7af1a61f95..9560748407 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -53,6 +53,7 @@ WAIT_EVENT_LOGICAL_APPLY_MAIN "LogicalApplyMain" "Waiting in main loop of logica
WAIT_EVENT_LOGICAL_LAUNCHER_MAIN "LogicalLauncherMain" "Waiting in main loop of logical replication launcher process."
WAIT_EVENT_LOGICAL_PARALLEL_APPLY_MAIN "LogicalParallelApplyMain" "Waiting in main loop of logical replication parallel apply process."
WAIT_EVENT_RECOVERY_WAL_STREAM "RecoveryWalStream" "Waiting in main loop of startup process for WAL to arrive, during streaming recovery."
+WAIT_EVENT_REPL_SLOT_SYNC_MAIN "ReplSlotSyncMain" "Waiting in main loop of worker for synchronizing slots to a standby from primary."
WAIT_EVENT_SYSLOGGER_MAIN "SysLoggerMain" "Waiting in main loop of syslogger process."
WAIT_EVENT_WAL_RECEIVER_MAIN "WalReceiverMain" "Waiting in main loop of WAL receiver process."
WAIT_EVENT_WAL_SENDER_MAIN "WalSenderMain" "Waiting in main loop of WAL sender process."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 59ab630ae4..4add70b54f 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -63,8 +63,12 @@
#include "postmaster/syslogger.h"
#include "postmaster/walwriter.h"
#include "replication/logicallauncher.h"
+#include "replication/logicalworker.h"
+#include "replication/reorderbuffer.h"
#include "replication/slot.h"
#include "replication/syncrep.h"
+#include "replication/walreceiver.h"
+#include "replication/walsender.h"
#include "storage/bufmgr.h"
#include "storage/large_object.h"
#include "storage/pg_shmem.h"
@@ -4557,6 +4561,28 @@ struct config_string ConfigureNamesString[] =
check_io_direct, assign_io_direct, NULL
},
+ {
+ {"synchronize_slot_names", PGC_SIGHUP, REPLICATION_STANDBY,
+ gettext_noop("Sets the names of replication slots which to synchronize from primary to standby."),
+ gettext_noop("Value of \"*\" means all."),
+ GUC_LIST_INPUT | GUC_LIST_QUOTE
+ },
+ &synchronize_slot_names,
+ "",
+ check_synchronize_slot_names, NULL, NULL
+ },
+
+ {
+ {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY,
+ gettext_noop("List of physical slots that must confirm changes before changes are sent to logical replication consumers."),
+ NULL,
+ GUC_LIST_INPUT | GUC_LIST_QUOTE
+ },
+ &standby_slot_names,
+ "",
+ check_standby_slot_names, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e4c0269fa3..7ea4364790 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -329,6 +329,8 @@
# method to choose sync standbys, number of sync standbys,
# and comma-separated list of application_name
# from standby(s); '*' = all
+#standby_slot_names = '' # physical standby slot names that logical replication
+ # waits for.
# - Standby Servers -
@@ -356,6 +358,7 @@
#wal_retrieve_retry_interval = 5s # time to wait before retrying to
# retrieve WAL after a failed attempt
#recovery_min_apply_delay = 0 # minimum delay for applying changes during recovery
+#synchronize_slot_names = '' # logical replication slots to sync to standby
# - Subscribers -
diff --git a/src/include/commands/subscriptioncmds.h b/src/include/commands/subscriptioncmds.h
index 214dc6c29e..0e77f9ee5c 100644
--- a/src/include/commands/subscriptioncmds.h
+++ b/src/include/commands/subscriptioncmds.h
@@ -17,6 +17,7 @@
#include "catalog/objectaddress.h"
#include "parser/parse_node.h"
+#include "replication/walreceiver.h"
extern ObjectAddress CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
bool isTopLevel);
@@ -28,4 +29,6 @@ extern void AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId);
extern char defGetStreamingMode(DefElem *def);
+extern void ReplicationSlotDropAtPubNode(WalReceiverConn *wrconn, char *slotname, bool missing_ok);
+
#endif /* SUBSCRIPTIONCMDS_H */
diff --git a/src/include/nodes/replnodes.h b/src/include/nodes/replnodes.h
index 4321ba8f86..980e0b2ee2 100644
--- a/src/include/nodes/replnodes.h
+++ b/src/include/nodes/replnodes.h
@@ -33,6 +33,15 @@ typedef struct IdentifySystemCmd
NodeTag type;
} IdentifySystemCmd;
+/* ----------------------
+ * LIST_SLOTS command
+ * ----------------------
+ */
+typedef struct ListSlotsCmd
+{
+ NodeTag type;
+ List *slot_names;
+} ListSlotsCmd;
/* ----------------------
* BASE_BACKUP command
diff --git a/src/include/replication/logicallauncher.h b/src/include/replication/logicallauncher.h
index a07c9cb311..80fdbf9657 100644
--- a/src/include/replication/logicallauncher.h
+++ b/src/include/replication/logicallauncher.h
@@ -31,4 +31,6 @@ extern bool IsLogicalLauncher(void);
extern pid_t GetLeaderApplyWorkerPid(pid_t pid);
+extern PGDLLIMPORT char *PrimaryConnInfo;
+
#endif /* LOGICALLAUNCHER_H */
diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h
index 39588da79f..6408753557 100644
--- a/src/include/replication/logicalworker.h
+++ b/src/include/replication/logicalworker.h
@@ -14,10 +14,16 @@
#include <signal.h>
+#include "utils/guc.h"
+
+extern char *synchronize_slot_names;
+extern char *standby_slot_names;
+
extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending;
extern void ApplyWorkerMain(Datum main_arg);
extern void ParallelApplyWorkerMain(Datum main_arg);
+extern void ReplSlotSyncMain(Datum main_arg);
extern bool IsLogicalWorker(void);
extern bool IsLogicalParallelApplyWorker(void);
@@ -29,4 +35,7 @@ extern void LogicalRepWorkersWakeupAtCommit(Oid subid);
extern void AtEOXact_LogicalRepWorkers(bool isCommit);
+extern bool check_synchronize_slot_names(char **newval, void **extra, GucSource source);
+extern bool check_standby_slot_names(char **newval, void **extra, GucSource source);
+
#endif /* LOGICALWORKER_H */
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index a8a89dc784..5dc2e0d30d 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -15,7 +15,6 @@
#include "storage/lwlock.h"
#include "storage/shmem.h"
#include "storage/spin.h"
-#include "replication/walreceiver.h"
/*
* Behaviour of replication slots, upon release or crash.
@@ -238,7 +237,6 @@ extern ReplicationSlot *SearchNamedReplicationSlot(const char *name, bool need_l
extern int ReplicationSlotIndex(ReplicationSlot *slot);
extern bool ReplicationSlotName(int index, Name name);
extern void ReplicationSlotNameForTablesync(Oid suboid, Oid relid, char *syncslotname, Size szslot);
-extern void ReplicationSlotDropAtPubNode(WalReceiverConn *wrconn, char *slotname, bool missing_ok);
extern void StartupReplicationSlots(void);
extern void CheckPointReplicationSlots(void);
@@ -246,4 +244,7 @@ extern void CheckPointReplicationSlots(void);
extern void CheckSlotRequirements(void);
extern void CheckSlotPermissions(void);
+extern XLogRecPtr pg_logical_replication_slot_advance(XLogRecPtr moveto);
+
+
#endif /* SLOT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 281626fa6f..9e9d64faf2 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -20,6 +20,7 @@
#include "pgtime.h"
#include "port/atomics.h"
#include "replication/logicalproto.h"
+#include "replication/slot.h"
#include "replication/walsender.h"
#include "storage/condition_variable.h"
#include "storage/latch.h"
@@ -191,6 +192,17 @@ typedef struct
} proto;
} WalRcvStreamOptions;
+/*
+ * Slot information receiver from remote.
+ *
+ * Currently same as ReplicationSlotPersistentData except last_sync_time
+ */
+typedef struct WalRecvReplicationSlotData
+{
+ ReplicationSlotPersistentData persistent_data;
+ TimestampTz last_sync_time;
+} WalRecvReplicationSlotData;
+
struct WalReceiverConn;
typedef struct WalReceiverConn WalReceiverConn;
@@ -280,6 +292,11 @@ typedef void (*walrcv_get_senderinfo_fn) (WalReceiverConn *conn,
typedef char *(*walrcv_identify_system_fn) (WalReceiverConn *conn,
TimeLineID *primary_tli);
+/*
+ * TODO
+ */
+typedef List *(*walrcv_list_slots_fn) (WalReceiverConn *conn, const char *slots);
+
/*
* walrcv_server_version_fn
*
@@ -393,6 +410,7 @@ typedef struct WalReceiverFunctionsType
walrcv_get_conninfo_fn walrcv_get_conninfo;
walrcv_get_senderinfo_fn walrcv_get_senderinfo;
walrcv_identify_system_fn walrcv_identify_system;
+ walrcv_list_slots_fn walrcv_list_slots;
walrcv_server_version_fn walrcv_server_version;
walrcv_readtimelinehistoryfile_fn walrcv_readtimelinehistoryfile;
walrcv_startstreaming_fn walrcv_startstreaming;
@@ -417,6 +435,8 @@ extern PGDLLIMPORT WalReceiverFunctionsType *WalReceiverFunctions;
WalReceiverFunctions->walrcv_get_senderinfo(conn, sender_host, sender_port)
#define walrcv_identify_system(conn, primary_tli) \
WalReceiverFunctions->walrcv_identify_system(conn, primary_tli)
+#define walrcv_list_slots(conn, slots) \
+ WalReceiverFunctions->walrcv_list_slots(conn, slots)
#define walrcv_server_version(conn) \
WalReceiverFunctions->walrcv_server_version(conn)
#define walrcv_readtimelinehistoryfile(conn, tli, filename, content, size) \
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index 343e781896..d42cff3f2c 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -65,7 +65,7 @@ typedef struct LogicalRepWorker
* would be created for each transaction which will be deleted after the
* transaction is finished.
*/
- FileSet *stream_fileset;
+ struct FileSet *stream_fileset;
/*
* PID of leader apply worker if this slot is used for a parallel apply
@@ -228,15 +228,15 @@ extern PGDLLIMPORT bool in_remote_transaction;
extern PGDLLIMPORT bool InitializingApplyWorker;
extern void logicalrep_worker_attach(int slot);
-extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid,
+extern LogicalRepWorker *logicalrep_worker_find(Oid dbid, Oid subid, Oid relid,
bool only_running);
extern List *logicalrep_workers_find(Oid subid, bool only_running);
extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname,
Oid userid, Oid relid,
dsm_handle subworker_dsm);
-extern void logicalrep_worker_stop(Oid subid, Oid relid);
+extern void logicalrep_worker_stop(Oid dbid, Oid subid, Oid relid);
extern void logicalrep_pa_worker_stop(ParallelApplyWorkerInfo *winfo);
-extern void logicalrep_worker_wakeup(Oid subid, Oid relid);
+extern void logicalrep_worker_wakeup(Oid dbid, Oid subid, Oid relid);
extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker);
extern int logicalrep_sync_worker_count(Oid subid);
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index 2008958010..b6fcc8704e 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -42,6 +42,7 @@ tests += {
't/034_create_database.pl',
't/035_standby_logical_decoding.pl',
't/036_truncated_dropped.pl',
+ 't/037_slot_sync.pl',
],
},
}
diff --git a/src/test/recovery/t/037_slot_sync.pl b/src/test/recovery/t/037_slot_sync.pl
new file mode 100644
index 0000000000..0520042d96
--- /dev/null
+++ b/src/test/recovery/t/037_slot_sync.pl
@@ -0,0 +1,130 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+my $node_phys_standby = PostgreSQL::Test::Cluster->new('phys_standby');
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+
+# find $pat in logfile of $node after $off-th byte
+sub find_in_log
+{
+ my ($node, $pat, $off) = @_;
+
+ $off = 0 unless defined $off;
+ my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile);
+ return 0 if (length($log) <= $off);
+
+ $log = substr($log, $off);
+
+ return $log =~ m/$pat/;
+}
+
+# Check invalidation in the logfile
+sub check_for_invalidation
+{
+ my ($log_start, $test_name) = @_;
+
+ # message should be issued
+ ok( find_in_log(
+ $node_phys_standby,
+ "invalidating obsolete replication slot \"sub1\"", $log_start),
+ "sub1 slot invalidation is logged $test_name");
+}
+
+# Check conflicting status in pg_replication_slots.
+sub check_slots_conflicting_status
+{
+ my $res = $node_phys_standby->safe_psql(
+ 'postgres', qq(
+ select bool_and(conflicting) from pg_replication_slots;));
+
+ is($res, 't',
+ "Logical slot is reported as conflicting");
+}
+
+$node_primary->init(allows_streaming => 'logical');
+$node_primary->append_conf('postgresql.conf', "standby_slot_names = 'pslot1'");
+$node_primary->start;
+$node_primary->psql('postgres', q{SELECT pg_create_physical_replication_slot('pslot1');});
+
+$node_primary->backup('backup');
+
+$node_phys_standby->init_from_backup($node_primary, 'backup', has_streaming => 1);
+$node_phys_standby->append_conf('postgresql.conf', q{
+synchronize_slot_names = '*'
+primary_slot_name = 'pslot1'
+hot_standby_feedback = off
+});
+
+$node_phys_standby->start;
+
+$node_primary->safe_psql('postgres', "CREATE TABLE t1 (a int PRIMARY KEY)");
+$node_primary->safe_psql('postgres', "INSERT INTO t1 VALUES (1), (2), (3)");
+
+# Some tests need to wait for VACUUM to be replayed. But vacuum does not flush
+# WAL. An insert into flush_wal outside transaction does guarantee a flush.
+$node_primary->psql('postgres', q[CREATE TABLE flush_wal();]);
+
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+$node_subscriber->safe_psql('postgres', "CREATE TABLE t1 (a int PRIMARY KEY)");
+
+$node_primary->safe_psql('postgres', "CREATE PUBLICATION pub1 FOR TABLE t1");
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1 CONNECTION '" . ($node_primary->connstr . ' dbname=postgres') . "' PUBLICATION pub1");
+
+# Wait for initial sync of all subscriptions
+my $synced_query =
+ "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result = $node_primary->safe_psql('postgres',
+ "SELECT slot_name, plugin, database FROM pg_replication_slots WHERE slot_type = 'logical'");
+
+is($result, qq(sub1|pgoutput|postgres), 'logical slot on primary');
+
+# FIXME: standby needs restart to pick up new slots
+$node_phys_standby->restart;
+sleep 3;
+
+$result = $node_phys_standby->safe_psql('postgres',
+ "SELECT slot_name, plugin, database FROM pg_replication_slots");
+
+is($result, qq(sub1|pgoutput|postgres), 'logical slot on standby');
+
+$node_primary->safe_psql('postgres', "INSERT INTO t1 VALUES (4), (5), (6)");
+$node_primary->wait_for_catchup('sub1');
+
+$node_primary->wait_for_catchup($node_phys_standby->name);
+
+# Logical subscriber and physical replica are caught up at this point.
+
+# Drop the subscription so that catalog_xmin is unknown on the primary
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub1");
+
+# This should trigger a conflict as hot_standby_feedback is off on the standby
+$node_primary->safe_psql('postgres', qq[
+ CREATE TABLE conflict_test(x integer, y text);
+ DROP TABLE conflict_test;
+ VACUUM full pg_class;
+ INSERT INTO flush_wal DEFAULT VALUES; -- see create table flush_wal
+]);
+
+# Ensure physical replay catches up
+$node_primary->wait_for_catchup($node_phys_standby);
+
+# Check invalidation in the logfile
+check_for_invalidation(1, 'with vacuum FULL on pg_class');
+
+# Check conflicting status in pg_replication_slots.
+check_slots_conflicting_status();
+
+done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-09 07:30 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
@ 2023-07-10 03:36 ` Amit Kapila <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Amit Kapila @ 2023-07-10 03:36 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Drouvot, Bertrand <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
On Sun, Jul 9, 2023 at 1:01 PM Bharath Rupireddy
<[email protected]> wrote:
>
> On Fri, Jun 16, 2023 at 3:26 PM Amit Kapila <[email protected]> wrote:
> >
> >
> > 1. Can you please try to explain the functionality of the overall
> > patch somewhere in the form of comments and or commit message?
>
> IIUC, there are 2 core ideas of the feature:
>
> 1) It will never let the logical replication subscribers go ahead of
> physical replication standbys specified in standby_slot_names. It
> implements this by delaying decoding of commit records on the
> walsenders corresponding to logical replication subscribers on the
> primary until all the specified standbys confirm receiving the commit
> LSN.
>
> 2) The physical replication standbys will synchronize data of the
> specified logical replication slots (in synchronize_slot_names) from
> the primary, creating the logical replication slots if necessary.
> Since the logical replication subscribers will never go out of
> physical replication standbys, the standbys can safely synchronize the
> slots and keep the data necessary for subscribers to connect to it and
> work seamlessly even after a failover.
>
> If my understanding is right,
>
This matches my understanding as well.
> I have few thoughts here:
>
> 1. All the logical walsenders are delayed on the primary - per
> wait_for_standby_confirmation() despite the user being interested in
> only a few of them via synchronize_slot_names. Shouldn't the delay be
> for just the slots specified in synchronize_slot_names?
> 2. I think we can split the patch like this - 0001 can be the logical
> walsenders delaying decoding on the primary unless standbys confirm,
> 0002 standby synchronizing the logical slots.
>
Agreed with the above two points.
> 3. I think we need to change the GUC standby_slot_names to better
> reflect what it is used for - wait_for_replication_slot_names or
> wait_for_
>
I feel at this stage we can focus on getting the design and
implementation correct. We can improve GUC names later once we are
confident that the functionality is correct.
> 4. It allows specifying logical slots in standby_slot_names, meaning,
> it can disallow logical slots getting ahead of other logical slots
> specified in standby_slot_names. Should we allow this case with the
> thinking that if there's anyone using logical replication for failover
> (well, will anybody do that in production?).
>
I think on the contrary we should prohibit this case. We can always
extend this functionality later.
> 5. Similar to above, it allows specifying physical slots in
> synchronize_slot_names. Should we disallow?
>
We should prohibit that as well.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
@ 2023-07-20 11:34 ` shveta malik <[email protected]>
2023-07-21 06:05 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2 siblings, 1 reply; 59+ messages in thread
From: shveta malik @ 2023-07-20 11:34 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Drouvot, Bertrand <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; shveta malik <[email protected]>
On Fri, Jun 16, 2023 at 3:26 PM Amit Kapila <[email protected]> wrote:
>
> On Mon, Apr 17, 2023 at 7:37 PM Drouvot, Bertrand
> <[email protected]> wrote:
> >
> 3. As mentioned in the initial email, I think it would be better to
> replace LIST_SLOTS command with a SELECT query.
>
I had a look at this thread. I am interested to work on this and can
spend some time addressing the comments given here.
I tried to replace LIST_SLOTS command with a SELECT query. Attached
rebased patch and PoC patch for LIST_SLOTS removal. For LIST_SLOTs cmd
removal, below are the points where more analysis is needed.
1) I could not use the exposed libpqwalreceiver's functions
walrcv_exec/libpqrcv_exec in LogicalRepLauncher to run select query
instead of LIST_SLOTS cmd. This is because libpqrcv_exec() needs
database connection but since in LogicalReplauncher, we do not have
any (MyDatabseId is not set), so the API gives an error. Thus to make
it work for the time-being, I used 'libpqrcv_PQexec' which is not
dependent upon database connection. But since it is not exposed "yet"
to other layers, I temporarily added the new code to
libpqwalreceiver.c itself. In fact I reused the existing function
wrapper libpqrcv_list_slots and changed the functionality to get info
using select query rather than list_slots.
2) While using connect API walrcv_connect/libpqrcv_connect(), we need
to tell it whether it is for logical or physical replication. In the
existing patch, where we were using LIST_SLOTS cmd, we have this
connection made with logical=false. But now since we need to run
select query to get the same info, using connection with logical=false
gives error on primary while executing select query. "ERROR: cannot
execute SQL commands in WAL sender for physical replication".
And thus in ApplyLauncherStartSlotSync(), I have changed connect API
to use logical=true for the time being. I noticed that in the
existing patch, it was using logical=false in
ApplyLauncherStartSlotSync() while logical=true in
synchronize_slots(). Possibly due to the same fact that logical=false
connection will not allow synchronize_slots() to run select query on
primary while it worked for ApplyLauncherStartSlotSync() as it was
running list_slots cmd instead of select query.
I am exploring further on these points to figure out which one is the
correct way to deal with these. Meanwhile posting this WIP patch for
early feedback. I will try addressing other comments as well in next
versions.
thanks
Shveta
Attachments:
[application/octet-stream] v1-0001-Remove-list_slots-command.patch (12.0K, ../../CAJpy0uCMNz3XERP-Vzp-7rHFztJgc6d+xsmUVCqsxWPkZvQz0Q@mail.gmail.com/2-v1-0001-Remove-list_slots-command.patch)
download | inline diff:
From f7bc02999302c5c714672c8b904a21982e597483 Mon Sep 17 00:00:00 2001
From: Shveta Malik <[email protected]>
Date: Thu, 20 Jul 2023 14:24:25 +0530
Subject: [PATCH v1] Remove list_slots command
Now list_sots has been replaced by a select query.
---
.../libpqwalreceiver/libpqwalreceiver.c | 20 +-
src/backend/replication/logical/launcher.c | 2 +-
src/backend/replication/repl_gram.y | 32 +--
src/backend/replication/repl_scanner.l | 2 -
src/backend/replication/walsender.c | 195 ------------------
src/include/nodes/replnodes.h | 10 -
6 files changed, 16 insertions(+), 245 deletions(-)
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 0e13cc2417..6580b3ef5b 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -425,7 +425,11 @@ libpqrcv_list_slots(WalReceiverConn *conn, const char *slot_names)
WalRecvReplicationSlotData *slot_data;
initStringInfo(&s);
- appendStringInfoString(&s, "LIST_SLOTS");
+ appendStringInfo(&s,
+ "SELECT slot_name, plugin, slot_type, "
+ "datoid, database, temporary, xmin, "
+ "catalog_xmin, restart_lsn, confirmed_flush_lsn "
+ "FROM pg_catalog.pg_replication_slots");
if (strcmp(slot_names, "") != 0 && strcmp(slot_names, "*") != 0)
{
@@ -433,16 +437,18 @@ libpqrcv_list_slots(WalReceiverConn *conn, const char *slot_names)
List *namelist;
ListCell *lc;
- appendStringInfoChar(&s, ' ');
rawname = pstrdup(slot_names);
SplitIdentifierString(rawname, ',', &namelist);
- foreach (lc, namelist)
+
+ appendStringInfoString(&s, " AND slot_name IN (");
+ foreach(lc, namelist)
{
if (lc != list_head(namelist))
appendStringInfoChar(&s, ',');
appendStringInfo(&s, "%s",
- quote_identifier(lfirst(lc)));
+ quote_literal_cstr(lfirst(lc)));
}
+ appendStringInfoChar(&s, ')');
}
res = libpqrcv_PQexec(conn->streamConn, s.data);
@@ -482,10 +488,12 @@ libpqrcv_list_slots(WalReceiverConn *conn, const char *slot_names)
if (OidIsValid(slot_data->persistent_data.database))
elog(ERROR, "unexpected physical replication slot with database set");
}
- if (pg_strtoint32(PQgetvalue(res, i, 5)) == 1)
+
+ if (strcmp(PQgetvalue(res, i, 5), "t") == 0)
slot_data->persistent_data.persistency = RS_TEMPORARY;
- else
+ else if (strcmp(PQgetvalue(res, i, 5), "f") == 0)
slot_data->persistent_data.persistency = RS_PERSISTENT;
+
if (!PQgetisnull(res, i, 6))
slot_data->persistent_data.xmin = atooid(PQgetvalue(res, i, 6));
if (!PQgetisnull(res, i, 7))
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 640f7647cc..0bf9ed8504 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -1117,7 +1117,7 @@ ApplyLauncherStartSlotSync(long *wait_time)
if (strcmp(synchronize_slot_names, "") == 0)
return;
- wrconn = walrcv_connect(PrimaryConnInfo, false, false,
+ wrconn = walrcv_connect(PrimaryConnInfo, true, false,
"Logical Replication Launcher", &err);
if (!wrconn)
ereport(ERROR,
diff --git a/src/backend/replication/repl_gram.y b/src/backend/replication/repl_gram.y
index 12a4b74368..0c874e33cf 100644
--- a/src/backend/replication/repl_gram.y
+++ b/src/backend/replication/repl_gram.y
@@ -76,12 +76,11 @@ Node *replication_parse_result;
%token K_EXPORT_SNAPSHOT
%token K_NOEXPORT_SNAPSHOT
%token K_USE_SNAPSHOT
-%token K_LIST_SLOTS
%type <node> command
%type <node> base_backup start_replication start_logical_replication
create_replication_slot drop_replication_slot identify_system
- read_replication_slot timeline_history show list_slots
+ read_replication_slot timeline_history show
%type <list> generic_option_list
%type <defelt> generic_option
%type <uintval> opt_timeline
@@ -92,7 +91,6 @@ Node *replication_parse_result;
%type <boolval> opt_temporary
%type <list> create_slot_options create_slot_legacy_opt_list
%type <defelt> create_slot_legacy_opt
-%type <list> slot_name_list slot_name_list_opt
%%
@@ -116,7 +114,6 @@ command:
| read_replication_slot
| timeline_history
| show
- | list_slots
;
/*
@@ -129,33 +126,6 @@ identify_system:
}
;
-slot_name_list:
- IDENT
- {
- $$ = list_make1($1);
- }
- | slot_name_list ',' IDENT
- {
- $$ = lappend($1, $3);
- }
-
-slot_name_list_opt:
- slot_name_list { $$ = $1; }
- | /* EMPTY */ { $$ = NIL; }
- ;
-
-/*
- * LIST_SLOTS
- */
-list_slots:
- K_LIST_SLOTS slot_name_list_opt
- {
- ListSlotsCmd *cmd = makeNode(ListSlotsCmd);
- cmd->slot_names = $2;
- $$ = (Node *) cmd;
- }
- ;
-
/*
* READ_REPLICATION_SLOT %s
*/
diff --git a/src/backend/replication/repl_scanner.l b/src/backend/replication/repl_scanner.l
index 11064feb86..1cc7fb858c 100644
--- a/src/backend/replication/repl_scanner.l
+++ b/src/backend/replication/repl_scanner.l
@@ -128,7 +128,6 @@ DROP_REPLICATION_SLOT { return K_DROP_REPLICATION_SLOT; }
TIMELINE_HISTORY { return K_TIMELINE_HISTORY; }
PHYSICAL { return K_PHYSICAL; }
RESERVE_WAL { return K_RESERVE_WAL; }
-LIST_SLOTS { return K_LIST_SLOTS; }
LOGICAL { return K_LOGICAL; }
SLOT { return K_SLOT; }
TEMPORARY { return K_TEMPORARY; }
@@ -305,7 +304,6 @@ replication_scanner_is_replication_command(void)
case K_READ_REPLICATION_SLOT:
case K_TIMELINE_HISTORY:
case K_SHOW:
- case K_LIST_SLOTS:
/* Yes; push back the first token so we can parse later. */
repl_pushed_back_token = first_token;
return true;
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 26d07ae549..d27ef2985d 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -473,194 +473,6 @@ IdentifySystem(void)
end_tup_output(tstate);
}
-static int
-pg_qsort_namecmp(const void *a, const void *b)
-{
- return strncmp(NameStr(*(Name) a), NameStr(*(Name) b), NAMEDATALEN);
-}
-
-/*
- * Handle the LIST_SLOTS command.
- */
-static void
-ListSlots(ListSlotsCmd *cmd)
-{
- DestReceiver *dest;
- TupOutputState *tstate;
- TupleDesc tupdesc;
- NameData *slot_names;
- int numslot_names;
-
- numslot_names = list_length(cmd->slot_names);
- if (numslot_names)
- {
- ListCell *lc;
- int i = 0;
-
- slot_names = palloc(numslot_names * sizeof(NameData));
- foreach(lc, cmd->slot_names)
- {
- char *slot_name = lfirst(lc);
-
- ReplicationSlotValidateName(slot_name, ERROR);
- namestrcpy(&slot_names[i++], slot_name);
- }
-
- qsort(slot_names, numslot_names, sizeof(NameData), pg_qsort_namecmp);
- }
-
- dest = CreateDestReceiver(DestRemoteSimple);
-
- /* need a tuple descriptor representing four columns */
- tupdesc = CreateTemplateTupleDesc(10);
- TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 1, "slot_name",
- TEXTOID, -1, 0);
- TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 2, "plugin",
- TEXTOID, -1, 0);
- TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 3, "slot_type",
- TEXTOID, -1, 0);
- TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 4, "datoid",
- INT8OID, -1, 0);
- TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 5, "database",
- TEXTOID, -1, 0);
- TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 6, "temporary",
- INT4OID, -1, 0);
- TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 7, "xmin",
- INT8OID, -1, 0);
- TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 8, "catalog_xmin",
- INT8OID, -1, 0);
- TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 9, "restart_lsn",
- TEXTOID, -1, 0);
- TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 10, "confirmed_flush",
- TEXTOID, -1, 0);
-
- /* prepare for projection of tuples */
- tstate = begin_tup_output_tupdesc(dest, tupdesc, &TTSOpsVirtual);
-
- LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
- for (int slotno = 0; slotno < max_replication_slots; slotno++)
- {
- ReplicationSlot *slot = &ReplicationSlotCtl->replication_slots[slotno];
- char restart_lsn_str[MAXFNAMELEN];
- char confirmed_flush_lsn_str[MAXFNAMELEN];
- Datum values[10];
- bool nulls[10];
-
- ReplicationSlotPersistency persistency;
- TransactionId xmin;
- TransactionId catalog_xmin;
- XLogRecPtr restart_lsn;
- XLogRecPtr confirmed_flush_lsn;
- Oid datoid;
- NameData slot_name;
- NameData plugin;
- int i;
- int64 tmpbigint;
-
- if (!slot->in_use)
- continue;
-
- SpinLockAcquire(&slot->mutex);
-
- xmin = slot->data.xmin;
- catalog_xmin = slot->data.catalog_xmin;
- datoid = slot->data.database;
- restart_lsn = slot->data.restart_lsn;
- confirmed_flush_lsn = slot->data.confirmed_flush;
- namestrcpy(&slot_name, NameStr(slot->data.name));
- namestrcpy(&plugin, NameStr(slot->data.plugin));
- persistency = slot->data.persistency;
-
- SpinLockRelease(&slot->mutex);
-
- if (numslot_names &&
- !bsearch((void *) &slot_name, (void *) slot_names,
- numslot_names, sizeof(NameData), pg_qsort_namecmp))
- continue;
-
- memset(nulls, 0, sizeof(nulls));
-
- i = 0;
- values[i++] = CStringGetTextDatum(NameStr(slot_name));
-
- if (datoid == InvalidOid)
- nulls[i++] = true;
- else
- values[i++] = CStringGetTextDatum(NameStr(plugin));
-
- if (datoid == InvalidOid)
- values[i++] = CStringGetTextDatum("physical");
- else
- values[i++] = CStringGetTextDatum("logical");
-
- if (datoid == InvalidOid)
- nulls[i++] = true;
- else
- {
- tmpbigint = datoid;
- values[i++] = Int64GetDatum(tmpbigint);
- }
-
- if (datoid == InvalidOid)
- nulls[i++] = true;
- else
- {
- MemoryContext cur = CurrentMemoryContext;
-
- /* syscache access needs a transaction env. */
- StartTransactionCommand();
- /* make dbname live outside TX context */
- MemoryContextSwitchTo(cur);
- values[i++] = CStringGetTextDatum(get_database_name(datoid));
- CommitTransactionCommand();
- /* CommitTransactionCommand switches to TopMemoryContext */
- MemoryContextSwitchTo(cur);
- }
-
- values[i++] = Int32GetDatum(persistency == RS_TEMPORARY ? 1 : 0);
-
- if (xmin != InvalidTransactionId)
- {
- tmpbigint = xmin;
- values[i++] = Int64GetDatum(tmpbigint);
- }
- else
- nulls[i++] = true;
-
- if (catalog_xmin != InvalidTransactionId)
- {
- tmpbigint = catalog_xmin;
- values[i++] = Int64GetDatum(tmpbigint);
- }
- else
- nulls[i++] = true;
-
- if (restart_lsn != InvalidXLogRecPtr)
- {
- snprintf(restart_lsn_str, sizeof(restart_lsn_str), "%X/%X",
- LSN_FORMAT_ARGS(restart_lsn));
- values[i++] = CStringGetTextDatum(restart_lsn_str);
- }
- else
- nulls[i++] = true;
-
- if (confirmed_flush_lsn != InvalidXLogRecPtr)
- {
- snprintf(confirmed_flush_lsn_str, sizeof(confirmed_flush_lsn_str),
- "%X/%X", LSN_FORMAT_ARGS(confirmed_flush_lsn));
- values[i++] = CStringGetTextDatum(confirmed_flush_lsn_str);
- }
- else
- nulls[i++] = true;
-
- /* send it to dest */
- do_tup_output(tstate, values, nulls);
- }
- LWLockRelease(ReplicationSlotControlLock);
-
- end_tup_output(tstate);
-}
-
/* Handle READ_REPLICATION_SLOT command */
static void
ReadReplicationSlot(ReadReplicationSlotCmd *cmd)
@@ -2007,13 +1819,6 @@ exec_replication_command(const char *cmd_string)
EndReplicationCommand(cmdtag);
break;
- case T_ListSlotsCmd:
- cmdtag = "LIST_SLOTS";
- set_ps_display(cmdtag);
- ListSlots((ListSlotsCmd *) cmd_node);
- EndReplicationCommand(cmdtag);
- break;
-
case T_StartReplicationCmd:
{
StartReplicationCmd *cmd = (StartReplicationCmd *) cmd_node;
diff --git a/src/include/nodes/replnodes.h b/src/include/nodes/replnodes.h
index 980e0b2ee2..b9c7ed61c6 100644
--- a/src/include/nodes/replnodes.h
+++ b/src/include/nodes/replnodes.h
@@ -33,16 +33,6 @@ typedef struct IdentifySystemCmd
NodeTag type;
} IdentifySystemCmd;
-/* ----------------------
- * LIST_SLOTS command
- * ----------------------
- */
-typedef struct ListSlotsCmd
-{
- NodeTag type;
- List *slot_names;
-} ListSlotsCmd;
-
/* ----------------------
* BASE_BACKUP command
* ----------------------
--
2.34.1
[application/octet-stream] v7-0001-Synchronize-logical-replication-slots-from-primar.patch (62.6K, ../../CAJpy0uCMNz3XERP-Vzp-7rHFztJgc6d+xsmUVCqsxWPkZvQz0Q@mail.gmail.com/3-v7-0001-Synchronize-logical-replication-slots-from-primar.patch)
download | inline diff:
From 73450271337d3050f61bc8b76276a91011cde7d8 Mon Sep 17 00:00:00 2001
From: Shveta Malik <[email protected]>
Date: Thu, 20 Jul 2023 14:03:28 +0530
Subject: [PATCH v7] Synchronize logical replication slots from primary to
standby.
---
doc/src/sgml/config.sgml | 34 ++
src/backend/commands/subscriptioncmds.c | 4 +-
src/backend/postmaster/bgworker.c | 3 +
.../libpqwalreceiver/libpqwalreceiver.c | 95 ++++
src/backend/replication/logical/Makefile | 1 +
src/backend/replication/logical/launcher.c | 263 +++++++----
src/backend/replication/logical/meson.build | 1 +
.../replication/logical/reorderbuffer.c | 86 ++++
src/backend/replication/logical/slotsync.c | 413 ++++++++++++++++++
src/backend/replication/logical/tablesync.c | 13 +-
src/backend/replication/logical/worker.c | 3 +-
src/backend/replication/repl_gram.y | 32 +-
src/backend/replication/repl_scanner.l | 2 +
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 195 +++++++++
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 26 ++
src/backend/utils/misc/postgresql.conf.sample | 3 +
src/include/commands/subscriptioncmds.h | 3 +
src/include/nodes/replnodes.h | 9 +
src/include/replication/logicallauncher.h | 2 +
src/include/replication/logicalworker.h | 9 +
src/include/replication/slot.h | 5 +-
src/include/replication/walreceiver.h | 20 +
src/include/replication/worker_internal.h | 8 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/038_slot_sync.pl | 130 ++++++
27 files changed, 1270 insertions(+), 94 deletions(-)
create mode 100644 src/backend/replication/logical/slotsync.c
create mode 100644 src/test/recovery/t/038_slot_sync.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 11251fa05e..28200fd70e 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4397,6 +4397,23 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-standby-slot-names" xreflabel="standby_slot_names">
+ <term><varname>standby_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>standby_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ List of physical replication slots that logical replication waits for.
+ If a logical replication connection is meant to switch to a physical
+ standby after the standby is promoted, the physical replication slot
+ for the standby should be listed here. This ensures that logical
+ replication is not ahead of the physical standby.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
@@ -4545,6 +4562,23 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
</listitem>
</varlistentry>
+ <varlistentry id="guc-synchronize_slot_names" xreflabel="synchronize_slot_names">
+ <term><varname>synchronize_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>synchronize_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies a list of logical replication slots that a physical standby
+ should synchronize from the primary server. This is necessary to be
+ able to retarget those logical replication connections to this standby
+ if it gets promoted. Specify <literal>*</literal> to synchronize all
+ logical replication slots. The default is empty.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index d4e798baeb..42e9b1056c 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -993,7 +993,7 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data,
RemoveSubscriptionRel(sub->oid, relid);
- logicalrep_worker_stop(sub->oid, relid);
+ logicalrep_worker_stop(MyDatabaseId, sub->oid, relid);
/*
* For READY state, we would have already dropped the
@@ -1591,7 +1591,7 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel)
{
LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc);
- logicalrep_worker_stop(w->subid, w->relid);
+ logicalrep_worker_stop(w->dbid, w->subid, w->relid);
}
list_free(subworkers);
diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c
index 5b4bd71694..f2f4475c3b 100644
--- a/src/backend/postmaster/bgworker.c
+++ b/src/backend/postmaster/bgworker.c
@@ -129,6 +129,9 @@ static const struct
{
"ApplyWorkerMain", ApplyWorkerMain
},
+ {
+ "ReplSlotSyncMain", ReplSlotSyncMain
+ },
{
"ParallelApplyWorkerMain", ParallelApplyWorkerMain
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 60d5c1fc40..0e13cc2417 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -34,6 +34,7 @@
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
#include "utils/tuplestore.h"
+#include "utils/varlena.h"
PG_MODULE_MAGIC;
@@ -58,6 +59,7 @@ static void libpqrcv_get_senderinfo(WalReceiverConn *conn,
char **sender_host, int *sender_port);
static char *libpqrcv_identify_system(WalReceiverConn *conn,
TimeLineID *primary_tli);
+static List *libpqrcv_list_slots(WalReceiverConn *conn, const char *slot_names);
static int libpqrcv_server_version(WalReceiverConn *conn);
static void libpqrcv_readtimelinehistoryfile(WalReceiverConn *conn,
TimeLineID tli, char **filename,
@@ -96,6 +98,7 @@ static WalReceiverFunctionsType PQWalReceiverFunctions = {
.walrcv_receive = libpqrcv_receive,
.walrcv_send = libpqrcv_send,
.walrcv_create_slot = libpqrcv_create_slot,
+ .walrcv_list_slots = libpqrcv_list_slots,
.walrcv_get_backend_pid = libpqrcv_get_backend_pid,
.walrcv_exec = libpqrcv_exec,
.walrcv_disconnect = libpqrcv_disconnect
@@ -409,6 +412,98 @@ libpqrcv_server_version(WalReceiverConn *conn)
return PQserverVersion(conn->streamConn);
}
+/*
+ * Get list of slots from primary.
+ */
+static List *
+libpqrcv_list_slots(WalReceiverConn *conn, const char *slot_names)
+{
+ PGresult *res;
+ List *slotlist = NIL;
+ int ntuples;
+ StringInfoData s;
+ WalRecvReplicationSlotData *slot_data;
+
+ initStringInfo(&s);
+ appendStringInfoString(&s, "LIST_SLOTS");
+
+ if (strcmp(slot_names, "") != 0 && strcmp(slot_names, "*") != 0)
+ {
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+
+ appendStringInfoChar(&s, ' ');
+ rawname = pstrdup(slot_names);
+ SplitIdentifierString(rawname, ',', &namelist);
+ foreach (lc, namelist)
+ {
+ if (lc != list_head(namelist))
+ appendStringInfoChar(&s, ',');
+ appendStringInfo(&s, "%s",
+ quote_identifier(lfirst(lc)));
+ }
+ }
+
+ res = libpqrcv_PQexec(conn->streamConn, s.data);
+ pfree(s.data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ PQclear(res);
+ ereport(ERROR,
+ (errmsg("could not receive list of slots the primary server: %s",
+ pchomp(PQerrorMessage(conn->streamConn)))));
+ }
+ if (PQnfields(res) < 10)
+ {
+ int nfields = PQnfields(res);
+
+ PQclear(res);
+ ereport(ERROR,
+ (errmsg("invalid response from primary server"),
+ errdetail("Could not get list of slots: got %d fields, expected %d or more fields.",
+ nfields, 10)));
+ }
+
+ ntuples = PQntuples(res);
+ for (int i = 0; i < ntuples; i++)
+ {
+ char *slot_type;
+
+ slot_data = palloc0(sizeof(WalRecvReplicationSlotData));
+ namestrcpy(&slot_data->persistent_data.name, PQgetvalue(res, i, 0));
+ if (!PQgetisnull(res, i, 1))
+ namestrcpy(&slot_data->persistent_data.plugin, PQgetvalue(res, i, 1));
+ slot_type = PQgetvalue(res, i, 2);
+ if (!PQgetisnull(res, i, 3))
+ slot_data->persistent_data.database = atooid(PQgetvalue(res, i, 3));
+ if (strcmp(slot_type, "physical") == 0)
+ {
+ if (OidIsValid(slot_data->persistent_data.database))
+ elog(ERROR, "unexpected physical replication slot with database set");
+ }
+ if (pg_strtoint32(PQgetvalue(res, i, 5)) == 1)
+ slot_data->persistent_data.persistency = RS_TEMPORARY;
+ else
+ slot_data->persistent_data.persistency = RS_PERSISTENT;
+ if (!PQgetisnull(res, i, 6))
+ slot_data->persistent_data.xmin = atooid(PQgetvalue(res, i, 6));
+ if (!PQgetisnull(res, i, 7))
+ slot_data->persistent_data.catalog_xmin = atooid(PQgetvalue(res, i, 7));
+ if (!PQgetisnull(res, i, 8))
+ slot_data->persistent_data.restart_lsn = strtou64(PQgetvalue(res, i, 8), NULL, 10);
+ if (!PQgetisnull(res, i, 9))
+ slot_data->persistent_data.confirmed_flush = strtou64(PQgetvalue(res, i, 9), NULL, 10);
+
+ slot_data->last_sync_time = 0;
+ slotlist = lappend(slotlist, slot_data);
+ }
+
+ PQclear(res);
+
+ return slotlist;
+}
+
/*
* Start streaming WAL data from given streaming options.
*
diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile
index 2dc25e37bb..ba03eeff1c 100644
--- a/src/backend/replication/logical/Makefile
+++ b/src/backend/replication/logical/Makefile
@@ -25,6 +25,7 @@ OBJS = \
proto.o \
relation.o \
reorderbuffer.o \
+ slotsync.o \
snapbuild.o \
tablesync.o \
worker.o
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 542af7d863..640f7647cc 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -22,6 +22,7 @@
#include "access/htup_details.h"
#include "access/tableam.h"
#include "access/xact.h"
+#include "catalog/pg_authid.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
#include "funcapi.h"
@@ -246,7 +247,7 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker,
* We are only interested in the leader apply worker or table sync worker.
*/
LogicalRepWorker *
-logicalrep_worker_find(Oid subid, Oid relid, bool only_running)
+logicalrep_worker_find(Oid dbid, Oid subid, Oid relid, bool only_running)
{
int i;
LogicalRepWorker *res = NULL;
@@ -262,8 +263,8 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running)
if (isParallelApplyWorker(w))
continue;
- if (w->in_use && w->subid == subid && w->relid == relid &&
- (!only_running || w->proc))
+ if (w->in_use && w->dbid == dbid && w->subid == subid &&
+ w->relid == relid && (!only_running || w->proc))
{
res = w;
break;
@@ -320,9 +321,13 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid,
/* Sanity check - tablesync worker cannot be a subworker */
Assert(!(is_parallel_apply_worker && OidIsValid(relid)));
- ereport(DEBUG1,
- (errmsg_internal("starting logical replication worker for subscription \"%s\"",
- subname)));
+ if (OidIsValid(subid))
+ ereport(DEBUG1,
+ (errmsg_internal("starting logical replication worker for subscription \"%s\"",
+ subname)));
+ else
+ ereport(DEBUG1,
+ (errmsg_internal("starting replication slot synchronization worker")));
/* Report this after the initial starting message for consistency. */
if (max_replication_slots == 0)
@@ -359,7 +364,9 @@ retry:
* reason we do this is because if some worker failed to start up and its
* parent has crashed while waiting, the in_use state was never cleared.
*/
- if (worker == NULL || nsyncworkers >= max_sync_workers_per_subscription)
+ if (worker == NULL ||
+ (OidIsValid(relid) &&
+ nsyncworkers >= max_sync_workers_per_subscription))
{
bool did_cleanup = false;
@@ -455,15 +462,20 @@ retry:
memset(&bgw, 0, sizeof(bgw));
bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
BGWORKER_BACKEND_DATABASE_CONNECTION;
- bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ bgw.bgw_start_time = BgWorkerStart_ConsistentState;
snprintf(bgw.bgw_library_name, MAXPGPATH, "postgres");
- if (is_parallel_apply_worker)
+ if (!OidIsValid(subid))
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ReplSlotSyncMain");
+ else if (is_parallel_apply_worker)
snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ParallelApplyWorkerMain");
else
snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain");
- if (OidIsValid(relid))
+ if (!OidIsValid(subid))
+ snprintf(bgw.bgw_name, BGW_MAXLEN,
+ "replication slot synchronization worker");
+ else if (OidIsValid(relid))
snprintf(bgw.bgw_name, BGW_MAXLEN,
"logical replication worker for subscription %u sync %u", subid, relid);
else if (is_parallel_apply_worker)
@@ -591,13 +603,13 @@ logicalrep_worker_stop_internal(LogicalRepWorker *worker, int signo)
* Stop the logical replication worker for subid/relid, if any.
*/
void
-logicalrep_worker_stop(Oid subid, Oid relid)
+logicalrep_worker_stop(Oid dbid, Oid subid, Oid relid)
{
LogicalRepWorker *worker;
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- worker = logicalrep_worker_find(subid, relid, false);
+ worker = logicalrep_worker_find(dbid, subid, relid, false);
if (worker)
{
@@ -658,13 +670,13 @@ logicalrep_pa_worker_stop(ParallelApplyWorkerInfo *winfo)
* Wake up (using latch) any logical replication worker for specified sub/rel.
*/
void
-logicalrep_worker_wakeup(Oid subid, Oid relid)
+logicalrep_worker_wakeup(Oid dbid, Oid subid, Oid relid)
{
LogicalRepWorker *worker;
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- worker = logicalrep_worker_find(subid, relid, true);
+ worker = logicalrep_worker_find(dbid, subid, relid, true);
if (worker)
logicalrep_worker_wakeup_ptr(worker);
@@ -909,7 +921,7 @@ ApplyLauncherRegister(void)
memset(&bgw, 0, sizeof(bgw));
bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
BGWORKER_BACKEND_DATABASE_CONNECTION;
- bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ bgw.bgw_start_time = BgWorkerStart_ConsistentState;
snprintf(bgw.bgw_library_name, MAXPGPATH, "postgres");
snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyLauncherMain");
snprintf(bgw.bgw_name, BGW_MAXLEN,
@@ -1092,6 +1104,157 @@ ApplyLauncherWakeup(void)
kill(LogicalRepCtx->launcher_pid, SIGUSR1);
}
+static void
+ApplyLauncherStartSlotSync(long *wait_time)
+{
+ WalReceiverConn *wrconn;
+ char *err;
+ List *slots;
+ ListCell *lc;
+ MemoryContext tmpctx;
+ MemoryContext oldctx;
+
+ if (strcmp(synchronize_slot_names, "") == 0)
+ return;
+
+ wrconn = walrcv_connect(PrimaryConnInfo, false, false,
+ "Logical Replication Launcher", &err);
+ if (!wrconn)
+ ereport(ERROR,
+ (errmsg("could not connect to the primary server: %s", err)));
+
+ /* Use temporary context for the slot list and worker info. */
+ tmpctx = AllocSetContextCreate(TopMemoryContext,
+ "Logical Replication Launcher slot sync ctx",
+ ALLOCSET_DEFAULT_SIZES);
+ oldctx = MemoryContextSwitchTo(tmpctx);
+
+ slots = walrcv_list_slots(wrconn, synchronize_slot_names);
+
+ foreach(lc, slots)
+ {
+ WalRecvReplicationSlotData *slot_data = lfirst(lc);
+ LogicalRepWorker *w;
+ TimestampTz last_sync;
+ TimestampTz now;
+ long elapsed;
+
+ if (!OidIsValid(slot_data->persistent_data.database))
+ continue;
+
+ LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
+ w = logicalrep_worker_find(slot_data->persistent_data.database, InvalidOid,
+ InvalidOid, false);
+ LWLockRelease(LogicalRepWorkerLock);
+
+ if (w != NULL)
+ continue; /* worker is running already */
+
+ /*
+ * If the worker is eligible to start now, launch it. Otherwise,
+ * adjust wait_time so that we'll wake up as soon as it can be
+ * started.
+ *
+ * Each apply worker can only be restarted once per
+ * wal_retrieve_retry_interval, so that errors do not cause us to
+ * repeatedly restart the worker as fast as possible.
+ */
+ last_sync = slot_data->last_sync_time;
+ now = GetCurrentTimestamp();
+ if (last_sync == 0 ||
+ (elapsed = TimestampDifferenceMilliseconds(last_sync, now)) >= wal_retrieve_retry_interval)
+ {
+ slot_data->last_sync_time = now;
+ logicalrep_worker_launch(slot_data->persistent_data.database,
+ InvalidOid, NULL,
+ BOOTSTRAP_SUPERUSERID, InvalidOid,
+ DSM_HANDLE_INVALID);
+ }
+ else
+ {
+ *wait_time = Min(*wait_time,
+ wal_retrieve_retry_interval - elapsed);
+ }
+ }
+
+ /* Switch back to original memory context. */
+ MemoryContextSwitchTo(oldctx);
+ /* Clean the temporary memory. */
+ MemoryContextDelete(tmpctx);
+
+ walrcv_disconnect(wrconn);
+}
+
+static void
+ApplyLauncherStartSubs(long *wait_time)
+{
+ List *sublist;
+ ListCell *lc;
+ MemoryContext subctx;
+ MemoryContext oldctx;
+
+ /* Use temporary context to avoid leaking memory across cycles. */
+ subctx = AllocSetContextCreate(TopMemoryContext,
+ "Logical Replication Launcher sublist",
+ ALLOCSET_DEFAULT_SIZES);
+ oldctx = MemoryContextSwitchTo(subctx);
+
+ /* Start the missing workers for enabled subscriptions. */
+ sublist = get_subscription_list();
+ foreach(lc, sublist)
+ {
+ Subscription *sub = (Subscription *) lfirst(lc);
+ LogicalRepWorker *w;
+ TimestampTz last_start;
+ TimestampTz now;
+ long elapsed;
+
+ if (!sub->enabled)
+ continue;
+
+ LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
+ w = logicalrep_worker_find(sub->dbid, sub->oid, InvalidOid, false);
+ LWLockRelease(LogicalRepWorkerLock);
+
+ if (w != NULL)
+ continue; /* worker is running already */
+
+ /*
+ * If the worker is eligible to start now, launch it. Otherwise,
+ * adjust wait_time so that we'll wake up as soon as it can be
+ * started.
+ *
+ * Each subscription's apply worker can only be restarted once per
+ * wal_retrieve_retry_interval, so that errors do not cause us to
+ * repeatedly restart the worker as fast as possible. In cases
+ * where a restart is expected (e.g., subscription parameter
+ * changes), another process should remove the last-start entry
+ * for the subscription so that the worker can be restarted
+ * without waiting for wal_retrieve_retry_interval to elapse.
+ */
+ last_start = ApplyLauncherGetWorkerStartTime(sub->oid);
+ now = GetCurrentTimestamp();
+ if (last_start == 0 ||
+ (elapsed = TimestampDifferenceMilliseconds(last_start, now)) >= wal_retrieve_retry_interval)
+ {
+ ApplyLauncherSetWorkerStartTime(sub->oid, now);
+ logicalrep_worker_launch(sub->dbid, sub->oid, sub->name,
+ sub->owner, InvalidOid,
+ DSM_HANDLE_INVALID);
+ }
+ else
+ {
+ *wait_time = Min(*wait_time,
+ wal_retrieve_retry_interval - elapsed);
+ }
+ }
+
+ /* Switch back to original memory context. */
+ MemoryContextSwitchTo(oldctx);
+ /* Clean the temporary memory. */
+ MemoryContextDelete(subctx);
+}
+
/*
* Main loop for the apply launcher process.
*/
@@ -1117,78 +1280,20 @@ ApplyLauncherMain(Datum main_arg)
*/
BackgroundWorkerInitializeConnection(NULL, NULL, 0);
+ load_file("libpqwalreceiver", false);
+
/* Enter main loop */
for (;;)
{
int rc;
- List *sublist;
- ListCell *lc;
- MemoryContext subctx;
- MemoryContext oldctx;
long wait_time = DEFAULT_NAPTIME_PER_CYCLE;
CHECK_FOR_INTERRUPTS();
- /* Use temporary context to avoid leaking memory across cycles. */
- subctx = AllocSetContextCreate(TopMemoryContext,
- "Logical Replication Launcher sublist",
- ALLOCSET_DEFAULT_SIZES);
- oldctx = MemoryContextSwitchTo(subctx);
-
- /* Start any missing workers for enabled subscriptions. */
- sublist = get_subscription_list();
- foreach(lc, sublist)
- {
- Subscription *sub = (Subscription *) lfirst(lc);
- LogicalRepWorker *w;
- TimestampTz last_start;
- TimestampTz now;
- long elapsed;
-
- if (!sub->enabled)
- continue;
-
- LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- w = logicalrep_worker_find(sub->oid, InvalidOid, false);
- LWLockRelease(LogicalRepWorkerLock);
-
- if (w != NULL)
- continue; /* worker is running already */
-
- /*
- * If the worker is eligible to start now, launch it. Otherwise,
- * adjust wait_time so that we'll wake up as soon as it can be
- * started.
- *
- * Each subscription's apply worker can only be restarted once per
- * wal_retrieve_retry_interval, so that errors do not cause us to
- * repeatedly restart the worker as fast as possible. In cases
- * where a restart is expected (e.g., subscription parameter
- * changes), another process should remove the last-start entry
- * for the subscription so that the worker can be restarted
- * without waiting for wal_retrieve_retry_interval to elapse.
- */
- last_start = ApplyLauncherGetWorkerStartTime(sub->oid);
- now = GetCurrentTimestamp();
- if (last_start == 0 ||
- (elapsed = TimestampDifferenceMilliseconds(last_start, now)) >= wal_retrieve_retry_interval)
- {
- ApplyLauncherSetWorkerStartTime(sub->oid, now);
- logicalrep_worker_launch(sub->dbid, sub->oid, sub->name,
- sub->owner, InvalidOid,
- DSM_HANDLE_INVALID);
- }
- else
- {
- wait_time = Min(wait_time,
- wal_retrieve_retry_interval - elapsed);
- }
- }
-
- /* Switch back to original memory context. */
- MemoryContextSwitchTo(oldctx);
- /* Clean the temporary memory. */
- MemoryContextDelete(subctx);
+ if (!RecoveryInProgress())
+ ApplyLauncherStartSubs(&wait_time);
+ else
+ ApplyLauncherStartSlotSync(&wait_time);
/* Wait for more work. */
rc = WaitLatch(MyLatch,
diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build
index d48cd4c590..9e52ec421f 100644
--- a/src/backend/replication/logical/meson.build
+++ b/src/backend/replication/logical/meson.build
@@ -11,6 +11,7 @@ backend_sources += files(
'proto.c',
'relation.c',
'reorderbuffer.c',
+ 'slotsync.c',
'snapbuild.c',
'tablesync.c',
'worker.c',
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 26d252bd87..98c93712ef 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -95,11 +95,14 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "replication/logical.h"
+#include "replication/logicalworker.h"
#include "replication/reorderbuffer.h"
#include "replication/slot.h"
#include "replication/snapbuild.h" /* just for SnapBuildSnapDecRefcount */
#include "storage/bufmgr.h"
#include "storage/fd.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
@@ -107,6 +110,7 @@
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
+#include "utils/varlena.h"
/* entry for a hash table we use to map from xid to our transaction state */
@@ -2053,6 +2057,85 @@ ReorderBufferResetTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
}
+static void
+wait_for_standby_confirmation(XLogRecPtr commit_lsn)
+{
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+ XLogRecPtr flush_pos = InvalidXLogRecPtr;
+
+ if (strcmp(standby_slot_names, "") == 0)
+ return;
+
+ rawname = pstrdup(standby_slot_names);
+ SplitIdentifierString(rawname, ',', &namelist);
+
+ while (true)
+ {
+ int wait_slots_remaining;
+ XLogRecPtr oldest_flush_pos = InvalidXLogRecPtr;
+ int rc;
+
+ wait_slots_remaining = list_length(namelist);
+
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (int i = 0; i < max_replication_slots; i++)
+ {
+ ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i];
+ bool inlist;
+
+ if (!s->in_use)
+ continue;
+
+ inlist = false;
+ foreach (lc, namelist)
+ {
+ char *name = lfirst(lc);
+ if (strcmp(name, NameStr(s->data.name)) == 0)
+ {
+ inlist = true;
+ break;
+ }
+ }
+ if (!inlist)
+ continue;
+
+ SpinLockAcquire(&s->mutex);
+
+ if (s->data.database == InvalidOid)
+ /* Physical slots advance restart_lsn on flush and ignore confirmed_flush_lsn */
+ flush_pos = s->data.restart_lsn;
+ else
+ /* For logical slots we must wait for commit and flush */
+ flush_pos = s->data.confirmed_flush;
+
+ SpinLockRelease(&s->mutex);
+
+ /* We want to find out the min(flush pos) over all named slots */
+ if (oldest_flush_pos == InvalidXLogRecPtr
+ || oldest_flush_pos > flush_pos)
+ oldest_flush_pos = flush_pos;
+
+ if (flush_pos >= commit_lsn && wait_slots_remaining > 0)
+ wait_slots_remaining --;
+ }
+ LWLockRelease(ReplicationSlotControlLock);
+
+ if (wait_slots_remaining == 0)
+ return;
+
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ 1000L, PG_WAIT_EXTENSION);
+
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+
+ CHECK_FOR_INTERRUPTS();
+ }
+}
+
/*
* Helper function for ReorderBufferReplay and ReorderBufferStreamTXN.
*
@@ -2502,6 +2585,9 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
* Call either PREPARE (for two-phase transactions) or COMMIT (for
* regular ones).
*/
+
+ wait_for_standby_confirmation(commit_lsn);
+
if (rbtxn_prepared(txn))
rb->prepare(rb, txn, commit_lsn);
else
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
new file mode 100644
index 0000000000..529ddb21ae
--- /dev/null
+++ b/src/backend/replication/logical/slotsync.c
@@ -0,0 +1,413 @@
+/*-------------------------------------------------------------------------
+ * slotsync.c
+ * PostgreSQL worker for synchronizing slots to a standby from primary
+ *
+ * Copyright (c) 2016-2018, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/replication/logical/slotsync.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "commands/dbcommands.h"
+#include "pgstat.h"
+#include "postmaster/bgworker.h"
+#include "replication/logicallauncher.h"
+#include "replication/logicalworker.h"
+#include "replication/walreceiver.h"
+#include "replication/worker_internal.h"
+#include "storage/ipc.h"
+#include "storage/procarray.h"
+#include "utils/builtins.h"
+#include "utils/pg_lsn.h"
+#include "utils/varlena.h"
+
+char *synchronize_slot_names;
+char *standby_slot_names;
+
+/*
+ * Wait for remote slot to pass localy reserved position.
+ */
+static void
+wait_for_primary_slot_catchup(WalReceiverConn *wrconn, char *slot_name,
+ XLogRecPtr min_lsn)
+{
+ WalRcvExecResult *res;
+ TupleTableSlot *slot;
+ Oid slotRow[1] = {LSNOID};
+ StringInfoData cmd;
+ bool isnull;
+ XLogRecPtr restart_lsn;
+
+ for (;;)
+ {
+ int rc;
+
+ CHECK_FOR_INTERRUPTS();
+
+ initStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT restart_lsn"
+ " FROM pg_catalog.pg_replication_slots"
+ " WHERE slot_name = %s",
+ quote_literal_cstr(slot_name));
+ res = walrcv_exec(wrconn, cmd.data, 1, slotRow);
+
+ if (res->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errmsg("could not fetch slot info for slot \"%s\" from primary: %s",
+ slot_name, res->err)));
+
+ slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+ if (!tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+ ereport(ERROR,
+ (errmsg("slot \"%s\" disapeared from provider",
+ slot_name)));
+
+ restart_lsn = DatumGetLSN(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ ExecClearTuple(slot);
+ walrcv_clear_result(res);
+
+ if (restart_lsn >= min_lsn)
+ break;
+
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ wal_retrieve_retry_interval,
+ WAIT_EVENT_REPL_SLOT_SYNC_MAIN);
+
+ ResetLatch(MyLatch);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+ }
+}
+
+/*
+ * Synchronize single slot to given position.
+ *
+ * This optionally creates new slot if there is no existing one.
+ */
+static void
+synchronize_one_slot(WalReceiverConn *wrconn, char *slot_name, char *database,
+ char *plugin_name, XLogRecPtr target_lsn)
+{
+ bool found = false;
+ XLogRecPtr endlsn;
+
+ /* Search for the named slot and mark it active if we find it. */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (int i = 0; i < max_replication_slots; i++)
+ {
+ ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i];
+
+ if (!s->in_use)
+ continue;
+
+ if (strcmp(NameStr(s->data.name), slot_name) == 0)
+ {
+ found = true;
+ break;
+ }
+ }
+ LWLockRelease(ReplicationSlotControlLock);
+
+ StartTransactionCommand();
+
+ /* Already existing slot, acquire */
+ if (found)
+ {
+ ReplicationSlotAcquire(slot_name, true);
+
+ if (target_lsn < MyReplicationSlot->data.confirmed_flush)
+ {
+ elog(DEBUG1,
+ "not synchronizing slot %s; synchronization would move it backward",
+ slot_name);
+
+ ReplicationSlotRelease();
+ CommitTransactionCommand();
+ return;
+ }
+ }
+ /* Otherwise create the slot first. */
+ else
+ {
+ TransactionId xmin_horizon = InvalidTransactionId;
+ ReplicationSlot *slot;
+
+ ReplicationSlotCreate(slot_name, true, RS_EPHEMERAL, false);
+ slot = MyReplicationSlot;
+
+ SpinLockAcquire(&slot->mutex);
+ slot->data.database = get_database_oid(database, false);
+ namestrcpy(&slot->data.plugin, plugin_name);
+ SpinLockRelease(&slot->mutex);
+
+ ReplicationSlotReserveWal();
+
+ LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
+ xmin_horizon = GetOldestSafeDecodingTransactionId(true);
+ slot->effective_catalog_xmin = xmin_horizon;
+ slot->data.catalog_xmin = xmin_horizon;
+ ReplicationSlotsComputeRequiredXmin(true);
+ LWLockRelease(ProcArrayLock);
+
+ if (target_lsn < MyReplicationSlot->data.restart_lsn)
+ {
+ ereport(LOG,
+ errmsg("waiting for remote slot \"%s\" LSN (%X/%X) to pass local slot LSN (%X/%X)",
+ slot_name,
+ LSN_FORMAT_ARGS(target_lsn), LSN_FORMAT_ARGS(MyReplicationSlot->data.restart_lsn)));
+
+ wait_for_primary_slot_catchup(wrconn, slot_name,
+ MyReplicationSlot->data.restart_lsn);
+ }
+
+ ReplicationSlotPersist();
+ }
+
+ endlsn = pg_logical_replication_slot_advance(target_lsn);
+
+ elog(DEBUG3, "synchronized slot %s to lsn (%X/%X)",
+ slot_name, LSN_FORMAT_ARGS(endlsn));
+
+ ReplicationSlotRelease();
+ CommitTransactionCommand();
+}
+
+static void
+synchronize_slots(void)
+{
+ WalRcvExecResult *res;
+ WalReceiverConn *wrconn = NULL;
+ TupleTableSlot *slot;
+ Oid slotRow[3] = {TEXTOID, TEXTOID, LSNOID};
+ StringInfoData s;
+ char *database;
+ char *err;
+ MemoryContext oldctx = CurrentMemoryContext;
+
+ if (!WalRcv)
+ return;
+
+ /* syscache access needs a transaction env. */
+ StartTransactionCommand();
+ /* make dbname live outside TX context */
+ MemoryContextSwitchTo(oldctx);
+
+ database = get_database_name(MyDatabaseId);
+ initStringInfo(&s);
+ appendStringInfo(&s, "%s dbname=%s", PrimaryConnInfo, database);
+ wrconn = walrcv_connect(s.data, true, false, "slot_sync", &err);
+
+ if (wrconn == NULL)
+ ereport(ERROR,
+ (errmsg("could not connect to the primary server: %s", err)));
+
+ resetStringInfo(&s);
+ appendStringInfo(&s,
+ "SELECT slot_name, plugin, confirmed_flush_lsn"
+ " FROM pg_catalog.pg_replication_slots"
+ " WHERE database = %s",
+ quote_literal_cstr(database));
+ if (strcmp(synchronize_slot_names, "") != 0 && strcmp(synchronize_slot_names, "*") != 0)
+ {
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+
+ rawname = pstrdup(synchronize_slot_names);
+ SplitIdentifierString(rawname, ',', &namelist);
+
+ appendStringInfoString(&s, " AND slot_name IN (");
+ foreach (lc, namelist)
+ {
+ if (lc != list_head(namelist))
+ appendStringInfoChar(&s, ',');
+ appendStringInfo(&s, "%s",
+ quote_literal_cstr(lfirst(lc)));
+ }
+ appendStringInfoChar(&s, ')');
+ }
+
+ res = walrcv_exec(wrconn, s.data, 3, slotRow);
+ pfree(s.data);
+
+ if (res->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errmsg("could not fetch slot info from primary: %s",
+ res->err)));
+
+ CommitTransactionCommand();
+ /* CommitTransactionCommand switches to TopMemoryContext */
+ MemoryContextSwitchTo(oldctx);
+
+ slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+ {
+ char *slot_name;
+ char *plugin_name;
+ XLogRecPtr confirmed_flush_lsn;
+ bool isnull;
+
+ slot_name = TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ plugin_name = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
+ Assert(!isnull);
+
+ confirmed_flush_lsn = DatumGetLSN(slot_getattr(slot, 3, &isnull));
+ Assert(!isnull);
+
+ synchronize_one_slot(wrconn, slot_name, database, plugin_name,
+ confirmed_flush_lsn);
+
+ ExecClearTuple(slot);
+ }
+
+ walrcv_clear_result(res);
+ pfree(database);
+
+ walrcv_disconnect(wrconn);
+}
+
+/*
+ * The main loop of our worker process.
+ */
+void
+ReplSlotSyncMain(Datum main_arg)
+{
+ int worker_slot = DatumGetInt32(main_arg);
+
+ /* Attach to slot */
+ logicalrep_worker_attach(worker_slot);
+
+ /* Establish signal handlers. */
+ BackgroundWorkerUnblockSignals();
+
+ /* Load the libpq-specific functions */
+ load_file("libpqwalreceiver", false);
+
+ /* Connect to our database. */
+ BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->userid,
+ 0);
+
+ StartTransactionCommand();
+ ereport(LOG,
+ (errmsg("replication slot synchronization worker for database \"%s\" has started",
+ get_database_name(MyLogicalRepWorker->dbid))));
+ CommitTransactionCommand();
+
+ /* Main wait loop. */
+ for (;;)
+ {
+ int rc;
+
+ CHECK_FOR_INTERRUPTS();
+
+ if (!RecoveryInProgress())
+ return;
+
+ if (strcmp(synchronize_slot_names, "") == 0)
+ return;
+
+ synchronize_slots();
+
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ wal_retrieve_retry_interval,
+ WAIT_EVENT_REPL_SLOT_SYNC_MAIN);
+
+ ResetLatch(MyLatch);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+ }
+}
+
+/*
+ * Routines for handling the GUC variable(s)
+ */
+
+bool
+check_synchronize_slot_names(char **newval, void **extra, GucSource source)
+{
+ /* Special handling for "*" which means all. */
+ if (strcmp(*newval, "*") == 0)
+ {
+ return true;
+ }
+ else
+ {
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Parse string into list of identifiers */
+ if (!SplitIdentifierString(rawname, ',', &namelist))
+ {
+ /* syntax error in name list */
+ GUC_check_errdetail("List syntax is invalid.");
+ pfree(rawname);
+ list_free(namelist);
+ return false;
+ }
+
+ foreach(lc, namelist)
+ {
+ char *curname = (char *) lfirst(lc);
+
+ ReplicationSlotValidateName(curname, ERROR);
+ }
+
+ pfree(rawname);
+ list_free(namelist);
+ }
+
+ return true;
+}
+
+
+bool
+check_standby_slot_names(char **newval, void **extra, GucSource source)
+{
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Parse string into list of identifiers */
+ if (!SplitIdentifierString(rawname, ',', &namelist))
+ {
+ /* syntax error in name list */
+ GUC_check_errdetail("List syntax is invalid.");
+ pfree(rawname);
+ list_free(namelist);
+ return false;
+ }
+
+ foreach(lc, namelist)
+ {
+ char *curname = (char *) lfirst(lc);
+
+ ReplicationSlotValidateName(curname, ERROR);
+ }
+
+ pfree(rawname);
+ list_free(namelist);
+
+ return true;
+}
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 6d461654ab..5a98d2b699 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -100,6 +100,7 @@
#include "catalog/pg_subscription_rel.h"
#include "catalog/pg_type.h"
#include "commands/copy.h"
+#include "commands/subscriptioncmds.h"
#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "parser/parse_relation.h"
@@ -156,7 +157,8 @@ finish_sync_worker(void)
CommitTransactionCommand();
/* Find the leader apply worker and signal it. */
- logicalrep_worker_wakeup(MyLogicalRepWorker->subid, InvalidOid);
+ logicalrep_worker_wakeup(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid, InvalidOid);
/* Stop gracefully */
proc_exit(0);
@@ -196,7 +198,8 @@ wait_for_relation_state_change(Oid relid, char expected_state)
/* Check if the sync worker is still running and bail if not. */
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- worker = logicalrep_worker_find(MyLogicalRepWorker->subid, relid,
+ worker = logicalrep_worker_find(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid, relid,
false);
LWLockRelease(LogicalRepWorkerLock);
if (!worker)
@@ -243,7 +246,8 @@ wait_for_worker_state_change(char expected_state)
* waiting.
*/
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- worker = logicalrep_worker_find(MyLogicalRepWorker->subid,
+ worker = logicalrep_worker_find(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid,
InvalidOid, false);
if (worker && worker->proc)
logicalrep_worker_wakeup_ptr(worker);
@@ -509,7 +513,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn)
*/
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- syncworker = logicalrep_worker_find(MyLogicalRepWorker->subid,
+ syncworker = logicalrep_worker_find(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid,
rstate->relid, false);
if (syncworker)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index dd353fd1cb..72f39ace7b 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1589,7 +1589,8 @@ apply_handle_stream_start(StringInfo s)
* Signal the leader apply worker, as it may be waiting for
* us.
*/
- logicalrep_worker_wakeup(MyLogicalRepWorker->subid, InvalidOid);
+ logicalrep_worker_wakeup(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid, InvalidOid);
}
parallel_stream_nchanges = 0;
diff --git a/src/backend/replication/repl_gram.y b/src/backend/replication/repl_gram.y
index 0c874e33cf..12a4b74368 100644
--- a/src/backend/replication/repl_gram.y
+++ b/src/backend/replication/repl_gram.y
@@ -76,11 +76,12 @@ Node *replication_parse_result;
%token K_EXPORT_SNAPSHOT
%token K_NOEXPORT_SNAPSHOT
%token K_USE_SNAPSHOT
+%token K_LIST_SLOTS
%type <node> command
%type <node> base_backup start_replication start_logical_replication
create_replication_slot drop_replication_slot identify_system
- read_replication_slot timeline_history show
+ read_replication_slot timeline_history show list_slots
%type <list> generic_option_list
%type <defelt> generic_option
%type <uintval> opt_timeline
@@ -91,6 +92,7 @@ Node *replication_parse_result;
%type <boolval> opt_temporary
%type <list> create_slot_options create_slot_legacy_opt_list
%type <defelt> create_slot_legacy_opt
+%type <list> slot_name_list slot_name_list_opt
%%
@@ -114,6 +116,7 @@ command:
| read_replication_slot
| timeline_history
| show
+ | list_slots
;
/*
@@ -126,6 +129,33 @@ identify_system:
}
;
+slot_name_list:
+ IDENT
+ {
+ $$ = list_make1($1);
+ }
+ | slot_name_list ',' IDENT
+ {
+ $$ = lappend($1, $3);
+ }
+
+slot_name_list_opt:
+ slot_name_list { $$ = $1; }
+ | /* EMPTY */ { $$ = NIL; }
+ ;
+
+/*
+ * LIST_SLOTS
+ */
+list_slots:
+ K_LIST_SLOTS slot_name_list_opt
+ {
+ ListSlotsCmd *cmd = makeNode(ListSlotsCmd);
+ cmd->slot_names = $2;
+ $$ = (Node *) cmd;
+ }
+ ;
+
/*
* READ_REPLICATION_SLOT %s
*/
diff --git a/src/backend/replication/repl_scanner.l b/src/backend/replication/repl_scanner.l
index 1cc7fb858c..11064feb86 100644
--- a/src/backend/replication/repl_scanner.l
+++ b/src/backend/replication/repl_scanner.l
@@ -128,6 +128,7 @@ DROP_REPLICATION_SLOT { return K_DROP_REPLICATION_SLOT; }
TIMELINE_HISTORY { return K_TIMELINE_HISTORY; }
PHYSICAL { return K_PHYSICAL; }
RESERVE_WAL { return K_RESERVE_WAL; }
+LIST_SLOTS { return K_LIST_SLOTS; }
LOGICAL { return K_LOGICAL; }
SLOT { return K_SLOT; }
TEMPORARY { return K_TEMPORARY; }
@@ -304,6 +305,7 @@ replication_scanner_is_replication_command(void)
case K_READ_REPLICATION_SLOT:
case K_TIMELINE_HISTORY:
case K_SHOW:
+ case K_LIST_SLOTS:
/* Yes; push back the first token so we can parse later. */
repl_pushed_back_token = first_token;
return true;
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 6035cf4816..83ada6db6a 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -467,7 +467,7 @@ pg_physical_replication_slot_advance(XLogRecPtr moveto)
* WAL and removal of old catalog tuples. As decoding is done in fast_forward
* mode, no changes are generated anyway.
*/
-static XLogRecPtr
+XLogRecPtr
pg_logical_replication_slot_advance(XLogRecPtr moveto)
{
LogicalDecodingContext *ctx;
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d27ef2985d..26d07ae549 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -473,6 +473,194 @@ IdentifySystem(void)
end_tup_output(tstate);
}
+static int
+pg_qsort_namecmp(const void *a, const void *b)
+{
+ return strncmp(NameStr(*(Name) a), NameStr(*(Name) b), NAMEDATALEN);
+}
+
+/*
+ * Handle the LIST_SLOTS command.
+ */
+static void
+ListSlots(ListSlotsCmd *cmd)
+{
+ DestReceiver *dest;
+ TupOutputState *tstate;
+ TupleDesc tupdesc;
+ NameData *slot_names;
+ int numslot_names;
+
+ numslot_names = list_length(cmd->slot_names);
+ if (numslot_names)
+ {
+ ListCell *lc;
+ int i = 0;
+
+ slot_names = palloc(numslot_names * sizeof(NameData));
+ foreach(lc, cmd->slot_names)
+ {
+ char *slot_name = lfirst(lc);
+
+ ReplicationSlotValidateName(slot_name, ERROR);
+ namestrcpy(&slot_names[i++], slot_name);
+ }
+
+ qsort(slot_names, numslot_names, sizeof(NameData), pg_qsort_namecmp);
+ }
+
+ dest = CreateDestReceiver(DestRemoteSimple);
+
+ /* need a tuple descriptor representing four columns */
+ tupdesc = CreateTemplateTupleDesc(10);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 1, "slot_name",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 2, "plugin",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 3, "slot_type",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 4, "datoid",
+ INT8OID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 5, "database",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 6, "temporary",
+ INT4OID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 7, "xmin",
+ INT8OID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 8, "catalog_xmin",
+ INT8OID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 9, "restart_lsn",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 10, "confirmed_flush",
+ TEXTOID, -1, 0);
+
+ /* prepare for projection of tuples */
+ tstate = begin_tup_output_tupdesc(dest, tupdesc, &TTSOpsVirtual);
+
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (int slotno = 0; slotno < max_replication_slots; slotno++)
+ {
+ ReplicationSlot *slot = &ReplicationSlotCtl->replication_slots[slotno];
+ char restart_lsn_str[MAXFNAMELEN];
+ char confirmed_flush_lsn_str[MAXFNAMELEN];
+ Datum values[10];
+ bool nulls[10];
+
+ ReplicationSlotPersistency persistency;
+ TransactionId xmin;
+ TransactionId catalog_xmin;
+ XLogRecPtr restart_lsn;
+ XLogRecPtr confirmed_flush_lsn;
+ Oid datoid;
+ NameData slot_name;
+ NameData plugin;
+ int i;
+ int64 tmpbigint;
+
+ if (!slot->in_use)
+ continue;
+
+ SpinLockAcquire(&slot->mutex);
+
+ xmin = slot->data.xmin;
+ catalog_xmin = slot->data.catalog_xmin;
+ datoid = slot->data.database;
+ restart_lsn = slot->data.restart_lsn;
+ confirmed_flush_lsn = slot->data.confirmed_flush;
+ namestrcpy(&slot_name, NameStr(slot->data.name));
+ namestrcpy(&plugin, NameStr(slot->data.plugin));
+ persistency = slot->data.persistency;
+
+ SpinLockRelease(&slot->mutex);
+
+ if (numslot_names &&
+ !bsearch((void *) &slot_name, (void *) slot_names,
+ numslot_names, sizeof(NameData), pg_qsort_namecmp))
+ continue;
+
+ memset(nulls, 0, sizeof(nulls));
+
+ i = 0;
+ values[i++] = CStringGetTextDatum(NameStr(slot_name));
+
+ if (datoid == InvalidOid)
+ nulls[i++] = true;
+ else
+ values[i++] = CStringGetTextDatum(NameStr(plugin));
+
+ if (datoid == InvalidOid)
+ values[i++] = CStringGetTextDatum("physical");
+ else
+ values[i++] = CStringGetTextDatum("logical");
+
+ if (datoid == InvalidOid)
+ nulls[i++] = true;
+ else
+ {
+ tmpbigint = datoid;
+ values[i++] = Int64GetDatum(tmpbigint);
+ }
+
+ if (datoid == InvalidOid)
+ nulls[i++] = true;
+ else
+ {
+ MemoryContext cur = CurrentMemoryContext;
+
+ /* syscache access needs a transaction env. */
+ StartTransactionCommand();
+ /* make dbname live outside TX context */
+ MemoryContextSwitchTo(cur);
+ values[i++] = CStringGetTextDatum(get_database_name(datoid));
+ CommitTransactionCommand();
+ /* CommitTransactionCommand switches to TopMemoryContext */
+ MemoryContextSwitchTo(cur);
+ }
+
+ values[i++] = Int32GetDatum(persistency == RS_TEMPORARY ? 1 : 0);
+
+ if (xmin != InvalidTransactionId)
+ {
+ tmpbigint = xmin;
+ values[i++] = Int64GetDatum(tmpbigint);
+ }
+ else
+ nulls[i++] = true;
+
+ if (catalog_xmin != InvalidTransactionId)
+ {
+ tmpbigint = catalog_xmin;
+ values[i++] = Int64GetDatum(tmpbigint);
+ }
+ else
+ nulls[i++] = true;
+
+ if (restart_lsn != InvalidXLogRecPtr)
+ {
+ snprintf(restart_lsn_str, sizeof(restart_lsn_str), "%X/%X",
+ LSN_FORMAT_ARGS(restart_lsn));
+ values[i++] = CStringGetTextDatum(restart_lsn_str);
+ }
+ else
+ nulls[i++] = true;
+
+ if (confirmed_flush_lsn != InvalidXLogRecPtr)
+ {
+ snprintf(confirmed_flush_lsn_str, sizeof(confirmed_flush_lsn_str),
+ "%X/%X", LSN_FORMAT_ARGS(confirmed_flush_lsn));
+ values[i++] = CStringGetTextDatum(confirmed_flush_lsn_str);
+ }
+ else
+ nulls[i++] = true;
+
+ /* send it to dest */
+ do_tup_output(tstate, values, nulls);
+ }
+ LWLockRelease(ReplicationSlotControlLock);
+
+ end_tup_output(tstate);
+}
+
/* Handle READ_REPLICATION_SLOT command */
static void
ReadReplicationSlot(ReadReplicationSlotCmd *cmd)
@@ -1819,6 +2007,13 @@ exec_replication_command(const char *cmd_string)
EndReplicationCommand(cmdtag);
break;
+ case T_ListSlotsCmd:
+ cmdtag = "LIST_SLOTS";
+ set_ps_display(cmdtag);
+ ListSlots((ListSlotsCmd *) cmd_node);
+ EndReplicationCommand(cmdtag);
+ break;
+
case T_StartReplicationCmd:
{
StartReplicationCmd *cmd = (StartReplicationCmd *) cmd_node;
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 3fabad96d9..7bfef97df1 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -53,6 +53,7 @@ WAIT_EVENT_LOGICAL_APPLY_MAIN LogicalApplyMain "Waiting in main loop of logical
WAIT_EVENT_LOGICAL_LAUNCHER_MAIN LogicalLauncherMain "Waiting in main loop of logical replication launcher process."
WAIT_EVENT_LOGICAL_PARALLEL_APPLY_MAIN LogicalParallelApplyMain "Waiting in main loop of logical replication parallel apply process."
WAIT_EVENT_RECOVERY_WAL_STREAM RecoveryWalStream "Waiting in main loop of startup process for WAL to arrive, during streaming recovery."
+WAIT_EVENT_REPL_SLOT_SYNC_MAIN ReplSlotSyncMain "Waiting in main loop of worker for synchronizing slots to a standby from primary."
WAIT_EVENT_SYSLOGGER_MAIN SysLoggerMain "Waiting in main loop of syslogger process."
WAIT_EVENT_WAL_RECEIVER_MAIN WalReceiverMain "Waiting in main loop of WAL receiver process."
WAIT_EVENT_WAL_SENDER_MAIN WalSenderMain "Waiting in main loop of WAL sender process."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index f9dba43b8c..b9f4e60bce 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -63,8 +63,12 @@
#include "postmaster/syslogger.h"
#include "postmaster/walwriter.h"
#include "replication/logicallauncher.h"
+#include "replication/logicalworker.h"
+#include "replication/reorderbuffer.h"
#include "replication/slot.h"
#include "replication/syncrep.h"
+#include "replication/walreceiver.h"
+#include "replication/walsender.h"
#include "storage/bufmgr.h"
#include "storage/large_object.h"
#include "storage/pg_shmem.h"
@@ -4551,6 +4555,28 @@ struct config_string ConfigureNamesString[] =
check_io_direct, assign_io_direct, NULL
},
+ {
+ {"synchronize_slot_names", PGC_SIGHUP, REPLICATION_STANDBY,
+ gettext_noop("Sets the names of replication slots which to synchronize from primary to standby."),
+ gettext_noop("Value of \"*\" means all."),
+ GUC_LIST_INPUT | GUC_LIST_QUOTE
+ },
+ &synchronize_slot_names,
+ "",
+ check_synchronize_slot_names, NULL, NULL
+ },
+
+ {
+ {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY,
+ gettext_noop("List of physical slots that must confirm changes before changes are sent to logical replication consumers."),
+ NULL,
+ GUC_LIST_INPUT | GUC_LIST_QUOTE
+ },
+ &standby_slot_names,
+ "",
+ check_standby_slot_names, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index c768af9a73..8962a9605c 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -328,6 +328,8 @@
# method to choose sync standbys, number of sync standbys,
# and comma-separated list of application_name
# from standby(s); '*' = all
+#standby_slot_names = '' # physical standby slot names that logical replication
+ # waits for.
# - Standby Servers -
@@ -355,6 +357,7 @@
#wal_retrieve_retry_interval = 5s # time to wait before retrying to
# retrieve WAL after a failed attempt
#recovery_min_apply_delay = 0 # minimum delay for applying changes during recovery
+#synchronize_slot_names = '' # logical replication slots to sync to standby
# - Subscribers -
diff --git a/src/include/commands/subscriptioncmds.h b/src/include/commands/subscriptioncmds.h
index 214dc6c29e..0e77f9ee5c 100644
--- a/src/include/commands/subscriptioncmds.h
+++ b/src/include/commands/subscriptioncmds.h
@@ -17,6 +17,7 @@
#include "catalog/objectaddress.h"
#include "parser/parse_node.h"
+#include "replication/walreceiver.h"
extern ObjectAddress CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
bool isTopLevel);
@@ -28,4 +29,6 @@ extern void AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId);
extern char defGetStreamingMode(DefElem *def);
+extern void ReplicationSlotDropAtPubNode(WalReceiverConn *wrconn, char *slotname, bool missing_ok);
+
#endif /* SUBSCRIPTIONCMDS_H */
diff --git a/src/include/nodes/replnodes.h b/src/include/nodes/replnodes.h
index 4321ba8f86..980e0b2ee2 100644
--- a/src/include/nodes/replnodes.h
+++ b/src/include/nodes/replnodes.h
@@ -33,6 +33,15 @@ typedef struct IdentifySystemCmd
NodeTag type;
} IdentifySystemCmd;
+/* ----------------------
+ * LIST_SLOTS command
+ * ----------------------
+ */
+typedef struct ListSlotsCmd
+{
+ NodeTag type;
+ List *slot_names;
+} ListSlotsCmd;
/* ----------------------
* BASE_BACKUP command
diff --git a/src/include/replication/logicallauncher.h b/src/include/replication/logicallauncher.h
index a07c9cb311..80fdbf9657 100644
--- a/src/include/replication/logicallauncher.h
+++ b/src/include/replication/logicallauncher.h
@@ -31,4 +31,6 @@ extern bool IsLogicalLauncher(void);
extern pid_t GetLeaderApplyWorkerPid(pid_t pid);
+extern PGDLLIMPORT char *PrimaryConnInfo;
+
#endif /* LOGICALLAUNCHER_H */
diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h
index 39588da79f..6408753557 100644
--- a/src/include/replication/logicalworker.h
+++ b/src/include/replication/logicalworker.h
@@ -14,10 +14,16 @@
#include <signal.h>
+#include "utils/guc.h"
+
+extern char *synchronize_slot_names;
+extern char *standby_slot_names;
+
extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending;
extern void ApplyWorkerMain(Datum main_arg);
extern void ParallelApplyWorkerMain(Datum main_arg);
+extern void ReplSlotSyncMain(Datum main_arg);
extern bool IsLogicalWorker(void);
extern bool IsLogicalParallelApplyWorker(void);
@@ -29,4 +35,7 @@ extern void LogicalRepWorkersWakeupAtCommit(Oid subid);
extern void AtEOXact_LogicalRepWorkers(bool isCommit);
+extern bool check_synchronize_slot_names(char **newval, void **extra, GucSource source);
+extern bool check_standby_slot_names(char **newval, void **extra, GucSource source);
+
#endif /* LOGICALWORKER_H */
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index a8a89dc784..5dc2e0d30d 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -15,7 +15,6 @@
#include "storage/lwlock.h"
#include "storage/shmem.h"
#include "storage/spin.h"
-#include "replication/walreceiver.h"
/*
* Behaviour of replication slots, upon release or crash.
@@ -238,7 +237,6 @@ extern ReplicationSlot *SearchNamedReplicationSlot(const char *name, bool need_l
extern int ReplicationSlotIndex(ReplicationSlot *slot);
extern bool ReplicationSlotName(int index, Name name);
extern void ReplicationSlotNameForTablesync(Oid suboid, Oid relid, char *syncslotname, Size szslot);
-extern void ReplicationSlotDropAtPubNode(WalReceiverConn *wrconn, char *slotname, bool missing_ok);
extern void StartupReplicationSlots(void);
extern void CheckPointReplicationSlots(void);
@@ -246,4 +244,7 @@ extern void CheckPointReplicationSlots(void);
extern void CheckSlotRequirements(void);
extern void CheckSlotPermissions(void);
+extern XLogRecPtr pg_logical_replication_slot_advance(XLogRecPtr moveto);
+
+
#endif /* SLOT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 281626fa6f..9e9d64faf2 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -20,6 +20,7 @@
#include "pgtime.h"
#include "port/atomics.h"
#include "replication/logicalproto.h"
+#include "replication/slot.h"
#include "replication/walsender.h"
#include "storage/condition_variable.h"
#include "storage/latch.h"
@@ -191,6 +192,17 @@ typedef struct
} proto;
} WalRcvStreamOptions;
+/*
+ * Slot information receiver from remote.
+ *
+ * Currently same as ReplicationSlotPersistentData except last_sync_time
+ */
+typedef struct WalRecvReplicationSlotData
+{
+ ReplicationSlotPersistentData persistent_data;
+ TimestampTz last_sync_time;
+} WalRecvReplicationSlotData;
+
struct WalReceiverConn;
typedef struct WalReceiverConn WalReceiverConn;
@@ -280,6 +292,11 @@ typedef void (*walrcv_get_senderinfo_fn) (WalReceiverConn *conn,
typedef char *(*walrcv_identify_system_fn) (WalReceiverConn *conn,
TimeLineID *primary_tli);
+/*
+ * TODO
+ */
+typedef List *(*walrcv_list_slots_fn) (WalReceiverConn *conn, const char *slots);
+
/*
* walrcv_server_version_fn
*
@@ -393,6 +410,7 @@ typedef struct WalReceiverFunctionsType
walrcv_get_conninfo_fn walrcv_get_conninfo;
walrcv_get_senderinfo_fn walrcv_get_senderinfo;
walrcv_identify_system_fn walrcv_identify_system;
+ walrcv_list_slots_fn walrcv_list_slots;
walrcv_server_version_fn walrcv_server_version;
walrcv_readtimelinehistoryfile_fn walrcv_readtimelinehistoryfile;
walrcv_startstreaming_fn walrcv_startstreaming;
@@ -417,6 +435,8 @@ extern PGDLLIMPORT WalReceiverFunctionsType *WalReceiverFunctions;
WalReceiverFunctions->walrcv_get_senderinfo(conn, sender_host, sender_port)
#define walrcv_identify_system(conn, primary_tli) \
WalReceiverFunctions->walrcv_identify_system(conn, primary_tli)
+#define walrcv_list_slots(conn, slots) \
+ WalReceiverFunctions->walrcv_list_slots(conn, slots)
#define walrcv_server_version(conn) \
WalReceiverFunctions->walrcv_server_version(conn)
#define walrcv_readtimelinehistoryfile(conn, tli, filename, content, size) \
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index 343e781896..d42cff3f2c 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -65,7 +65,7 @@ typedef struct LogicalRepWorker
* would be created for each transaction which will be deleted after the
* transaction is finished.
*/
- FileSet *stream_fileset;
+ struct FileSet *stream_fileset;
/*
* PID of leader apply worker if this slot is used for a parallel apply
@@ -228,15 +228,15 @@ extern PGDLLIMPORT bool in_remote_transaction;
extern PGDLLIMPORT bool InitializingApplyWorker;
extern void logicalrep_worker_attach(int slot);
-extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid,
+extern LogicalRepWorker *logicalrep_worker_find(Oid dbid, Oid subid, Oid relid,
bool only_running);
extern List *logicalrep_workers_find(Oid subid, bool only_running);
extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname,
Oid userid, Oid relid,
dsm_handle subworker_dsm);
-extern void logicalrep_worker_stop(Oid subid, Oid relid);
+extern void logicalrep_worker_stop(Oid dbid, Oid subid, Oid relid);
extern void logicalrep_pa_worker_stop(ParallelApplyWorkerInfo *winfo);
-extern void logicalrep_worker_wakeup(Oid subid, Oid relid);
+extern void logicalrep_worker_wakeup(Oid dbid, Oid subid, Oid relid);
extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker);
extern int logicalrep_sync_worker_count(Oid subid);
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index e7328e4894..86c3f9f1a4 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -43,6 +43,7 @@ tests += {
't/035_standby_logical_decoding.pl',
't/036_truncated_dropped.pl',
't/037_invalid_database.pl',
+ 't/038_slot_sync.pl',
],
},
}
diff --git a/src/test/recovery/t/038_slot_sync.pl b/src/test/recovery/t/038_slot_sync.pl
new file mode 100644
index 0000000000..0520042d96
--- /dev/null
+++ b/src/test/recovery/t/038_slot_sync.pl
@@ -0,0 +1,130 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+my $node_phys_standby = PostgreSQL::Test::Cluster->new('phys_standby');
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+
+# find $pat in logfile of $node after $off-th byte
+sub find_in_log
+{
+ my ($node, $pat, $off) = @_;
+
+ $off = 0 unless defined $off;
+ my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile);
+ return 0 if (length($log) <= $off);
+
+ $log = substr($log, $off);
+
+ return $log =~ m/$pat/;
+}
+
+# Check invalidation in the logfile
+sub check_for_invalidation
+{
+ my ($log_start, $test_name) = @_;
+
+ # message should be issued
+ ok( find_in_log(
+ $node_phys_standby,
+ "invalidating obsolete replication slot \"sub1\"", $log_start),
+ "sub1 slot invalidation is logged $test_name");
+}
+
+# Check conflicting status in pg_replication_slots.
+sub check_slots_conflicting_status
+{
+ my $res = $node_phys_standby->safe_psql(
+ 'postgres', qq(
+ select bool_and(conflicting) from pg_replication_slots;));
+
+ is($res, 't',
+ "Logical slot is reported as conflicting");
+}
+
+$node_primary->init(allows_streaming => 'logical');
+$node_primary->append_conf('postgresql.conf', "standby_slot_names = 'pslot1'");
+$node_primary->start;
+$node_primary->psql('postgres', q{SELECT pg_create_physical_replication_slot('pslot1');});
+
+$node_primary->backup('backup');
+
+$node_phys_standby->init_from_backup($node_primary, 'backup', has_streaming => 1);
+$node_phys_standby->append_conf('postgresql.conf', q{
+synchronize_slot_names = '*'
+primary_slot_name = 'pslot1'
+hot_standby_feedback = off
+});
+
+$node_phys_standby->start;
+
+$node_primary->safe_psql('postgres', "CREATE TABLE t1 (a int PRIMARY KEY)");
+$node_primary->safe_psql('postgres', "INSERT INTO t1 VALUES (1), (2), (3)");
+
+# Some tests need to wait for VACUUM to be replayed. But vacuum does not flush
+# WAL. An insert into flush_wal outside transaction does guarantee a flush.
+$node_primary->psql('postgres', q[CREATE TABLE flush_wal();]);
+
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+$node_subscriber->safe_psql('postgres', "CREATE TABLE t1 (a int PRIMARY KEY)");
+
+$node_primary->safe_psql('postgres', "CREATE PUBLICATION pub1 FOR TABLE t1");
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1 CONNECTION '" . ($node_primary->connstr . ' dbname=postgres') . "' PUBLICATION pub1");
+
+# Wait for initial sync of all subscriptions
+my $synced_query =
+ "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result = $node_primary->safe_psql('postgres',
+ "SELECT slot_name, plugin, database FROM pg_replication_slots WHERE slot_type = 'logical'");
+
+is($result, qq(sub1|pgoutput|postgres), 'logical slot on primary');
+
+# FIXME: standby needs restart to pick up new slots
+$node_phys_standby->restart;
+sleep 3;
+
+$result = $node_phys_standby->safe_psql('postgres',
+ "SELECT slot_name, plugin, database FROM pg_replication_slots");
+
+is($result, qq(sub1|pgoutput|postgres), 'logical slot on standby');
+
+$node_primary->safe_psql('postgres', "INSERT INTO t1 VALUES (4), (5), (6)");
+$node_primary->wait_for_catchup('sub1');
+
+$node_primary->wait_for_catchup($node_phys_standby->name);
+
+# Logical subscriber and physical replica are caught up at this point.
+
+# Drop the subscription so that catalog_xmin is unknown on the primary
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub1");
+
+# This should trigger a conflict as hot_standby_feedback is off on the standby
+$node_primary->safe_psql('postgres', qq[
+ CREATE TABLE conflict_test(x integer, y text);
+ DROP TABLE conflict_test;
+ VACUUM full pg_class;
+ INSERT INTO flush_wal DEFAULT VALUES; -- see create table flush_wal
+]);
+
+# Ensure physical replay catches up
+$node_primary->wait_for_catchup($node_phys_standby);
+
+# Check invalidation in the logfile
+check_for_invalidation(1, 'with vacuum FULL on pg_class');
+
+# Check conflicting status in pg_replication_slots.
+check_slots_conflicting_status();
+
+done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-20 11:34 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
@ 2023-07-21 06:05 ` Bharath Rupireddy <[email protected]>
2023-07-21 11:46 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Bharath Rupireddy @ 2023-07-21 06:05 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Amit Kapila <[email protected]>; Drouvot, Bertrand <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
On Thu, Jul 20, 2023 at 5:05 PM shveta malik <[email protected]> wrote:
>
> On Fri, Jun 16, 2023 at 3:26 PM Amit Kapila <[email protected]> wrote:
> >
> > On Mon, Apr 17, 2023 at 7:37 PM Drouvot, Bertrand
> > <[email protected]> wrote:
> > >
>
> > 3. As mentioned in the initial email, I think it would be better to
> > replace LIST_SLOTS command with a SELECT query.
> >
>
> I had a look at this thread. I am interested to work on this and can
> spend some time addressing the comments given here.
Thanks for your interest. Coincidentally, I started to split the patch
into 2 recently - 0001 making the specified logical wal senders wait
for specified standbys to ack, 0002 synchronize logical slots. I think
I'll have these patches ready by early next week. For 0002, I'll
consider your latest changes having LIST_SLOTS removed.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-20 11:34 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-21 06:05 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
@ 2023-07-21 11:46 ` shveta malik <[email protected]>
2023-07-24 02:32 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: shveta malik @ 2023-07-21 11:46 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Amit Kapila <[email protected]>; Drouvot, Bertrand <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; shveta malik <[email protected]>
On Fri, Jul 21, 2023 at 11:36 AM Bharath Rupireddy
<[email protected]> wrote:
>
> On Thu, Jul 20, 2023 at 5:05 PM shveta malik <[email protected]> wrote:
> >
> > On Fri, Jun 16, 2023 at 3:26 PM Amit Kapila <[email protected]> wrote:
> > >
> > > On Mon, Apr 17, 2023 at 7:37 PM Drouvot, Bertrand
> > > <[email protected]> wrote:
> > > >
> >
> > > 3. As mentioned in the initial email, I think it would be better to
> > > replace LIST_SLOTS command with a SELECT query.
> > >
> >
> > I had a look at this thread. I am interested to work on this and can
> > spend some time addressing the comments given here.
>
> Thanks for your interest. Coincidentally, I started to split the patch
> into 2 recently - 0001 making the specified logical wal senders wait
> for specified standbys to ack, 0002 synchronize logical slots. I think
> I'll have these patches ready by early next week. For 0002, I'll
> consider your latest changes having LIST_SLOTS removed.
>
Thanks Bharat for letting us know. It is okay to split the patch, it
may definitely help to understand the modules better but shall we take
a step back and try to reevaluate the design first before moving to
other tasks?
I analyzed more on the issues stated in [1] for replacing LIST_SLOTS
with SELECT query. On rethinking, it might not be a good idea to
replace this cmd with SELECT in Launcher code-path, because we do not
have any database-connection in launcher and 'select' query needs one
and thus we need to supply dbname to it. We may take the
primary-dbname info in a new GUC from users, but I feel retaining LIST
cmd is a better idea over adding a new GUC. But I do not see a reason
why we should get complete replication-slots info in LIST command. The
logic in Launcher is to get distinct database-ids info out of all the
slots and start worker per database-id. Since we are only interested
in database-id, I think we should change LIST_SLOTS to something like
LIST_DBID_FOR_LOGICAL_SLOTS. This new command may get only unique
database-ids for all the logical-slots (or the ones mentioned in
synchronize_slot_names) from primary. By doing so, we can avoid huge
network traffic in cases where the number of replication slots is
quite high considering that max_replication_slots can go upto
MAX_BACKENDS:2^18-1. So I plan to make this change where we retain
LIST cmd over SELECT query but make this cmd's output restricted to
only database-Ids. Thoughts?
Secondly, I was thinking if the design proposed in the patch is the
best one. No doubt, it is the most simplistic design and thus may
prove very efficient for scenarios where we have a reasonable number
of workers starting and each one actively busy in
slots-synchronisation, handling almost equivalent load. But since we
are starting one worker per database id, it may not be most efficient
for cases where not all the databases are actively being used. We may
have some workers (started for databases not in use) just waking up
and sending queries to primary and then going back to sleep and in the
process generating network traffic, while others may be heavily loaded
to deal with large numbers of active slots for a heavily loaded
database. I feel the design should be adaptable to load conditions
i.e. if we have more number of actively used slots, then we should
have more workers spawned to handle it and when the work is less then
the number of spawned workers should be less.
I have not thought it thoroughly yet and also not sure whether it will
actually come out as a better one, but it or more such designs should
be considered before we start fixing bugs in this patch. Kindly let me
know if there are already discussions around it which I might have
missed. Any feedback is appreciated.
[1] : https://www.postgresql.org/message-id/CAJpy0uCMNz3XERP-Vzp-7rHFztJgc6d%2BxsmUVCqsxWPkZvQz0Q%40mail.g...
thanks
Shveta
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-20 11:34 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-21 06:05 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-21 11:46 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
@ 2023-07-24 02:32 ` Bharath Rupireddy <[email protected]>
2023-07-24 03:30 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-26 12:25 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-08-04 07:40 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
0 siblings, 3 replies; 59+ messages in thread
From: Bharath Rupireddy @ 2023-07-24 02:32 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Amit Kapila <[email protected]>; Drouvot, Bertrand <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
On Fri, Jul 21, 2023 at 5:16 PM shveta malik <[email protected]> wrote:
>
> Thanks Bharat for letting us know. It is okay to split the patch, it
> may definitely help to understand the modules better but shall we take
> a step back and try to reevaluate the design first before moving to
> other tasks?
Agree that design comes first. FWIW, I'm attaching the v9 patch set
that I have with me. It can't be a perfect patch set unless the design
is finalized.
> I analyzed more on the issues stated in [1] for replacing LIST_SLOTS
> with SELECT query. On rethinking, it might not be a good idea to
> replace this cmd with SELECT in Launcher code-path
I think there are open fundamental design aspects, before optimizing
LIST_SLOTS, see below. I'm sure we can come back to this later.
> Secondly, I was thinking if the design proposed in the patch is the
> best one. No doubt, it is the most simplistic design and thus may
> .......... Any feedback is appreciated.
Here are my thoughts about this feature:
Current design:
1. On primary, never allow walsenders associated with logical
replication slots to go ahead of physical standbys that are candidates
for future primary after failover. This enables subscribers to connect
to new primary after failover.
2. On all candidate standbys, periodically sync logical slots from
primary (creating the slots if necessary) with one slot sync worker
per logical slot.
Important considerations:
1. Does this design guarantee the row versions required by subscribers
aren't removed on candidate standbys as raised here -
https://www.postgresql.org/message-id/20220218222319.yozkbhren7vkjbi5%40alap3.anarazel.de?
It seems safe with logical decoding on standbys feature. Also, a
test-case from upthread is already in patch sets (in v9 too)
https://www.postgresql.org/message-id/CAAaqYe9FdKODa1a9n%3Dqj%2Bw3NiB9gkwvhRHhcJNginuYYRCnLrg%40mail....
However, we need to verify the use cases extensively.
2. All candidate standbys will start one slot sync worker per logical
slot which might not be scalable. Is having one (or a few more - not
necessarily one for each logical slot) worker for all logical slots
enough?
It seems safe to have one worker for all logical slots - it's not a
problem even if the worker takes a bit of time to get to sync a
logical slot on a candidate standby, because the standby is ensured to
retain all the WAL and row versions required to decode and send to the
logical slots.
3. Indefinite waiting of logical walsenders for candidate standbys may
not be a good idea. Is having a timeout for logical walsenders a good
idea?
A problem with timeout is that it can make logical slots unusable
after failover.
4. All candidate standbys retain WAL required by logical slots. Amount
of WAL retained may be huge if there's a replication lag with logical
replication subscribers.
This turns out to be a typical problem with replication, so there's
nothing much this feature can do to prevent WAL file accumulation
except for asking one to monitor replication lag and WAL file growth.
5. Logical subscribers replication lag will depend on all candidate
standbys replication lag. If candidate standbys are too far from
primary and logical subscribers are too close, still logical
subscribers will have replication lag. There's nothing much this
feature can do to prevent this except for calling it out in
documentation.
6. This feature might need to prevent the GUCs from deviating on
primary and the candidate standbys - there's no point in syncing a
logical slot on candidate standbys if logical walsender related to it
on primary isn't keeping itself behind all the candidate standbys. If
preventing this from happening proves to be tough, calling it out in
documentation to keep GUCs the same is a good start.
7. There are some important review comments provided upthread as far
as this design and patches are concerned -
https://www.postgresql.org/message-id/20220207204557.74mgbhowydjco4mh%40alap3.anarazel.de
and https://www.postgresql.org/message-id/20220207203222.22aktwxrt3fcllru%40alap3.anarazel.de.
I'm sure we can come to these once the design is clear.
Please feel free to add the list if I'm missing anything.
Thoughts?
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/x-patch] v9-0001-Allow-logical-walsenders-to-wait-for-physical-sta.patch (21.1K, ../../CALj2ACV+VX9McnogGNyFCjZW+qnPvdmjnBjttotygs8+7D5JuA@mail.gmail.com/2-v9-0001-Allow-logical-walsenders-to-wait-for-physical-sta.patch)
download | inline diff:
From 666a73e79d9965779488db1cac6cd2d0a2c73ffb Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Sat, 22 Jul 2023 10:17:48 +0000
Subject: [PATCH v9] Allow logical walsenders to wait for physical standbys
---
doc/src/sgml/config.sgml | 42 ++++
.../replication/logical/reorderbuffer.c | 9 +
src/backend/replication/slot.c | 216 +++++++++++++++++-
src/backend/utils/misc/guc_tables.c | 30 +++
src/backend/utils/misc/postgresql.conf.sample | 4 +
src/include/replication/slot.h | 4 +
src/include/utils/guc_hooks.h | 4 +
src/test/recovery/meson.build | 1 +
src/test/recovery/t/050_verify_slot_order.pl | 146 ++++++++++++
9 files changed, 455 insertions(+), 1 deletion(-)
create mode 100644 src/test/recovery/t/050_verify_slot_order.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 11251fa05e..83a7d2e87e 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4397,6 +4397,24 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-standby-slot-names" xreflabel="standby_slot_names">
+ <term><varname>standby_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>standby_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ List of physical replication slots that logical replication waits for.
+ Specify <literal>*</literal> to wait for all physical replication
+ slots. If a logical replication connection is meant to switch to a
+ physical standby after the standby is promoted, the physical
+ replication slot for the standby should be listed here. This ensures
+ that logical replication is not ahead of the physical standby.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
@@ -4545,6 +4563,30 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
</listitem>
</varlistentry>
+ <varlistentry id="guc-synchronize_slot_names" xreflabel="synchronize_slot_names">
+ <term><varname>synchronize_slot_names</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>synchronize_slot_names</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies a list of logical replication slots that a streaming
+ replication standby should synchronize from the primary server. This is
+ necessary to be able to retarget those logical replication connections
+ to this standby if it gets promoted. Specify <literal>*</literal> to
+ synchronize all logical replication slots. The default is empty. On
+ primary, the logical walsenders associated with logical replication
+ slots specified in this parameter will wait for the standby servers
+ specified in <xref linkend="guc-standby-slot-names"/> parameter. In
+ other words, primary ensures those logical replication slots will
+ never get ahead of the standby servers. On standby server, the logical
+ replication slots specified are synchronized from the primary. Set this
+ parameter to same value on both primary and standby.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 26d252bd87..f7a7050d2c 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -100,6 +100,7 @@
#include "replication/snapbuild.h" /* just for SnapBuildSnapDecRefcount */
#include "storage/bufmgr.h"
#include "storage/fd.h"
+#include "storage/ipc.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
@@ -107,6 +108,7 @@
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
+#include "utils/varlena.h"
/* entry for a hash table we use to map from xid to our transaction state */
@@ -2498,6 +2500,13 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
else
{
+ /*
+ * Before we send out the last set of changes to logical decoding
+ * output plugin, wait for specified streaming replication standby
+ * servers (if any) to confirm receipt of WAL upto commit_lsn.
+ */
+ WaitForStandbyLSN(commit_lsn);
+
/*
* Call either PREPARE (for two-phase transactions) or COMMIT (for
* regular ones).
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 1dc27264f6..dc1d11a564 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -52,6 +52,8 @@
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/builtins.h"
+#include "utils/guc_hooks.h"
+#include "utils/varlena.h"
/*
* Replication slot on-disk data structure.
@@ -98,9 +100,11 @@ ReplicationSlotCtlData *ReplicationSlotCtl = NULL;
/* My backend's replication slot in the shared memory array */
ReplicationSlot *MyReplicationSlot = NULL;
-/* GUC variable */
+/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+char *synchronize_slot_names;
+char *standby_slot_names;
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropAcquired(void);
@@ -111,6 +115,8 @@ static void RestoreSlotFromDisk(const char *name);
static void CreateSlotOnDisk(ReplicationSlot *slot);
static void SaveSlotToPath(ReplicationSlot *slot, const char *dir, int elevel);
+static bool validate_slot_names(char **newval);
+
/*
* Report shared-memory space needed by ReplicationSlotsShmemInit.
*/
@@ -2085,3 +2091,211 @@ RestoreSlotFromDisk(const char *name)
(errmsg("too many replication slots active before shutdown"),
errhint("Increase max_replication_slots and try again.")));
}
+
+/*
+ * A helper function to simplify check_hook implementation for
+ * synchronize_slot_names and standby_slot_names GUCs.
+ */
+static bool
+validate_slot_names(char **newval)
+{
+ char *rawname;
+ List *elemlist;
+
+ /* Need a modifiable copy of string */
+ rawname = pstrdup(*newval);
+
+ /* Parse string into list of identifiers */
+ if (!SplitIdentifierString(rawname, ',', &elemlist))
+ {
+ /* syntax error in name list */
+ GUC_check_errdetail("List syntax is invalid.");
+ pfree(rawname);
+ list_free(elemlist);
+ return false;
+ }
+
+ pfree(rawname);
+ list_free(elemlist);
+ return true;
+}
+
+/*
+ * GUC check_hook for synchronize_slot_names
+ */
+bool
+check_synchronize_slot_names(char **newval, void **extra, GucSource source)
+{
+ /* Special handling for "*" which means all. */
+ if (strcmp(*newval, "*") == 0)
+ return true;
+
+ if (strcmp(*newval, "") == 0)
+ return true;
+
+ return validate_slot_names(newval);
+}
+
+/*
+ * GUC check_hook for standby_slot_names
+ */
+bool
+check_standby_slot_names(char **newval, void **extra, GucSource source)
+{
+ /* Special handling for "*" which means all. */
+ if (strcmp(*newval, "*") == 0)
+ return true;
+
+ if (strcmp(*newval, "") == 0)
+ return true;
+
+ return validate_slot_names(newval);
+}
+
+/*
+ * Function in which logical walsender (the caller) corresponding to a logical
+ * slot specified in synchronize_slot_names GUC value waits for one or more
+ * physical standbys corresponding to specified physical slots in
+ * standby_slot_names GUC value.
+ */
+void
+WaitForStandbyLSN(XLogRecPtr wait_for_lsn)
+{
+ char *rawname;
+ List *elemlist;
+ ListCell *l;
+ ReplicationSlot *slot;
+
+ Assert(MyReplicationSlot != NULL);
+ Assert(SlotIsLogical(MyReplicationSlot));
+
+ if (strcmp(standby_slot_names, "") == 0)
+ return;
+
+ /*
+ * Check if the slot associated with this logical walsender is asked to
+ * wait for physical standbys.
+ */
+ if (strcmp(synchronize_slot_names, "") == 0)
+ return;
+
+ /* "*" means all logical walsenders should wait for physical standbys. */
+ if (strcmp(synchronize_slot_names, "*") != 0)
+ {
+ bool shouldwait = false;
+
+ rawname = pstrdup(synchronize_slot_names);
+ SplitIdentifierString(rawname, ',', &elemlist);
+
+ foreach (l, elemlist)
+ {
+ char *name = lfirst(l);
+ if (strcmp(name, NameStr(MyReplicationSlot->data.name)) == 0)
+ {
+ shouldwait = true;
+ break;
+ }
+ }
+
+ pfree(rawname);
+ rawname = NULL;
+ list_free(elemlist);
+ elemlist = NIL;
+
+ if (!shouldwait)
+ return;
+ }
+
+ rawname = pstrdup(standby_slot_names);
+ SplitIdentifierString(rawname, ',', &elemlist);
+
+retry:
+
+ foreach (l, elemlist)
+ {
+ char *name = lfirst(l);
+ XLogRecPtr restart_lsn;
+ bool invalidated;
+
+ slot = SearchNamedReplicationSlot(name, true);
+
+ /*
+ * It may happen that the slot specified in standby_slot_names GUC
+ * value is dropped, so let's skip over it.
+ */
+ if (!slot)
+ {
+ ereport(WARNING,
+ errmsg("replication slot \"%s\" specified in parameter \"%s\" does not exist, ignoring",
+ name, "standby_slot_names"));
+ elemlist = foreach_delete_current(elemlist, l);
+ continue;
+ }
+
+ /*
+ * It may happen that the physical slot specified in standby_slot_names
+ * is dropped without removing it from the GUC value, and a logical
+ * slot has been created with the same name meanwhile. Let's skip over
+ * it.
+ *
+ * NB: We might think to modify the GUC value automatically while
+ * dropping a physical replication slot, but that won't be a nice idea
+ * given that the slot can sometimes be dropped in process exit paths
+ * (check ReplicationSlotCleanup call sites), so modifying GUC value
+ * there isn't a great idea.
+ */
+ if (SlotIsLogical(slot))
+ {
+ ereport(WARNING,
+ errmsg("cannot have logical replication slot \"%s\" in parameter \"%s\", ignoring",
+ name, "standby_slot_names"));
+ elemlist = foreach_delete_current(elemlist, l);
+ continue;
+ }
+
+ /* physical slots advance restart_lsn on remote flush */
+ SpinLockAcquire(&slot->mutex);
+ restart_lsn = slot->data.restart_lsn;
+ invalidated = slot->data.invalidated != RS_INVAL_NONE;
+ SpinLockRelease(&slot->mutex);
+
+ /*
+ * Specified physical slot may have been invalidated, so no point in
+ * waiting for it.
+ */
+ if (restart_lsn == InvalidXLogRecPtr || invalidated)
+ {
+ ereport(WARNING,
+ errmsg("physical slot \"%s\" specified in parameter \"%s\" has been invalidated, ignoring",
+ name, "standby_slot_names"));
+ elemlist = foreach_delete_current(elemlist, l);
+ continue;
+ }
+
+ /* If the slot is past the wait_for_lsn, no need to wait anymore */
+ if (restart_lsn >= wait_for_lsn)
+ {
+ elemlist = foreach_delete_current(elemlist, l);
+ continue;
+ }
+ }
+
+ if (list_length(elemlist) == 0)
+ {
+ pfree(rawname);
+ return; /* Exit if done waiting for everyone */
+ }
+
+ /* XXX: Is waiting for 1 second before retrying enough or more or less? */
+
+ /* XXX: Need to have a new wait event type. */
+ (void) WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ 1000L,
+ WAIT_EVENT_WAL_SENDER_WAIT_WAL);
+ ResetLatch(MyLatch);
+
+ CHECK_FOR_INTERRUPTS();
+
+ goto retry;
+}
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index f9dba43b8c..d72b6b95b6 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -4551,6 +4551,36 @@ struct config_string ConfigureNamesString[] =
check_io_direct, assign_io_direct, NULL
},
+ /*
+ * XXX: synchronize_slot_names needs to be specified on both primary and
+ * standby, therefore, we might need a new group REPLICATION.
+ */
+ {
+ {"synchronize_slot_names", PGC_SIGHUP, REPLICATION_STANDBY,
+ gettext_noop("List of replication slot names to synchronize from "
+ "primary to streaming replication standby server."),
+ gettext_noop("Value of \"*\" means all."),
+ GUC_LIST_INPUT | GUC_LIST_QUOTE
+ },
+ &synchronize_slot_names,
+ "",
+ check_synchronize_slot_names, NULL, NULL
+ },
+
+ {
+ {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY,
+ gettext_noop("List of streaming replication standby server slot "
+ "names that logical walsenders waits for."),
+ gettext_noop("Decoded changes are sent out to plugins by logical "
+ "walsenders only after specified replication slots "
+ "confirm receiving WAL."),
+ GUC_LIST_INPUT | GUC_LIST_QUOTE
+ },
+ &standby_slot_names,
+ "",
+ check_standby_slot_names, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index c768af9a73..63daf586f3 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -328,6 +328,8 @@
# method to choose sync standbys, number of sync standbys,
# and comma-separated list of application_name
# from standby(s); '*' = all
+#standby_slot_names = '' # streaming replication standby server slot names that
+ # logical walsenders waits for
# - Standby Servers -
@@ -355,6 +357,8 @@
#wal_retrieve_retry_interval = 5s # time to wait before retrying to
# retrieve WAL after a failed attempt
#recovery_min_apply_delay = 0 # minimum delay for applying changes during recovery
+#synchronize_slot_names = '' # replication slot names to synchronize from
+ # primary to streaming replication standby server
# - Subscribers -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index a8a89dc784..2765f99ccf 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -203,6 +203,8 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
+extern PGDLLIMPORT char *synchronize_slot_names;
+extern PGDLLIMPORT char *standby_slot_names;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -246,4 +248,6 @@ extern void CheckPointReplicationSlots(void);
extern void CheckSlotRequirements(void);
extern void CheckSlotPermissions(void);
+extern void WaitForStandbyLSN(XLogRecPtr wait_for_lsn);
+
#endif /* SLOT_H */
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 2ecb9fc086..259aefb9d7 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -159,5 +159,9 @@ extern void assign_wal_consistency_checking(const char *newval, void *extra);
extern void assign_xlog_sync_method(int new_sync_method, void *extra);
extern bool check_io_direct(char **newval, void **extra, GucSource source);
extern void assign_io_direct(const char *newval, void *extra);
+extern bool check_synchronize_slot_names(char **newval, void **extra,
+ GucSource source);
+extern bool check_standby_slot_names(char **newval, void **extra,
+ GucSource source);
#endif /* GUC_HOOKS_H */
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index e7328e4894..ee590eeac7 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -43,6 +43,7 @@ tests += {
't/035_standby_logical_decoding.pl',
't/036_truncated_dropped.pl',
't/037_invalid_database.pl',
+ 't/050_verify_slot_order.pl',
],
},
}
diff --git a/src/test/recovery/t/050_verify_slot_order.pl b/src/test/recovery/t/050_verify_slot_order.pl
new file mode 100644
index 0000000000..402b704e3f
--- /dev/null
+++ b/src/test/recovery/t/050_verify_slot_order.pl
@@ -0,0 +1,146 @@
+
+# Copyright (c) 2023, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Test primary disallowing specified logical replication slots getting ahead of
+# specified physical replication slots. It uses the following set up:
+#
+# | ----> standby1 (connected via streaming replication)
+# | ----> standby2 (connected via streaming replication)
+# primary ----- |
+# | ----> subscriber1 (connected via logical replication)
+# | ----> subscriber2 (connected via logical replication)
+#
+# Set up is configured in such a way that primary never lets subscriber1 ahead
+# of standby1.
+
+# Create primary
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Configure primary to disallow specified logical replication slot (lsub1_slot)
+# getting ahead of specified physical replication slot (sb1_slot).
+$primary->append_conf(
+ 'postgresql.conf', qq(
+standby_slot_names = 'sb1_slot'
+synchronize_slot_names = 'lsub1_slot'
+));
+$primary->start;
+
+$primary->psql('postgres',
+ q{SELECT pg_create_physical_replication_slot('sb1_slot');});
+$primary->psql('postgres',
+ q{SELECT pg_create_physical_replication_slot('sb2_slot');});
+
+$primary->safe_psql('postgres', "CREATE TABLE tab_int (a int PRIMARY KEY);");
+
+my $backup_name = 'backup';
+$primary->backup($backup_name);
+
+# Create a standby
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup(
+ $primary, $backup_name,
+ has_streaming => 1,
+ has_restoring => 1);
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+primary_slot_name = 'sb1_slot'
+));
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+
+# Create another standby
+my $standby2 = PostgreSQL::Test::Cluster->new('standby2');
+$standby2->init_from_backup(
+ $primary, $backup_name,
+ has_streaming => 1,
+ has_restoring => 1);
+$standby2->append_conf(
+ 'postgresql.conf', qq(
+primary_slot_name = 'sb2_slot'
+));
+$standby2->start;
+$primary->wait_for_replay_catchup($standby2);
+
+# Create publication on primary
+my $publisher = $primary;
+$publisher->safe_psql('postgres', "CREATE PUBLICATION mypub FOR TABLE tab_int;");
+my $publisher_connstr = $publisher->connstr . ' dbname=postgres';
+
+# Create a subscriber node, wait for sync to complete
+my $subscriber1 = PostgreSQL::Test::Cluster->new('subscriber1');
+$subscriber1->init(allows_streaming => 'logical');
+$subscriber1->start;
+$subscriber1->safe_psql('postgres', "CREATE TABLE tab_int (a int PRIMARY KEY);");
+$subscriber1->safe_psql('postgres',
+ "CREATE SUBSCRIPTION mysub1 CONNECTION '$publisher_connstr' "
+ . "PUBLICATION mypub WITH (slot_name = lsub1_slot);");
+$subscriber1->wait_for_subscription_sync;
+
+# Create another subscriber node, wait for sync to complete
+my $subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2');
+$subscriber2->init(allows_streaming => 'logical');
+$subscriber2->start;
+$subscriber2->safe_psql('postgres', "CREATE TABLE tab_int (a int PRIMARY KEY);");
+$subscriber2->safe_psql('postgres',
+ "CREATE SUBSCRIPTION mysub2 CONNECTION '$publisher_connstr' "
+ . "PUBLICATION mypub WITH (slot_name = lsub2_slot);");
+$subscriber2->wait_for_subscription_sync;
+
+# Stop the standby associated with specified physical replication slot so that
+# the logical replication slot won't receive changes until the standby comes
+# up.
+$standby1->stop;
+
+# Create some data on primary
+my $primary_row_count = 10;
+my $primary_insert_time = time();
+$primary->safe_psql('postgres',
+ "INSERT INTO tab_int SELECT generate_series(1, $primary_row_count);");
+
+# Wait for the standby that's up and running gets the data from primary
+$primary->wait_for_replay_catchup($standby2);
+my $result = $standby2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby2 gets data from primary");
+
+# Wait for the subscriber that's up and running and not specified in
+# synchronize_slot_names GUC on primary gets the data from primary without
+# waiting for any standbys.
+$publisher->wait_for_catchup('mysub2');
+$result = $subscriber2->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "subscriber2 gets data from primary");
+
+# The subscriber that's up and running and specified in synchronize_slot_names
+# GUC on primary doesn't get the data from primary and keeps waiting for the
+# standby specified in standby_slot_names.
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = 0 FROM tab_int;");
+is($result, 't', "subscriber1 doesn't get data from primary until standby1 acknowledges changes");
+
+# Start the standby specified in standby_slot_names and wait for it to catch
+# up with the primary.
+$standby1->start;
+$primary->wait_for_replay_catchup($standby1);
+$result = $standby1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "standby1 gets data from primary");
+
+# Now that the standby specified in standby_slot_names is up and running,
+# primary must send the decoded changes to subscriber specified in
+# synchronize_slot_names. While the standby was down, this subscriber didn't
+# receive any data from primary i.e. the primary didn't allow it to go ahead
+# of standby.
+$publisher->wait_for_catchup('mysub1');
+$result = $subscriber1->safe_psql('postgres',
+ "SELECT count(*) = $primary_row_count FROM tab_int;");
+is($result, 't', "subscriber1 gets data from primary after standby1 acknowledges changes");
+
+done_testing();
--
2.34.1
[application/x-patch] v9-0002-Add-logical-slot-sync-capability-to-physical-stan.patch (53.4K, ../../CALj2ACV+VX9McnogGNyFCjZW+qnPvdmjnBjttotygs8+7D5JuA@mail.gmail.com/3-v9-0002-Add-logical-slot-sync-capability-to-physical-stan.patch)
download | inline diff:
From 30d4af524c1b2128519d3b0af761874682961be5 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Sat, 22 Jul 2023 10:28:21 +0000
Subject: [PATCH v9] Add logical slot sync capability to physical standby
---
src/backend/commands/subscriptioncmds.c | 4 +-
src/backend/postmaster/bgworker.c | 3 +
.../libpqwalreceiver/libpqwalreceiver.c | 95 +++++
src/backend/replication/logical/Makefile | 1 +
src/backend/replication/logical/launcher.c | 263 +++++++++-----
src/backend/replication/logical/meson.build | 1 +
src/backend/replication/logical/slotsync.c | 332 ++++++++++++++++++
src/backend/replication/logical/tablesync.c | 13 +-
src/backend/replication/logical/worker.c | 3 +-
src/backend/replication/repl_gram.y | 32 +-
src/backend/replication/repl_scanner.l | 2 +
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 195 ++++++++++
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 3 +
src/include/commands/subscriptioncmds.h | 3 +
src/include/nodes/replnodes.h | 9 +
src/include/replication/logicallauncher.h | 2 +
src/include/replication/logicalworker.h | 1 +
src/include/replication/slot.h | 5 +-
src/include/replication/walreceiver.h | 20 ++
src/include/replication/worker_internal.h | 8 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/051_slot_sync.pl | 132 +++++++
24 files changed, 1037 insertions(+), 94 deletions(-)
create mode 100644 src/backend/replication/logical/slotsync.c
create mode 100644 src/test/recovery/t/051_slot_sync.pl
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index d4e798baeb..42e9b1056c 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -993,7 +993,7 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data,
RemoveSubscriptionRel(sub->oid, relid);
- logicalrep_worker_stop(sub->oid, relid);
+ logicalrep_worker_stop(MyDatabaseId, sub->oid, relid);
/*
* For READY state, we would have already dropped the
@@ -1591,7 +1591,7 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel)
{
LogicalRepWorker *w = (LogicalRepWorker *) lfirst(lc);
- logicalrep_worker_stop(w->subid, w->relid);
+ logicalrep_worker_stop(w->dbid, w->subid, w->relid);
}
list_free(subworkers);
diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c
index 5b4bd71694..f2f4475c3b 100644
--- a/src/backend/postmaster/bgworker.c
+++ b/src/backend/postmaster/bgworker.c
@@ -129,6 +129,9 @@ static const struct
{
"ApplyWorkerMain", ApplyWorkerMain
},
+ {
+ "ReplSlotSyncMain", ReplSlotSyncMain
+ },
{
"ParallelApplyWorkerMain", ParallelApplyWorkerMain
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 60d5c1fc40..0e13cc2417 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -34,6 +34,7 @@
#include "utils/memutils.h"
#include "utils/pg_lsn.h"
#include "utils/tuplestore.h"
+#include "utils/varlena.h"
PG_MODULE_MAGIC;
@@ -58,6 +59,7 @@ static void libpqrcv_get_senderinfo(WalReceiverConn *conn,
char **sender_host, int *sender_port);
static char *libpqrcv_identify_system(WalReceiverConn *conn,
TimeLineID *primary_tli);
+static List *libpqrcv_list_slots(WalReceiverConn *conn, const char *slot_names);
static int libpqrcv_server_version(WalReceiverConn *conn);
static void libpqrcv_readtimelinehistoryfile(WalReceiverConn *conn,
TimeLineID tli, char **filename,
@@ -96,6 +98,7 @@ static WalReceiverFunctionsType PQWalReceiverFunctions = {
.walrcv_receive = libpqrcv_receive,
.walrcv_send = libpqrcv_send,
.walrcv_create_slot = libpqrcv_create_slot,
+ .walrcv_list_slots = libpqrcv_list_slots,
.walrcv_get_backend_pid = libpqrcv_get_backend_pid,
.walrcv_exec = libpqrcv_exec,
.walrcv_disconnect = libpqrcv_disconnect
@@ -409,6 +412,98 @@ libpqrcv_server_version(WalReceiverConn *conn)
return PQserverVersion(conn->streamConn);
}
+/*
+ * Get list of slots from primary.
+ */
+static List *
+libpqrcv_list_slots(WalReceiverConn *conn, const char *slot_names)
+{
+ PGresult *res;
+ List *slotlist = NIL;
+ int ntuples;
+ StringInfoData s;
+ WalRecvReplicationSlotData *slot_data;
+
+ initStringInfo(&s);
+ appendStringInfoString(&s, "LIST_SLOTS");
+
+ if (strcmp(slot_names, "") != 0 && strcmp(slot_names, "*") != 0)
+ {
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+
+ appendStringInfoChar(&s, ' ');
+ rawname = pstrdup(slot_names);
+ SplitIdentifierString(rawname, ',', &namelist);
+ foreach (lc, namelist)
+ {
+ if (lc != list_head(namelist))
+ appendStringInfoChar(&s, ',');
+ appendStringInfo(&s, "%s",
+ quote_identifier(lfirst(lc)));
+ }
+ }
+
+ res = libpqrcv_PQexec(conn->streamConn, s.data);
+ pfree(s.data);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ PQclear(res);
+ ereport(ERROR,
+ (errmsg("could not receive list of slots the primary server: %s",
+ pchomp(PQerrorMessage(conn->streamConn)))));
+ }
+ if (PQnfields(res) < 10)
+ {
+ int nfields = PQnfields(res);
+
+ PQclear(res);
+ ereport(ERROR,
+ (errmsg("invalid response from primary server"),
+ errdetail("Could not get list of slots: got %d fields, expected %d or more fields.",
+ nfields, 10)));
+ }
+
+ ntuples = PQntuples(res);
+ for (int i = 0; i < ntuples; i++)
+ {
+ char *slot_type;
+
+ slot_data = palloc0(sizeof(WalRecvReplicationSlotData));
+ namestrcpy(&slot_data->persistent_data.name, PQgetvalue(res, i, 0));
+ if (!PQgetisnull(res, i, 1))
+ namestrcpy(&slot_data->persistent_data.plugin, PQgetvalue(res, i, 1));
+ slot_type = PQgetvalue(res, i, 2);
+ if (!PQgetisnull(res, i, 3))
+ slot_data->persistent_data.database = atooid(PQgetvalue(res, i, 3));
+ if (strcmp(slot_type, "physical") == 0)
+ {
+ if (OidIsValid(slot_data->persistent_data.database))
+ elog(ERROR, "unexpected physical replication slot with database set");
+ }
+ if (pg_strtoint32(PQgetvalue(res, i, 5)) == 1)
+ slot_data->persistent_data.persistency = RS_TEMPORARY;
+ else
+ slot_data->persistent_data.persistency = RS_PERSISTENT;
+ if (!PQgetisnull(res, i, 6))
+ slot_data->persistent_data.xmin = atooid(PQgetvalue(res, i, 6));
+ if (!PQgetisnull(res, i, 7))
+ slot_data->persistent_data.catalog_xmin = atooid(PQgetvalue(res, i, 7));
+ if (!PQgetisnull(res, i, 8))
+ slot_data->persistent_data.restart_lsn = strtou64(PQgetvalue(res, i, 8), NULL, 10);
+ if (!PQgetisnull(res, i, 9))
+ slot_data->persistent_data.confirmed_flush = strtou64(PQgetvalue(res, i, 9), NULL, 10);
+
+ slot_data->last_sync_time = 0;
+ slotlist = lappend(slotlist, slot_data);
+ }
+
+ PQclear(res);
+
+ return slotlist;
+}
+
/*
* Start streaming WAL data from given streaming options.
*
diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile
index 2dc25e37bb..ba03eeff1c 100644
--- a/src/backend/replication/logical/Makefile
+++ b/src/backend/replication/logical/Makefile
@@ -25,6 +25,7 @@ OBJS = \
proto.o \
relation.o \
reorderbuffer.o \
+ slotsync.o \
snapbuild.o \
tablesync.o \
worker.o
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 542af7d863..640f7647cc 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -22,6 +22,7 @@
#include "access/htup_details.h"
#include "access/tableam.h"
#include "access/xact.h"
+#include "catalog/pg_authid.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
#include "funcapi.h"
@@ -246,7 +247,7 @@ WaitForReplicationWorkerAttach(LogicalRepWorker *worker,
* We are only interested in the leader apply worker or table sync worker.
*/
LogicalRepWorker *
-logicalrep_worker_find(Oid subid, Oid relid, bool only_running)
+logicalrep_worker_find(Oid dbid, Oid subid, Oid relid, bool only_running)
{
int i;
LogicalRepWorker *res = NULL;
@@ -262,8 +263,8 @@ logicalrep_worker_find(Oid subid, Oid relid, bool only_running)
if (isParallelApplyWorker(w))
continue;
- if (w->in_use && w->subid == subid && w->relid == relid &&
- (!only_running || w->proc))
+ if (w->in_use && w->dbid == dbid && w->subid == subid &&
+ w->relid == relid && (!only_running || w->proc))
{
res = w;
break;
@@ -320,9 +321,13 @@ logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname, Oid userid,
/* Sanity check - tablesync worker cannot be a subworker */
Assert(!(is_parallel_apply_worker && OidIsValid(relid)));
- ereport(DEBUG1,
- (errmsg_internal("starting logical replication worker for subscription \"%s\"",
- subname)));
+ if (OidIsValid(subid))
+ ereport(DEBUG1,
+ (errmsg_internal("starting logical replication worker for subscription \"%s\"",
+ subname)));
+ else
+ ereport(DEBUG1,
+ (errmsg_internal("starting replication slot synchronization worker")));
/* Report this after the initial starting message for consistency. */
if (max_replication_slots == 0)
@@ -359,7 +364,9 @@ retry:
* reason we do this is because if some worker failed to start up and its
* parent has crashed while waiting, the in_use state was never cleared.
*/
- if (worker == NULL || nsyncworkers >= max_sync_workers_per_subscription)
+ if (worker == NULL ||
+ (OidIsValid(relid) &&
+ nsyncworkers >= max_sync_workers_per_subscription))
{
bool did_cleanup = false;
@@ -455,15 +462,20 @@ retry:
memset(&bgw, 0, sizeof(bgw));
bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
BGWORKER_BACKEND_DATABASE_CONNECTION;
- bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ bgw.bgw_start_time = BgWorkerStart_ConsistentState;
snprintf(bgw.bgw_library_name, MAXPGPATH, "postgres");
- if (is_parallel_apply_worker)
+ if (!OidIsValid(subid))
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ReplSlotSyncMain");
+ else if (is_parallel_apply_worker)
snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ParallelApplyWorkerMain");
else
snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain");
- if (OidIsValid(relid))
+ if (!OidIsValid(subid))
+ snprintf(bgw.bgw_name, BGW_MAXLEN,
+ "replication slot synchronization worker");
+ else if (OidIsValid(relid))
snprintf(bgw.bgw_name, BGW_MAXLEN,
"logical replication worker for subscription %u sync %u", subid, relid);
else if (is_parallel_apply_worker)
@@ -591,13 +603,13 @@ logicalrep_worker_stop_internal(LogicalRepWorker *worker, int signo)
* Stop the logical replication worker for subid/relid, if any.
*/
void
-logicalrep_worker_stop(Oid subid, Oid relid)
+logicalrep_worker_stop(Oid dbid, Oid subid, Oid relid)
{
LogicalRepWorker *worker;
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- worker = logicalrep_worker_find(subid, relid, false);
+ worker = logicalrep_worker_find(dbid, subid, relid, false);
if (worker)
{
@@ -658,13 +670,13 @@ logicalrep_pa_worker_stop(ParallelApplyWorkerInfo *winfo)
* Wake up (using latch) any logical replication worker for specified sub/rel.
*/
void
-logicalrep_worker_wakeup(Oid subid, Oid relid)
+logicalrep_worker_wakeup(Oid dbid, Oid subid, Oid relid)
{
LogicalRepWorker *worker;
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- worker = logicalrep_worker_find(subid, relid, true);
+ worker = logicalrep_worker_find(dbid, subid, relid, true);
if (worker)
logicalrep_worker_wakeup_ptr(worker);
@@ -909,7 +921,7 @@ ApplyLauncherRegister(void)
memset(&bgw, 0, sizeof(bgw));
bgw.bgw_flags = BGWORKER_SHMEM_ACCESS |
BGWORKER_BACKEND_DATABASE_CONNECTION;
- bgw.bgw_start_time = BgWorkerStart_RecoveryFinished;
+ bgw.bgw_start_time = BgWorkerStart_ConsistentState;
snprintf(bgw.bgw_library_name, MAXPGPATH, "postgres");
snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyLauncherMain");
snprintf(bgw.bgw_name, BGW_MAXLEN,
@@ -1092,6 +1104,157 @@ ApplyLauncherWakeup(void)
kill(LogicalRepCtx->launcher_pid, SIGUSR1);
}
+static void
+ApplyLauncherStartSlotSync(long *wait_time)
+{
+ WalReceiverConn *wrconn;
+ char *err;
+ List *slots;
+ ListCell *lc;
+ MemoryContext tmpctx;
+ MemoryContext oldctx;
+
+ if (strcmp(synchronize_slot_names, "") == 0)
+ return;
+
+ wrconn = walrcv_connect(PrimaryConnInfo, false, false,
+ "Logical Replication Launcher", &err);
+ if (!wrconn)
+ ereport(ERROR,
+ (errmsg("could not connect to the primary server: %s", err)));
+
+ /* Use temporary context for the slot list and worker info. */
+ tmpctx = AllocSetContextCreate(TopMemoryContext,
+ "Logical Replication Launcher slot sync ctx",
+ ALLOCSET_DEFAULT_SIZES);
+ oldctx = MemoryContextSwitchTo(tmpctx);
+
+ slots = walrcv_list_slots(wrconn, synchronize_slot_names);
+
+ foreach(lc, slots)
+ {
+ WalRecvReplicationSlotData *slot_data = lfirst(lc);
+ LogicalRepWorker *w;
+ TimestampTz last_sync;
+ TimestampTz now;
+ long elapsed;
+
+ if (!OidIsValid(slot_data->persistent_data.database))
+ continue;
+
+ LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
+ w = logicalrep_worker_find(slot_data->persistent_data.database, InvalidOid,
+ InvalidOid, false);
+ LWLockRelease(LogicalRepWorkerLock);
+
+ if (w != NULL)
+ continue; /* worker is running already */
+
+ /*
+ * If the worker is eligible to start now, launch it. Otherwise,
+ * adjust wait_time so that we'll wake up as soon as it can be
+ * started.
+ *
+ * Each apply worker can only be restarted once per
+ * wal_retrieve_retry_interval, so that errors do not cause us to
+ * repeatedly restart the worker as fast as possible.
+ */
+ last_sync = slot_data->last_sync_time;
+ now = GetCurrentTimestamp();
+ if (last_sync == 0 ||
+ (elapsed = TimestampDifferenceMilliseconds(last_sync, now)) >= wal_retrieve_retry_interval)
+ {
+ slot_data->last_sync_time = now;
+ logicalrep_worker_launch(slot_data->persistent_data.database,
+ InvalidOid, NULL,
+ BOOTSTRAP_SUPERUSERID, InvalidOid,
+ DSM_HANDLE_INVALID);
+ }
+ else
+ {
+ *wait_time = Min(*wait_time,
+ wal_retrieve_retry_interval - elapsed);
+ }
+ }
+
+ /* Switch back to original memory context. */
+ MemoryContextSwitchTo(oldctx);
+ /* Clean the temporary memory. */
+ MemoryContextDelete(tmpctx);
+
+ walrcv_disconnect(wrconn);
+}
+
+static void
+ApplyLauncherStartSubs(long *wait_time)
+{
+ List *sublist;
+ ListCell *lc;
+ MemoryContext subctx;
+ MemoryContext oldctx;
+
+ /* Use temporary context to avoid leaking memory across cycles. */
+ subctx = AllocSetContextCreate(TopMemoryContext,
+ "Logical Replication Launcher sublist",
+ ALLOCSET_DEFAULT_SIZES);
+ oldctx = MemoryContextSwitchTo(subctx);
+
+ /* Start the missing workers for enabled subscriptions. */
+ sublist = get_subscription_list();
+ foreach(lc, sublist)
+ {
+ Subscription *sub = (Subscription *) lfirst(lc);
+ LogicalRepWorker *w;
+ TimestampTz last_start;
+ TimestampTz now;
+ long elapsed;
+
+ if (!sub->enabled)
+ continue;
+
+ LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
+ w = logicalrep_worker_find(sub->dbid, sub->oid, InvalidOid, false);
+ LWLockRelease(LogicalRepWorkerLock);
+
+ if (w != NULL)
+ continue; /* worker is running already */
+
+ /*
+ * If the worker is eligible to start now, launch it. Otherwise,
+ * adjust wait_time so that we'll wake up as soon as it can be
+ * started.
+ *
+ * Each subscription's apply worker can only be restarted once per
+ * wal_retrieve_retry_interval, so that errors do not cause us to
+ * repeatedly restart the worker as fast as possible. In cases
+ * where a restart is expected (e.g., subscription parameter
+ * changes), another process should remove the last-start entry
+ * for the subscription so that the worker can be restarted
+ * without waiting for wal_retrieve_retry_interval to elapse.
+ */
+ last_start = ApplyLauncherGetWorkerStartTime(sub->oid);
+ now = GetCurrentTimestamp();
+ if (last_start == 0 ||
+ (elapsed = TimestampDifferenceMilliseconds(last_start, now)) >= wal_retrieve_retry_interval)
+ {
+ ApplyLauncherSetWorkerStartTime(sub->oid, now);
+ logicalrep_worker_launch(sub->dbid, sub->oid, sub->name,
+ sub->owner, InvalidOid,
+ DSM_HANDLE_INVALID);
+ }
+ else
+ {
+ *wait_time = Min(*wait_time,
+ wal_retrieve_retry_interval - elapsed);
+ }
+ }
+
+ /* Switch back to original memory context. */
+ MemoryContextSwitchTo(oldctx);
+ /* Clean the temporary memory. */
+ MemoryContextDelete(subctx);
+}
+
/*
* Main loop for the apply launcher process.
*/
@@ -1117,78 +1280,20 @@ ApplyLauncherMain(Datum main_arg)
*/
BackgroundWorkerInitializeConnection(NULL, NULL, 0);
+ load_file("libpqwalreceiver", false);
+
/* Enter main loop */
for (;;)
{
int rc;
- List *sublist;
- ListCell *lc;
- MemoryContext subctx;
- MemoryContext oldctx;
long wait_time = DEFAULT_NAPTIME_PER_CYCLE;
CHECK_FOR_INTERRUPTS();
- /* Use temporary context to avoid leaking memory across cycles. */
- subctx = AllocSetContextCreate(TopMemoryContext,
- "Logical Replication Launcher sublist",
- ALLOCSET_DEFAULT_SIZES);
- oldctx = MemoryContextSwitchTo(subctx);
-
- /* Start any missing workers for enabled subscriptions. */
- sublist = get_subscription_list();
- foreach(lc, sublist)
- {
- Subscription *sub = (Subscription *) lfirst(lc);
- LogicalRepWorker *w;
- TimestampTz last_start;
- TimestampTz now;
- long elapsed;
-
- if (!sub->enabled)
- continue;
-
- LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- w = logicalrep_worker_find(sub->oid, InvalidOid, false);
- LWLockRelease(LogicalRepWorkerLock);
-
- if (w != NULL)
- continue; /* worker is running already */
-
- /*
- * If the worker is eligible to start now, launch it. Otherwise,
- * adjust wait_time so that we'll wake up as soon as it can be
- * started.
- *
- * Each subscription's apply worker can only be restarted once per
- * wal_retrieve_retry_interval, so that errors do not cause us to
- * repeatedly restart the worker as fast as possible. In cases
- * where a restart is expected (e.g., subscription parameter
- * changes), another process should remove the last-start entry
- * for the subscription so that the worker can be restarted
- * without waiting for wal_retrieve_retry_interval to elapse.
- */
- last_start = ApplyLauncherGetWorkerStartTime(sub->oid);
- now = GetCurrentTimestamp();
- if (last_start == 0 ||
- (elapsed = TimestampDifferenceMilliseconds(last_start, now)) >= wal_retrieve_retry_interval)
- {
- ApplyLauncherSetWorkerStartTime(sub->oid, now);
- logicalrep_worker_launch(sub->dbid, sub->oid, sub->name,
- sub->owner, InvalidOid,
- DSM_HANDLE_INVALID);
- }
- else
- {
- wait_time = Min(wait_time,
- wal_retrieve_retry_interval - elapsed);
- }
- }
-
- /* Switch back to original memory context. */
- MemoryContextSwitchTo(oldctx);
- /* Clean the temporary memory. */
- MemoryContextDelete(subctx);
+ if (!RecoveryInProgress())
+ ApplyLauncherStartSubs(&wait_time);
+ else
+ ApplyLauncherStartSlotSync(&wait_time);
/* Wait for more work. */
rc = WaitLatch(MyLatch,
diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build
index d48cd4c590..9e52ec421f 100644
--- a/src/backend/replication/logical/meson.build
+++ b/src/backend/replication/logical/meson.build
@@ -11,6 +11,7 @@ backend_sources += files(
'proto.c',
'relation.c',
'reorderbuffer.c',
+ 'slotsync.c',
'snapbuild.c',
'tablesync.c',
'worker.c',
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
new file mode 100644
index 0000000000..77457001e7
--- /dev/null
+++ b/src/backend/replication/logical/slotsync.c
@@ -0,0 +1,332 @@
+/*-------------------------------------------------------------------------
+ * slotsync.c
+ * PostgreSQL worker for synchronizing slots to a standby from primary
+ *
+ * Copyright (c) 2016-2018, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/replication/logical/slotsync.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "commands/dbcommands.h"
+#include "pgstat.h"
+#include "postmaster/bgworker.h"
+#include "replication/logicallauncher.h"
+#include "replication/logicalworker.h"
+#include "replication/walreceiver.h"
+#include "replication/worker_internal.h"
+#include "storage/ipc.h"
+#include "storage/procarray.h"
+#include "utils/builtins.h"
+#include "utils/guc_hooks.h"
+#include "utils/pg_lsn.h"
+#include "utils/varlena.h"
+
+/*
+ * Wait for remote slot to pass localy reserved position.
+ */
+static void
+wait_for_primary_slot_catchup(WalReceiverConn *wrconn, char *slot_name,
+ XLogRecPtr min_lsn)
+{
+ WalRcvExecResult *res;
+ TupleTableSlot *slot;
+ Oid slotRow[1] = {LSNOID};
+ StringInfoData cmd;
+ bool isnull;
+ XLogRecPtr restart_lsn;
+
+ for (;;)
+ {
+ int rc;
+
+ CHECK_FOR_INTERRUPTS();
+
+ initStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT restart_lsn"
+ " FROM pg_catalog.pg_replication_slots"
+ " WHERE slot_name = %s",
+ quote_literal_cstr(slot_name));
+ res = walrcv_exec(wrconn, cmd.data, 1, slotRow);
+
+ if (res->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errmsg("could not fetch slot info for slot \"%s\" from primary: %s",
+ slot_name, res->err)));
+
+ slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+ if (!tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+ ereport(ERROR,
+ (errmsg("slot \"%s\" disapeared from provider",
+ slot_name)));
+
+ restart_lsn = DatumGetLSN(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ ExecClearTuple(slot);
+ walrcv_clear_result(res);
+
+ if (restart_lsn >= min_lsn)
+ break;
+
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ wal_retrieve_retry_interval,
+ WAIT_EVENT_REPL_SLOT_SYNC_MAIN);
+
+ ResetLatch(MyLatch);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+ }
+}
+
+/*
+ * Synchronize single slot to given position.
+ *
+ * This optionally creates new slot if there is no existing one.
+ */
+static void
+synchronize_one_slot(WalReceiverConn *wrconn, char *slot_name, char *database,
+ char *plugin_name, XLogRecPtr target_lsn)
+{
+ bool found = false;
+ XLogRecPtr endlsn;
+
+ /* Search for the named slot and mark it active if we find it. */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (int i = 0; i < max_replication_slots; i++)
+ {
+ ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i];
+
+ if (!s->in_use)
+ continue;
+
+ if (strcmp(NameStr(s->data.name), slot_name) == 0)
+ {
+ found = true;
+ break;
+ }
+ }
+ LWLockRelease(ReplicationSlotControlLock);
+
+ StartTransactionCommand();
+
+ /* Already existing slot, acquire */
+ if (found)
+ {
+ ReplicationSlotAcquire(slot_name, true);
+
+ if (target_lsn < MyReplicationSlot->data.confirmed_flush)
+ {
+ elog(DEBUG1,
+ "not synchronizing slot %s; synchronization would move it backward",
+ slot_name);
+
+ ReplicationSlotRelease();
+ CommitTransactionCommand();
+ return;
+ }
+ }
+ /* Otherwise create the slot first. */
+ else
+ {
+ TransactionId xmin_horizon = InvalidTransactionId;
+ ReplicationSlot *slot;
+
+ ReplicationSlotCreate(slot_name, true, RS_EPHEMERAL, false);
+ slot = MyReplicationSlot;
+
+ SpinLockAcquire(&slot->mutex);
+ slot->data.database = get_database_oid(database, false);
+ namestrcpy(&slot->data.plugin, plugin_name);
+ SpinLockRelease(&slot->mutex);
+
+ ReplicationSlotReserveWal();
+
+ LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
+ xmin_horizon = GetOldestSafeDecodingTransactionId(true);
+ slot->effective_catalog_xmin = xmin_horizon;
+ slot->data.catalog_xmin = xmin_horizon;
+ ReplicationSlotsComputeRequiredXmin(true);
+ LWLockRelease(ProcArrayLock);
+
+ if (target_lsn < MyReplicationSlot->data.restart_lsn)
+ {
+ ereport(LOG,
+ errmsg("waiting for remote slot \"%s\" LSN (%X/%X) to pass local slot LSN (%X/%X)",
+ slot_name,
+ LSN_FORMAT_ARGS(target_lsn), LSN_FORMAT_ARGS(MyReplicationSlot->data.restart_lsn)));
+
+ wait_for_primary_slot_catchup(wrconn, slot_name,
+ MyReplicationSlot->data.restart_lsn);
+ }
+
+ ReplicationSlotPersist();
+ }
+
+ endlsn = pg_logical_replication_slot_advance(target_lsn);
+
+ elog(DEBUG3, "synchronized slot %s to lsn (%X/%X)",
+ slot_name, LSN_FORMAT_ARGS(endlsn));
+
+ ReplicationSlotRelease();
+ CommitTransactionCommand();
+}
+
+static void
+synchronize_slots(void)
+{
+ WalRcvExecResult *res;
+ WalReceiverConn *wrconn = NULL;
+ TupleTableSlot *slot;
+ Oid slotRow[3] = {TEXTOID, TEXTOID, LSNOID};
+ StringInfoData s;
+ char *database;
+ char *err;
+ MemoryContext oldctx = CurrentMemoryContext;
+
+ if (!WalRcv)
+ return;
+
+ /* syscache access needs a transaction env. */
+ StartTransactionCommand();
+ /* make dbname live outside TX context */
+ MemoryContextSwitchTo(oldctx);
+
+ database = get_database_name(MyDatabaseId);
+ initStringInfo(&s);
+ appendStringInfo(&s, "%s dbname=%s", PrimaryConnInfo, database);
+ wrconn = walrcv_connect(s.data, true, false, "slot_sync", &err);
+
+ if (wrconn == NULL)
+ ereport(ERROR,
+ (errmsg("could not connect to the primary server: %s", err)));
+
+ resetStringInfo(&s);
+ appendStringInfo(&s,
+ "SELECT slot_name, plugin, confirmed_flush_lsn"
+ " FROM pg_catalog.pg_replication_slots"
+ " WHERE database = %s",
+ quote_literal_cstr(database));
+ if (strcmp(synchronize_slot_names, "") != 0 && strcmp(synchronize_slot_names, "*") != 0)
+ {
+ char *rawname;
+ List *namelist;
+ ListCell *lc;
+
+ rawname = pstrdup(synchronize_slot_names);
+ SplitIdentifierString(rawname, ',', &namelist);
+
+ appendStringInfoString(&s, " AND slot_name IN (");
+ foreach (lc, namelist)
+ {
+ if (lc != list_head(namelist))
+ appendStringInfoChar(&s, ',');
+ appendStringInfo(&s, "%s",
+ quote_literal_cstr(lfirst(lc)));
+ }
+ appendStringInfoChar(&s, ')');
+ }
+
+ res = walrcv_exec(wrconn, s.data, 3, slotRow);
+ pfree(s.data);
+
+ if (res->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errmsg("could not fetch slot info from primary: %s",
+ res->err)));
+
+ CommitTransactionCommand();
+ /* CommitTransactionCommand switches to TopMemoryContext */
+ MemoryContextSwitchTo(oldctx);
+
+ slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+ {
+ char *slot_name;
+ char *plugin_name;
+ XLogRecPtr confirmed_flush_lsn;
+ bool isnull;
+
+ slot_name = TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ plugin_name = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
+ Assert(!isnull);
+
+ confirmed_flush_lsn = DatumGetLSN(slot_getattr(slot, 3, &isnull));
+ Assert(!isnull);
+
+ synchronize_one_slot(wrconn, slot_name, database, plugin_name,
+ confirmed_flush_lsn);
+
+ ExecClearTuple(slot);
+ }
+
+ walrcv_clear_result(res);
+ pfree(database);
+
+ walrcv_disconnect(wrconn);
+}
+
+/*
+ * The main loop of our worker process.
+ */
+void
+ReplSlotSyncMain(Datum main_arg)
+{
+ int worker_slot = DatumGetInt32(main_arg);
+
+ /* Attach to slot */
+ logicalrep_worker_attach(worker_slot);
+
+ /* Establish signal handlers. */
+ BackgroundWorkerUnblockSignals();
+
+ /* Load the libpq-specific functions */
+ load_file("libpqwalreceiver", false);
+
+ /* Connect to our database. */
+ BackgroundWorkerInitializeConnectionByOid(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->userid,
+ 0);
+
+ StartTransactionCommand();
+ ereport(LOG,
+ (errmsg("replication slot synchronization worker for database \"%s\" has started",
+ get_database_name(MyLogicalRepWorker->dbid))));
+ CommitTransactionCommand();
+
+ /* Main wait loop. */
+ for (;;)
+ {
+ int rc;
+
+ CHECK_FOR_INTERRUPTS();
+
+ if (!RecoveryInProgress())
+ return;
+
+ if (strcmp(synchronize_slot_names, "") == 0)
+ return;
+
+ synchronize_slots();
+
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ wal_retrieve_retry_interval,
+ WAIT_EVENT_REPL_SLOT_SYNC_MAIN);
+
+ ResetLatch(MyLatch);
+
+ /* emergency bailout if postmaster has died */
+ if (rc & WL_POSTMASTER_DEATH)
+ proc_exit(1);
+ }
+}
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 6d461654ab..5a98d2b699 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -100,6 +100,7 @@
#include "catalog/pg_subscription_rel.h"
#include "catalog/pg_type.h"
#include "commands/copy.h"
+#include "commands/subscriptioncmds.h"
#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "parser/parse_relation.h"
@@ -156,7 +157,8 @@ finish_sync_worker(void)
CommitTransactionCommand();
/* Find the leader apply worker and signal it. */
- logicalrep_worker_wakeup(MyLogicalRepWorker->subid, InvalidOid);
+ logicalrep_worker_wakeup(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid, InvalidOid);
/* Stop gracefully */
proc_exit(0);
@@ -196,7 +198,8 @@ wait_for_relation_state_change(Oid relid, char expected_state)
/* Check if the sync worker is still running and bail if not. */
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- worker = logicalrep_worker_find(MyLogicalRepWorker->subid, relid,
+ worker = logicalrep_worker_find(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid, relid,
false);
LWLockRelease(LogicalRepWorkerLock);
if (!worker)
@@ -243,7 +246,8 @@ wait_for_worker_state_change(char expected_state)
* waiting.
*/
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- worker = logicalrep_worker_find(MyLogicalRepWorker->subid,
+ worker = logicalrep_worker_find(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid,
InvalidOid, false);
if (worker && worker->proc)
logicalrep_worker_wakeup_ptr(worker);
@@ -509,7 +513,8 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn)
*/
LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
- syncworker = logicalrep_worker_find(MyLogicalRepWorker->subid,
+ syncworker = logicalrep_worker_find(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid,
rstate->relid, false);
if (syncworker)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index dd353fd1cb..72f39ace7b 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1589,7 +1589,8 @@ apply_handle_stream_start(StringInfo s)
* Signal the leader apply worker, as it may be waiting for
* us.
*/
- logicalrep_worker_wakeup(MyLogicalRepWorker->subid, InvalidOid);
+ logicalrep_worker_wakeup(MyLogicalRepWorker->dbid,
+ MyLogicalRepWorker->subid, InvalidOid);
}
parallel_stream_nchanges = 0;
diff --git a/src/backend/replication/repl_gram.y b/src/backend/replication/repl_gram.y
index 0c874e33cf..12a4b74368 100644
--- a/src/backend/replication/repl_gram.y
+++ b/src/backend/replication/repl_gram.y
@@ -76,11 +76,12 @@ Node *replication_parse_result;
%token K_EXPORT_SNAPSHOT
%token K_NOEXPORT_SNAPSHOT
%token K_USE_SNAPSHOT
+%token K_LIST_SLOTS
%type <node> command
%type <node> base_backup start_replication start_logical_replication
create_replication_slot drop_replication_slot identify_system
- read_replication_slot timeline_history show
+ read_replication_slot timeline_history show list_slots
%type <list> generic_option_list
%type <defelt> generic_option
%type <uintval> opt_timeline
@@ -91,6 +92,7 @@ Node *replication_parse_result;
%type <boolval> opt_temporary
%type <list> create_slot_options create_slot_legacy_opt_list
%type <defelt> create_slot_legacy_opt
+%type <list> slot_name_list slot_name_list_opt
%%
@@ -114,6 +116,7 @@ command:
| read_replication_slot
| timeline_history
| show
+ | list_slots
;
/*
@@ -126,6 +129,33 @@ identify_system:
}
;
+slot_name_list:
+ IDENT
+ {
+ $$ = list_make1($1);
+ }
+ | slot_name_list ',' IDENT
+ {
+ $$ = lappend($1, $3);
+ }
+
+slot_name_list_opt:
+ slot_name_list { $$ = $1; }
+ | /* EMPTY */ { $$ = NIL; }
+ ;
+
+/*
+ * LIST_SLOTS
+ */
+list_slots:
+ K_LIST_SLOTS slot_name_list_opt
+ {
+ ListSlotsCmd *cmd = makeNode(ListSlotsCmd);
+ cmd->slot_names = $2;
+ $$ = (Node *) cmd;
+ }
+ ;
+
/*
* READ_REPLICATION_SLOT %s
*/
diff --git a/src/backend/replication/repl_scanner.l b/src/backend/replication/repl_scanner.l
index 1cc7fb858c..11064feb86 100644
--- a/src/backend/replication/repl_scanner.l
+++ b/src/backend/replication/repl_scanner.l
@@ -128,6 +128,7 @@ DROP_REPLICATION_SLOT { return K_DROP_REPLICATION_SLOT; }
TIMELINE_HISTORY { return K_TIMELINE_HISTORY; }
PHYSICAL { return K_PHYSICAL; }
RESERVE_WAL { return K_RESERVE_WAL; }
+LIST_SLOTS { return K_LIST_SLOTS; }
LOGICAL { return K_LOGICAL; }
SLOT { return K_SLOT; }
TEMPORARY { return K_TEMPORARY; }
@@ -304,6 +305,7 @@ replication_scanner_is_replication_command(void)
case K_READ_REPLICATION_SLOT:
case K_TIMELINE_HISTORY:
case K_SHOW:
+ case K_LIST_SLOTS:
/* Yes; push back the first token so we can parse later. */
repl_pushed_back_token = first_token;
return true;
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 6035cf4816..83ada6db6a 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -467,7 +467,7 @@ pg_physical_replication_slot_advance(XLogRecPtr moveto)
* WAL and removal of old catalog tuples. As decoding is done in fast_forward
* mode, no changes are generated anyway.
*/
-static XLogRecPtr
+XLogRecPtr
pg_logical_replication_slot_advance(XLogRecPtr moveto)
{
LogicalDecodingContext *ctx;
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d27ef2985d..26d07ae549 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -473,6 +473,194 @@ IdentifySystem(void)
end_tup_output(tstate);
}
+static int
+pg_qsort_namecmp(const void *a, const void *b)
+{
+ return strncmp(NameStr(*(Name) a), NameStr(*(Name) b), NAMEDATALEN);
+}
+
+/*
+ * Handle the LIST_SLOTS command.
+ */
+static void
+ListSlots(ListSlotsCmd *cmd)
+{
+ DestReceiver *dest;
+ TupOutputState *tstate;
+ TupleDesc tupdesc;
+ NameData *slot_names;
+ int numslot_names;
+
+ numslot_names = list_length(cmd->slot_names);
+ if (numslot_names)
+ {
+ ListCell *lc;
+ int i = 0;
+
+ slot_names = palloc(numslot_names * sizeof(NameData));
+ foreach(lc, cmd->slot_names)
+ {
+ char *slot_name = lfirst(lc);
+
+ ReplicationSlotValidateName(slot_name, ERROR);
+ namestrcpy(&slot_names[i++], slot_name);
+ }
+
+ qsort(slot_names, numslot_names, sizeof(NameData), pg_qsort_namecmp);
+ }
+
+ dest = CreateDestReceiver(DestRemoteSimple);
+
+ /* need a tuple descriptor representing four columns */
+ tupdesc = CreateTemplateTupleDesc(10);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 1, "slot_name",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 2, "plugin",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 3, "slot_type",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 4, "datoid",
+ INT8OID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 5, "database",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 6, "temporary",
+ INT4OID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 7, "xmin",
+ INT8OID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 8, "catalog_xmin",
+ INT8OID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 9, "restart_lsn",
+ TEXTOID, -1, 0);
+ TupleDescInitBuiltinEntry(tupdesc, (AttrNumber) 10, "confirmed_flush",
+ TEXTOID, -1, 0);
+
+ /* prepare for projection of tuples */
+ tstate = begin_tup_output_tupdesc(dest, tupdesc, &TTSOpsVirtual);
+
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (int slotno = 0; slotno < max_replication_slots; slotno++)
+ {
+ ReplicationSlot *slot = &ReplicationSlotCtl->replication_slots[slotno];
+ char restart_lsn_str[MAXFNAMELEN];
+ char confirmed_flush_lsn_str[MAXFNAMELEN];
+ Datum values[10];
+ bool nulls[10];
+
+ ReplicationSlotPersistency persistency;
+ TransactionId xmin;
+ TransactionId catalog_xmin;
+ XLogRecPtr restart_lsn;
+ XLogRecPtr confirmed_flush_lsn;
+ Oid datoid;
+ NameData slot_name;
+ NameData plugin;
+ int i;
+ int64 tmpbigint;
+
+ if (!slot->in_use)
+ continue;
+
+ SpinLockAcquire(&slot->mutex);
+
+ xmin = slot->data.xmin;
+ catalog_xmin = slot->data.catalog_xmin;
+ datoid = slot->data.database;
+ restart_lsn = slot->data.restart_lsn;
+ confirmed_flush_lsn = slot->data.confirmed_flush;
+ namestrcpy(&slot_name, NameStr(slot->data.name));
+ namestrcpy(&plugin, NameStr(slot->data.plugin));
+ persistency = slot->data.persistency;
+
+ SpinLockRelease(&slot->mutex);
+
+ if (numslot_names &&
+ !bsearch((void *) &slot_name, (void *) slot_names,
+ numslot_names, sizeof(NameData), pg_qsort_namecmp))
+ continue;
+
+ memset(nulls, 0, sizeof(nulls));
+
+ i = 0;
+ values[i++] = CStringGetTextDatum(NameStr(slot_name));
+
+ if (datoid == InvalidOid)
+ nulls[i++] = true;
+ else
+ values[i++] = CStringGetTextDatum(NameStr(plugin));
+
+ if (datoid == InvalidOid)
+ values[i++] = CStringGetTextDatum("physical");
+ else
+ values[i++] = CStringGetTextDatum("logical");
+
+ if (datoid == InvalidOid)
+ nulls[i++] = true;
+ else
+ {
+ tmpbigint = datoid;
+ values[i++] = Int64GetDatum(tmpbigint);
+ }
+
+ if (datoid == InvalidOid)
+ nulls[i++] = true;
+ else
+ {
+ MemoryContext cur = CurrentMemoryContext;
+
+ /* syscache access needs a transaction env. */
+ StartTransactionCommand();
+ /* make dbname live outside TX context */
+ MemoryContextSwitchTo(cur);
+ values[i++] = CStringGetTextDatum(get_database_name(datoid));
+ CommitTransactionCommand();
+ /* CommitTransactionCommand switches to TopMemoryContext */
+ MemoryContextSwitchTo(cur);
+ }
+
+ values[i++] = Int32GetDatum(persistency == RS_TEMPORARY ? 1 : 0);
+
+ if (xmin != InvalidTransactionId)
+ {
+ tmpbigint = xmin;
+ values[i++] = Int64GetDatum(tmpbigint);
+ }
+ else
+ nulls[i++] = true;
+
+ if (catalog_xmin != InvalidTransactionId)
+ {
+ tmpbigint = catalog_xmin;
+ values[i++] = Int64GetDatum(tmpbigint);
+ }
+ else
+ nulls[i++] = true;
+
+ if (restart_lsn != InvalidXLogRecPtr)
+ {
+ snprintf(restart_lsn_str, sizeof(restart_lsn_str), "%X/%X",
+ LSN_FORMAT_ARGS(restart_lsn));
+ values[i++] = CStringGetTextDatum(restart_lsn_str);
+ }
+ else
+ nulls[i++] = true;
+
+ if (confirmed_flush_lsn != InvalidXLogRecPtr)
+ {
+ snprintf(confirmed_flush_lsn_str, sizeof(confirmed_flush_lsn_str),
+ "%X/%X", LSN_FORMAT_ARGS(confirmed_flush_lsn));
+ values[i++] = CStringGetTextDatum(confirmed_flush_lsn_str);
+ }
+ else
+ nulls[i++] = true;
+
+ /* send it to dest */
+ do_tup_output(tstate, values, nulls);
+ }
+ LWLockRelease(ReplicationSlotControlLock);
+
+ end_tup_output(tstate);
+}
+
/* Handle READ_REPLICATION_SLOT command */
static void
ReadReplicationSlot(ReadReplicationSlotCmd *cmd)
@@ -1819,6 +2007,13 @@ exec_replication_command(const char *cmd_string)
EndReplicationCommand(cmdtag);
break;
+ case T_ListSlotsCmd:
+ cmdtag = "LIST_SLOTS";
+ set_ps_display(cmdtag);
+ ListSlots((ListSlotsCmd *) cmd_node);
+ EndReplicationCommand(cmdtag);
+ break;
+
case T_StartReplicationCmd:
{
StartReplicationCmd *cmd = (StartReplicationCmd *) cmd_node;
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 3fabad96d9..7bfef97df1 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -53,6 +53,7 @@ WAIT_EVENT_LOGICAL_APPLY_MAIN LogicalApplyMain "Waiting in main loop of logical
WAIT_EVENT_LOGICAL_LAUNCHER_MAIN LogicalLauncherMain "Waiting in main loop of logical replication launcher process."
WAIT_EVENT_LOGICAL_PARALLEL_APPLY_MAIN LogicalParallelApplyMain "Waiting in main loop of logical replication parallel apply process."
WAIT_EVENT_RECOVERY_WAL_STREAM RecoveryWalStream "Waiting in main loop of startup process for WAL to arrive, during streaming recovery."
+WAIT_EVENT_REPL_SLOT_SYNC_MAIN ReplSlotSyncMain "Waiting in main loop of worker for synchronizing slots to a standby from primary."
WAIT_EVENT_SYSLOGGER_MAIN SysLoggerMain "Waiting in main loop of syslogger process."
WAIT_EVENT_WAL_RECEIVER_MAIN WalReceiverMain "Waiting in main loop of WAL receiver process."
WAIT_EVENT_WAL_SENDER_MAIN WalSenderMain "Waiting in main loop of WAL sender process."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index d72b6b95b6..131e32273c 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -63,8 +63,11 @@
#include "postmaster/syslogger.h"
#include "postmaster/walwriter.h"
#include "replication/logicallauncher.h"
+#include "replication/reorderbuffer.h"
#include "replication/slot.h"
#include "replication/syncrep.h"
+#include "replication/walreceiver.h"
+#include "replication/walsender.h"
#include "storage/bufmgr.h"
#include "storage/large_object.h"
#include "storage/pg_shmem.h"
diff --git a/src/include/commands/subscriptioncmds.h b/src/include/commands/subscriptioncmds.h
index 214dc6c29e..0e77f9ee5c 100644
--- a/src/include/commands/subscriptioncmds.h
+++ b/src/include/commands/subscriptioncmds.h
@@ -17,6 +17,7 @@
#include "catalog/objectaddress.h"
#include "parser/parse_node.h"
+#include "replication/walreceiver.h"
extern ObjectAddress CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
bool isTopLevel);
@@ -28,4 +29,6 @@ extern void AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId);
extern char defGetStreamingMode(DefElem *def);
+extern void ReplicationSlotDropAtPubNode(WalReceiverConn *wrconn, char *slotname, bool missing_ok);
+
#endif /* SUBSCRIPTIONCMDS_H */
diff --git a/src/include/nodes/replnodes.h b/src/include/nodes/replnodes.h
index 4321ba8f86..980e0b2ee2 100644
--- a/src/include/nodes/replnodes.h
+++ b/src/include/nodes/replnodes.h
@@ -33,6 +33,15 @@ typedef struct IdentifySystemCmd
NodeTag type;
} IdentifySystemCmd;
+/* ----------------------
+ * LIST_SLOTS command
+ * ----------------------
+ */
+typedef struct ListSlotsCmd
+{
+ NodeTag type;
+ List *slot_names;
+} ListSlotsCmd;
/* ----------------------
* BASE_BACKUP command
diff --git a/src/include/replication/logicallauncher.h b/src/include/replication/logicallauncher.h
index a07c9cb311..80fdbf9657 100644
--- a/src/include/replication/logicallauncher.h
+++ b/src/include/replication/logicallauncher.h
@@ -31,4 +31,6 @@ extern bool IsLogicalLauncher(void);
extern pid_t GetLeaderApplyWorkerPid(pid_t pid);
+extern PGDLLIMPORT char *PrimaryConnInfo;
+
#endif /* LOGICALLAUNCHER_H */
diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h
index 39588da79f..4bb190ab81 100644
--- a/src/include/replication/logicalworker.h
+++ b/src/include/replication/logicalworker.h
@@ -18,6 +18,7 @@ extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending;
extern void ApplyWorkerMain(Datum main_arg);
extern void ParallelApplyWorkerMain(Datum main_arg);
+extern void ReplSlotSyncMain(Datum main_arg);
extern bool IsLogicalWorker(void);
extern bool IsLogicalParallelApplyWorker(void);
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 2765f99ccf..1d44a64736 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -15,7 +15,6 @@
#include "storage/lwlock.h"
#include "storage/shmem.h"
#include "storage/spin.h"
-#include "replication/walreceiver.h"
/*
* Behaviour of replication slots, upon release or crash.
@@ -240,7 +239,6 @@ extern ReplicationSlot *SearchNamedReplicationSlot(const char *name, bool need_l
extern int ReplicationSlotIndex(ReplicationSlot *slot);
extern bool ReplicationSlotName(int index, Name name);
extern void ReplicationSlotNameForTablesync(Oid suboid, Oid relid, char *syncslotname, Size szslot);
-extern void ReplicationSlotDropAtPubNode(WalReceiverConn *wrconn, char *slotname, bool missing_ok);
extern void StartupReplicationSlots(void);
extern void CheckPointReplicationSlots(void);
@@ -250,4 +248,7 @@ extern void CheckSlotPermissions(void);
extern void WaitForStandbyLSN(XLogRecPtr wait_for_lsn);
+extern XLogRecPtr pg_logical_replication_slot_advance(XLogRecPtr moveto);
+
+
#endif /* SLOT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 281626fa6f..9e9d64faf2 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -20,6 +20,7 @@
#include "pgtime.h"
#include "port/atomics.h"
#include "replication/logicalproto.h"
+#include "replication/slot.h"
#include "replication/walsender.h"
#include "storage/condition_variable.h"
#include "storage/latch.h"
@@ -191,6 +192,17 @@ typedef struct
} proto;
} WalRcvStreamOptions;
+/*
+ * Slot information receiver from remote.
+ *
+ * Currently same as ReplicationSlotPersistentData except last_sync_time
+ */
+typedef struct WalRecvReplicationSlotData
+{
+ ReplicationSlotPersistentData persistent_data;
+ TimestampTz last_sync_time;
+} WalRecvReplicationSlotData;
+
struct WalReceiverConn;
typedef struct WalReceiverConn WalReceiverConn;
@@ -280,6 +292,11 @@ typedef void (*walrcv_get_senderinfo_fn) (WalReceiverConn *conn,
typedef char *(*walrcv_identify_system_fn) (WalReceiverConn *conn,
TimeLineID *primary_tli);
+/*
+ * TODO
+ */
+typedef List *(*walrcv_list_slots_fn) (WalReceiverConn *conn, const char *slots);
+
/*
* walrcv_server_version_fn
*
@@ -393,6 +410,7 @@ typedef struct WalReceiverFunctionsType
walrcv_get_conninfo_fn walrcv_get_conninfo;
walrcv_get_senderinfo_fn walrcv_get_senderinfo;
walrcv_identify_system_fn walrcv_identify_system;
+ walrcv_list_slots_fn walrcv_list_slots;
walrcv_server_version_fn walrcv_server_version;
walrcv_readtimelinehistoryfile_fn walrcv_readtimelinehistoryfile;
walrcv_startstreaming_fn walrcv_startstreaming;
@@ -417,6 +435,8 @@ extern PGDLLIMPORT WalReceiverFunctionsType *WalReceiverFunctions;
WalReceiverFunctions->walrcv_get_senderinfo(conn, sender_host, sender_port)
#define walrcv_identify_system(conn, primary_tli) \
WalReceiverFunctions->walrcv_identify_system(conn, primary_tli)
+#define walrcv_list_slots(conn, slots) \
+ WalReceiverFunctions->walrcv_list_slots(conn, slots)
#define walrcv_server_version(conn) \
WalReceiverFunctions->walrcv_server_version(conn)
#define walrcv_readtimelinehistoryfile(conn, tli, filename, content, size) \
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index 343e781896..d42cff3f2c 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -65,7 +65,7 @@ typedef struct LogicalRepWorker
* would be created for each transaction which will be deleted after the
* transaction is finished.
*/
- FileSet *stream_fileset;
+ struct FileSet *stream_fileset;
/*
* PID of leader apply worker if this slot is used for a parallel apply
@@ -228,15 +228,15 @@ extern PGDLLIMPORT bool in_remote_transaction;
extern PGDLLIMPORT bool InitializingApplyWorker;
extern void logicalrep_worker_attach(int slot);
-extern LogicalRepWorker *logicalrep_worker_find(Oid subid, Oid relid,
+extern LogicalRepWorker *logicalrep_worker_find(Oid dbid, Oid subid, Oid relid,
bool only_running);
extern List *logicalrep_workers_find(Oid subid, bool only_running);
extern bool logicalrep_worker_launch(Oid dbid, Oid subid, const char *subname,
Oid userid, Oid relid,
dsm_handle subworker_dsm);
-extern void logicalrep_worker_stop(Oid subid, Oid relid);
+extern void logicalrep_worker_stop(Oid dbid, Oid subid, Oid relid);
extern void logicalrep_pa_worker_stop(ParallelApplyWorkerInfo *winfo);
-extern void logicalrep_worker_wakeup(Oid subid, Oid relid);
+extern void logicalrep_worker_wakeup(Oid dbid, Oid subid, Oid relid);
extern void logicalrep_worker_wakeup_ptr(LogicalRepWorker *worker);
extern int logicalrep_sync_worker_count(Oid subid);
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index ee590eeac7..ca043d2009 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -44,6 +44,7 @@ tests += {
't/036_truncated_dropped.pl',
't/037_invalid_database.pl',
't/050_verify_slot_order.pl',
+ 't/051_slot_sync.pl',
],
},
}
diff --git a/src/test/recovery/t/051_slot_sync.pl b/src/test/recovery/t/051_slot_sync.pl
new file mode 100644
index 0000000000..febe4e3db8
--- /dev/null
+++ b/src/test/recovery/t/051_slot_sync.pl
@@ -0,0 +1,132 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+my $node_phys_standby = PostgreSQL::Test::Cluster->new('phys_standby');
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+
+# find $pat in logfile of $node after $off-th byte
+sub find_in_log
+{
+ my ($node, $pat, $off) = @_;
+
+ $off = 0 unless defined $off;
+ my $log = PostgreSQL::Test::Utils::slurp_file($node->logfile);
+ return 0 if (length($log) <= $off);
+
+ $log = substr($log, $off);
+
+ return $log =~ m/$pat/;
+}
+
+# Check invalidation in the logfile
+sub check_for_invalidation
+{
+ my ($log_start, $test_name) = @_;
+
+ # message should be issued
+ ok( find_in_log(
+ $node_phys_standby,
+ "invalidating obsolete replication slot \"sub1\"", $log_start),
+ "sub1 slot invalidation is logged $test_name");
+}
+
+# Check conflicting status in pg_replication_slots.
+sub check_slots_conflicting_status
+{
+ my $res = $node_phys_standby->safe_psql(
+ 'postgres', qq(
+ select bool_and(conflicting) from pg_replication_slots;));
+
+ is($res, 't',
+ "Logical slot is reported as conflicting");
+}
+
+$node_primary->init(allows_streaming => 'logical');
+$node_primary->append_conf('postgresql.conf', q{
+synchronize_slot_names = '*'
+standby_slot_names = 'pslot1'
+});
+$node_primary->start;
+$node_primary->psql('postgres', q{SELECT pg_create_physical_replication_slot('pslot1');});
+
+$node_primary->backup('backup');
+
+$node_phys_standby->init_from_backup($node_primary, 'backup', has_streaming => 1);
+$node_phys_standby->append_conf('postgresql.conf', q{
+synchronize_slot_names = '*'
+primary_slot_name = 'pslot1'
+hot_standby_feedback = off
+});
+$node_phys_standby->start;
+
+$node_primary->safe_psql('postgres', "CREATE TABLE t1 (a int PRIMARY KEY)");
+$node_primary->safe_psql('postgres', "INSERT INTO t1 VALUES (1), (2), (3)");
+
+# Some tests need to wait for VACUUM to be replayed. But vacuum does not flush
+# WAL. An insert into flush_wal outside transaction does guarantee a flush.
+$node_primary->psql('postgres', q[CREATE TABLE flush_wal();]);
+
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+$node_subscriber->safe_psql('postgres', "CREATE TABLE t1 (a int PRIMARY KEY)");
+
+$node_primary->safe_psql('postgres', "CREATE PUBLICATION pub1 FOR TABLE t1");
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1 CONNECTION '" . ($node_primary->connstr . ' dbname=postgres') . "' PUBLICATION pub1");
+
+# Wait for initial sync of all subscriptions
+my $synced_query =
+ "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result = $node_primary->safe_psql('postgres',
+ "SELECT slot_name, plugin, database FROM pg_replication_slots WHERE slot_type = 'logical'");
+
+is($result, qq(sub1|pgoutput|postgres), 'logical slot on primary');
+
+# FIXME: standby needs restart to pick up new slots
+$node_phys_standby->restart;
+sleep 3;
+
+$result = $node_phys_standby->safe_psql('postgres',
+ "SELECT slot_name, plugin, database FROM pg_replication_slots");
+
+is($result, qq(sub1|pgoutput|postgres), 'logical slot on standby');
+
+$node_primary->safe_psql('postgres', "INSERT INTO t1 VALUES (4), (5), (6)");
+$node_primary->wait_for_catchup('sub1');
+
+$node_primary->wait_for_catchup($node_phys_standby->name);
+
+# Logical subscriber and physical replica are caught up at this point.
+
+# Drop the subscription so that catalog_xmin is unknown on the primary
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub1");
+
+# This should trigger a conflict as hot_standby_feedback is off on the standby
+$node_primary->safe_psql('postgres', qq[
+ CREATE TABLE conflict_test(x integer, y text);
+ DROP TABLE conflict_test;
+ VACUUM full pg_class;
+ INSERT INTO flush_wal DEFAULT VALUES; -- see create table flush_wal
+]);
+
+# Ensure physical replay catches up
+$node_primary->wait_for_catchup($node_phys_standby);
+
+# Check invalidation in the logfile
+check_for_invalidation(1, 'with vacuum FULL on pg_class');
+
+# Check conflicting status in pg_replication_slots.
+check_slots_conflicting_status();
+
+done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-20 11:34 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-21 06:05 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-21 11:46 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-24 02:32 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
@ 2023-07-24 03:30 ` Amit Kapila <[email protected]>
2023-07-26 05:01 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-28 14:39 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2 siblings, 2 replies; 59+ messages in thread
From: Amit Kapila @ 2023-07-24 03:30 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: shveta malik <[email protected]>; Drouvot, Bertrand <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
On Mon, Jul 24, 2023 at 8:03 AM Bharath Rupireddy
<[email protected]> wrote:
>
> On Fri, Jul 21, 2023 at 5:16 PM shveta malik <[email protected]> wrote:
> >
> > Thanks Bharat for letting us know. It is okay to split the patch, it
> > may definitely help to understand the modules better but shall we take
> > a step back and try to reevaluate the design first before moving to
> > other tasks?
>
> Agree that design comes first. FWIW, I'm attaching the v9 patch set
> that I have with me. It can't be a perfect patch set unless the design
> is finalized.
>
> > I analyzed more on the issues stated in [1] for replacing LIST_SLOTS
> > with SELECT query. On rethinking, it might not be a good idea to
> > replace this cmd with SELECT in Launcher code-path
>
> I think there are open fundamental design aspects, before optimizing
> LIST_SLOTS, see below. I'm sure we can come back to this later.
>
> > Secondly, I was thinking if the design proposed in the patch is the
> > best one. No doubt, it is the most simplistic design and thus may
> > .......... Any feedback is appreciated.
>
> Here are my thoughts about this feature:
>
> Current design:
>
> 1. On primary, never allow walsenders associated with logical
> replication slots to go ahead of physical standbys that are candidates
> for future primary after failover. This enables subscribers to connect
> to new primary after failover.
> 2. On all candidate standbys, periodically sync logical slots from
> primary (creating the slots if necessary) with one slot sync worker
> per logical slot.
>
> Important considerations:
>
> 1. Does this design guarantee the row versions required by subscribers
> aren't removed on candidate standbys as raised here -
> https://www.postgresql.org/message-id/20220218222319.yozkbhren7vkjbi5%40alap3.anarazel.de?
>
> It seems safe with logical decoding on standbys feature. Also, a
> test-case from upthread is already in patch sets (in v9 too)
> https://www.postgresql.org/message-id/CAAaqYe9FdKODa1a9n%3Dqj%2Bw3NiB9gkwvhRHhcJNginuYYRCnLrg%40mail....
> However, we need to verify the use cases extensively.
>
Agreed.
> 2. All candidate standbys will start one slot sync worker per logical
> slot which might not be scalable.
>
Yeah, that doesn't sound like a good idea but IIRC, the proposed patch
is using one worker per database (for all slots corresponding to a
database).
> Is having one (or a few more - not
> necessarily one for each logical slot) worker for all logical slots
> enough?
>
I guess for a large number of slots the is a possibility of a large
gap in syncing the slots which probably means we need to retain
corresponding WAL for a much longer time on the primary. If we can
prove that the gap won't be large enough to matter then this would be
probably worth considering otherwise, I think we should find a way to
scale the number of workers to avoid the large gap.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-20 11:34 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-21 06:05 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-21 11:46 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-24 02:32 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-24 03:30 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
@ 2023-07-26 05:01 ` shveta malik <[email protected]>
2023-07-27 05:24 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
1 sibling, 1 reply; 59+ messages in thread
From: shveta malik @ 2023-07-26 05:01 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; Drouvot, Bertrand <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; shveta malik <[email protected]>
On Mon, Jul 24, 2023 at 9:00 AM Amit Kapila <[email protected]> wrote:
>
> On Mon, Jul 24, 2023 at 8:03 AM Bharath Rupireddy
> <[email protected]> wrote:
> >
> > On Fri, Jul 21, 2023 at 5:16 PM shveta malik <[email protected]> wrote:
> > >
> > > Thanks Bharat for letting us know. It is okay to split the patch, it
> > > may definitely help to understand the modules better but shall we take
> > > a step back and try to reevaluate the design first before moving to
> > > other tasks?
> >
> > Agree that design comes first. FWIW, I'm attaching the v9 patch set
> > that I have with me. It can't be a perfect patch set unless the design
> > is finalized.
> >
> > > I analyzed more on the issues stated in [1] for replacing LIST_SLOTS
> > > with SELECT query. On rethinking, it might not be a good idea to
> > > replace this cmd with SELECT in Launcher code-path
> >
> > I think there are open fundamental design aspects, before optimizing
> > LIST_SLOTS, see below. I'm sure we can come back to this later.
> >
> > > Secondly, I was thinking if the design proposed in the patch is the
> > > best one. No doubt, it is the most simplistic design and thus may
> > > .......... Any feedback is appreciated.
> >
> > Here are my thoughts about this feature:
> >
> > Current design:
> >
> > 1. On primary, never allow walsenders associated with logical
> > replication slots to go ahead of physical standbys that are candidates
> > for future primary after failover. This enables subscribers to connect
> > to new primary after failover.
> > 2. On all candidate standbys, periodically sync logical slots from
> > primary (creating the slots if necessary) with one slot sync worker
> > per logical slot.
> >
> > Important considerations:
> >
> > 1. Does this design guarantee the row versions required by subscribers
> > aren't removed on candidate standbys as raised here -
> > https://www.postgresql.org/message-id/20220218222319.yozkbhren7vkjbi5%40alap3.anarazel.de?
> >
> > It seems safe with logical decoding on standbys feature. Also, a
> > test-case from upthread is already in patch sets (in v9 too)
> > https://www.postgresql.org/message-id/CAAaqYe9FdKODa1a9n%3Dqj%2Bw3NiB9gkwvhRHhcJNginuYYRCnLrg%40mail....
> > However, we need to verify the use cases extensively.
> >
>
> Agreed.
>
> > 2. All candidate standbys will start one slot sync worker per logical
> > slot which might not be scalable.
> >
>
> Yeah, that doesn't sound like a good idea but IIRC, the proposed patch
> is using one worker per database (for all slots corresponding to a
> database).
>
> > Is having one (or a few more - not
> > necessarily one for each logical slot) worker for all logical slots
> > enough?
> >
>
> I guess for a large number of slots the is a possibility of a large
> gap in syncing the slots which probably means we need to retain
> corresponding WAL for a much longer time on the primary. If we can
> prove that the gap won't be large enough to matter then this would be
> probably worth considering otherwise, I think we should find a way to
> scale the number of workers to avoid the large gap.
>
How about this:
1) On standby, spawn 1 worker per database in the start (as it is
doing currently).
2) Maintain statistics on activity against each primary's database on
standby by any means. Could be by maintaining 'last_synced_time' and
'last_activity_seen time'. The last_synced_time is updated every time
we sync/recheck slots for that particular database. The
'last_activity_seen_time' changes only if we get any slot on that
database where actually confirmed_flush or say restart_lsn has changed
from what was maintained already.
3) If at any moment, we find that 'last_synced_time' -
'last_activity_seen' goes beyond a threshold, that means that DB is
not active currently. Add it to list of inactive DB
4) Launcher on the other hand is always checking if it needs to spawn
any other extra worker for any new DB. It will additionally check if
number of inactive databases (maintained on standby) has gone higher
(> some threshold), then it brings down the workers for those and
starts a common worker which takes care of all such inactive databases
(or merge all in 1), while workers for active databases remain as such
(i.e. one per db). Each worker maintains the list of DBs which it is
responsible for.
5) If in the list of these inactive databases, we again find any
active database using the above logic, then the launcher will spawn a
separate worker for that.
Pros:
Lesser workers on standby as per the load on primary.
Lesser poking of primary by standby i.e. standby will send queries to
get slot info for all inactive DBs in 1 run instead of each worker
sending such queries separately.
Cons: We might see spawning and freeing of workers more frequently.
Please let me know your thoughts on this.
thanks
Shveta
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-20 11:34 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-21 06:05 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-21 11:46 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-24 02:32 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-24 03:30 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-26 05:01 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
@ 2023-07-27 05:24 ` Amit Kapila <[email protected]>
2023-07-28 15:24 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Amit Kapila @ 2023-07-27 05:24 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; Drouvot, Bertrand <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
On Wed, Jul 26, 2023 at 10:31 AM shveta malik <[email protected]> wrote:
>
> On Mon, Jul 24, 2023 at 9:00 AM Amit Kapila <[email protected]> wrote:
> >
> > On Mon, Jul 24, 2023 at 8:03 AM Bharath Rupireddy
> > <[email protected]> wrote:
> > >
> > > Is having one (or a few more - not
> > > necessarily one for each logical slot) worker for all logical slots
> > > enough?
> > >
> >
> > I guess for a large number of slots the is a possibility of a large
> > gap in syncing the slots which probably means we need to retain
> > corresponding WAL for a much longer time on the primary. If we can
> > prove that the gap won't be large enough to matter then this would be
> > probably worth considering otherwise, I think we should find a way to
> > scale the number of workers to avoid the large gap.
> >
>
> How about this:
>
> 1) On standby, spawn 1 worker per database in the start (as it is
> doing currently).
>
> 2) Maintain statistics on activity against each primary's database on
> standby by any means. Could be by maintaining 'last_synced_time' and
> 'last_activity_seen time'. The last_synced_time is updated every time
> we sync/recheck slots for that particular database. The
> 'last_activity_seen_time' changes only if we get any slot on that
> database where actually confirmed_flush or say restart_lsn has changed
> from what was maintained already.
>
> 3) If at any moment, we find that 'last_synced_time' -
> 'last_activity_seen' goes beyond a threshold, that means that DB is
> not active currently. Add it to list of inactive DB
>
I think we should also increase the next_sync_time if in current sync,
there is no update.
> 4) Launcher on the other hand is always checking if it needs to spawn
> any other extra worker for any new DB. It will additionally check if
> number of inactive databases (maintained on standby) has gone higher
> (> some threshold), then it brings down the workers for those and
> starts a common worker which takes care of all such inactive databases
> (or merge all in 1), while workers for active databases remain as such
> (i.e. one per db). Each worker maintains the list of DBs which it is
> responsible for.
>
> 5) If in the list of these inactive databases, we again find any
> active database using the above logic, then the launcher will spawn a
> separate worker for that.
>
I wonder if we anyway some sort of design like this because we
shouldn't allow to spawn as many workers as the number of databases.
There has to be some existing or new GUC like max_sync_slot_workers
which decided the number of workers.
Overall, this sounds to be a more workload-adaptive approach as
compared to the current one.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-20 11:34 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-21 06:05 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-21 11:46 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-24 02:32 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-24 03:30 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-26 05:01 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-27 05:24 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
@ 2023-07-28 15:24 ` Bharath Rupireddy <[email protected]>
2023-08-01 11:31 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Bharath Rupireddy @ 2023-07-28 15:24 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: shveta malik <[email protected]>; Drouvot, Bertrand <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
On Thu, Jul 27, 2023 at 10:55 AM Amit Kapila <[email protected]> wrote:
>
> I wonder if we anyway some sort of design like this because we
> shouldn't allow to spawn as many workers as the number of databases.
> There has to be some existing or new GUC like max_sync_slot_workers
> which decided the number of workers.
It seems reasonable to not have one slot sync worker for each
database. IMV, the slot sync workers must be generic and independently
manageable - generic in the sense that given a database and primary
conninfo, each worker must sync all the slots related to the given
database, independently mangeable in the sense that separate GUC for
number of sync workers, launchable directly by logical replication
launcher dynamically. The work division amongst the sync workers can
be simple, the logical replication launcher builds a shared memory
structure based on number of slots to sync and starts the sync workers
dynamically, and each sync worker picks {dboid, slot name, conninfo}
from the shared memory, syncs it and proceeds with other slots.
Thoughts?
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-20 11:34 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-21 06:05 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-21 11:46 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-24 02:32 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-24 03:30 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-26 05:01 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-27 05:24 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-28 15:24 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
@ 2023-08-01 11:31 ` shveta malik <[email protected]>
2023-08-02 18:57 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: shveta malik @ 2023-08-01 11:31 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Amit Kapila <[email protected]>; Drouvot, Bertrand <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; shveta malik <[email protected]>
On Fri, Jul 28, 2023 at 8:54 PM Bharath Rupireddy
<[email protected]> wrote:
>
> On Thu, Jul 27, 2023 at 10:55 AM Amit Kapila <[email protected]> wrote:
> >
> > I wonder if we anyway some sort of design like this because we
> > shouldn't allow to spawn as many workers as the number of databases.
> > There has to be some existing or new GUC like max_sync_slot_workers
> > which decided the number of workers.
>
> It seems reasonable to not have one slot sync worker for each
> database. IMV, the slot sync workers must be generic and independently
> manageable - generic in the sense that given a database and primary
> conninfo, each worker must sync all the slots related to the given
> database, independently mangeable in the sense that separate GUC for
> number of sync workers, launchable directly by logical replication
> launcher dynamically.
yes agreed. The patch v10-0003 attempts to do the same.
> The work division amongst the sync workers can
> be simple, the logical replication launcher builds a shared memory
> structure based on number of slots to sync and starts the sync workers
> dynamically, and each sync worker picks {dboid, slot name, conninfo}
> from the shared memory, syncs it and proceeds with other slots.
>
Do you mean the logical replication launcher builds a shared memory
structure based
on the number of 'dbs' to sync as I understood from your initial comment?
thanks
Shveta
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-20 11:34 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-21 06:05 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-21 11:46 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-24 02:32 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-24 03:30 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-26 05:01 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-27 05:24 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-28 15:24 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-08-01 11:31 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
@ 2023-08-02 18:57 ` Bharath Rupireddy <[email protected]>
2023-08-03 10:49 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Bharath Rupireddy @ 2023-08-02 18:57 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Amit Kapila <[email protected]>; Drouvot, Bertrand <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
On Tue, Aug 1, 2023 at 5:01 PM shveta malik <[email protected]> wrote:
>
> > The work division amongst the sync workers can
> > be simple, the logical replication launcher builds a shared memory
> > structure based on number of slots to sync and starts the sync workers
> > dynamically, and each sync worker picks {dboid, slot name, conninfo}
> > from the shared memory, syncs it and proceeds with other slots.
>
> Do you mean the logical replication launcher builds a shared memory
> structure based
> on the number of 'dbs' to sync as I understood from your initial comment?
Yes. I haven't looked at the 0003 patch posted upthread. However, the
standby must do the following at a minimum:
- Make GUCs synchronize_slot_names and max_slot_sync_workers of
PGC_POSTMASTER type needing postmaster restart when changed as they
affect the number of slot sync workers.
- LR (logical replication) launcher connects to primary to fetch the
logical slots specified in synchronize_slot_names. This is a one-time
task.
- LR launcher prepares a dynamic shared memory (created via
dsm_create) with some state like locks for IPC and an array of
{slot_name, dboid_associated_with_slot, is_sync_in_progress} - maximum
number of elements in the array is the number of slots specified in
synchronize_slot_names. This is a one-time task.
- LR launcher decides the *best* number of slot sync workers - (based
on some perf numbers) it can just launch, say, one worker per 2 or 4
or 8 etc. slots.
- Each slot sync worker then picks up a slot from the DSM, connects to
primary using primary conn info, syncs it, and moves to another slot.
Not having the capability of on-demand stop/launch of slot sync
workers makes the above design simple IMO.
Thoughts?
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-20 11:34 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-21 06:05 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-21 11:46 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-24 02:32 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-24 03:30 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-26 05:01 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-27 05:24 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-28 15:24 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-08-01 11:31 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-08-02 18:57 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
@ 2023-08-03 10:49 ` shveta malik <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: shveta malik @ 2023-08-03 10:49 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Amit Kapila <[email protected]>; Drouvot, Bertrand <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; shveta malik <[email protected]>
On Thu, Aug 3, 2023 at 12:28 AM Bharath Rupireddy
<[email protected]> wrote:
>
> On Tue, Aug 1, 2023 at 5:01 PM shveta malik <[email protected]> wrote:
> >
> > > The work division amongst the sync workers can
> > > be simple, the logical replication launcher builds a shared memory
> > > structure based on number of slots to sync and starts the sync workers
> > > dynamically, and each sync worker picks {dboid, slot name, conninfo}
> > > from the shared memory, syncs it and proceeds with other slots.
> >
> > Do you mean the logical replication launcher builds a shared memory
> > structure based
> > on the number of 'dbs' to sync as I understood from your initial comment?
>
> Yes. I haven't looked at the 0003 patch posted upthread. However, the
> standby must do the following at a minimum:
>
> - Make GUCs synchronize_slot_names and max_slot_sync_workers of
> PGC_POSTMASTER type needing postmaster restart when changed as they
> affect the number of slot sync workers.
I agree that max_slot_sync_workers should be allowed to change only
during startup but I strongly feel that synchronize_slot_names should
be runtime modifiable. We should give that flexibility to the user.
> - LR (logical replication) launcher connects to primary to fetch the
> logical slots specified in synchronize_slot_names. This is a one-time
> task.
if synchronize_slot_names='*', we need to fetch slots info at regular
intervals even if it is not runtime modifiable. For a runtime
modifiable case, it is obvious to reftech it regular intervals.
> - LR launcher prepares a dynamic shared memory (created via
> dsm_create) with some state like locks for IPC and an array of
> {slot_name, dboid_associated_with_slot, is_sync_in_progress} - maximum
> number of elements in the array is the number of slots specified in
> synchronize_slot_names. This is a one-time task.
yes, we need dynamic-shared-memory but it is not a
one-time-allocation. If it were a one-time allocation, then there was
no need for DSM, only shared memory allocation was enough. It is not a
one time allocation in any of the designs. If it is slot based design,
slots may keep on varying for '*' case and if it is DB based design,
then number of DBs may go beyond the initial memory allocated and we
may need reallocation and relaunch of worker and thus the need of DSM.
> - LR launcher decides the *best* number of slot sync workers - (based
> on some perf numbers) it can just launch, say, one worker per 2 or 4
> or 8 etc. slots.
> - Each slot sync worker then picks up a slot from the DSM, connects to
> primary using primary conn info, syncs it, and moves to another slot.
>
The design based on slots i.e. launcher dividing the slots among the
available workers, could prove beneficial over db based division for a
case where number of slots per DB varies largely and we end up
assigning all DBs with lesser slots to one worker
while all heavily loaded DBs to another. But other than this, I see
lot of pain points:
1) Since we are going to do slots based synching, query construction
will be complex. We will have a query with a long 'where' clause:
where slots in (slot1, slot2, slots...).
2) Number of pings to primary will be more as we are pinging it slot
based instead of DB based. So the information which we could have
fetched collectively in one query (if it was db based) is now splitted
to multiple queries assuming that there could be cases where slots
belonging to the same DBs end up getting splitted among different
workers.
3) if number of slots < max number of workers, how are we going to
assign the worker? One slot per worker or all in one worker. If it is
one slot per worker, it will again be not that efficient as it will
result in more network traffic. This needs more thoughts and case to
case varying design.
> Not having the capability of on-demand stop/launch of slot sync
> workers makes the above design simple IMO.
>
We need to anyways relaunch workers when DSM is reallocated in case
Dbs (or sya slots) exceed some initial allocation limit.
thanks
Shveta
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-20 11:34 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-21 06:05 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-21 11:46 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-24 02:32 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-24 03:30 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
@ 2023-07-28 14:39 ` Bharath Rupireddy <[email protected]>
2023-08-04 09:14 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
1 sibling, 1 reply; 59+ messages in thread
From: Bharath Rupireddy @ 2023-07-28 14:39 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: shveta malik <[email protected]>; Drouvot, Bertrand <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
On Mon, Jul 24, 2023 at 9:00 AM Amit Kapila <[email protected]> wrote:
>
> > 2. All candidate standbys will start one slot sync worker per logical
> > slot which might not be scalable.
>
> Yeah, that doesn't sound like a good idea but IIRC, the proposed patch
> is using one worker per database (for all slots corresponding to a
> database).
Right. It's based on one worker for each database.
> > Is having one (or a few more - not
> > necessarily one for each logical slot) worker for all logical slots
> > enough?
>
> I guess for a large number of slots the is a possibility of a large
> gap in syncing the slots which probably means we need to retain
> corresponding WAL for a much longer time on the primary. If we can
> prove that the gap won't be large enough to matter then this would be
> probably worth considering otherwise, I think we should find a way to
> scale the number of workers to avoid the large gap.
I think the gap is largely determined by the time taken to advance
each slot and the amount of WAL that each logical slot moves ahead on
primary. I've measured the time it takes for
pg_logical_replication_slot_advance with different amounts WAL on my
system. It took 2595ms/5091ms/31238ms to advance the slot by
3.7GB/7.3GB/13GB respectively. To put things into perspective here,
imagine there are 3 logical slots to sync for a single slot sync
worker and each of them are in need of advancing the slot by
3.7GB/7.3GB/13GB of WAL. The slot sync worker gets to slot 1 again
after 2595ms+5091ms+31238ms (~40sec), gets to slot 2 again after
advance time of slot 1 with amount of WAL that the slot has moved
ahead on primary during 40sec, gets to slot 3 again after advance time
of slot 1 and slot 2 with amount of WAL that the slot has moved ahead
on primary and so on. If WAL generation on the primary is pretty fast,
and if the logical slot moves pretty fast on the primary, the time it
takes for a single sync worker to sync a slot can increase.
Now, let's think what happens if there's a large gap, IOW, a logical
slot on standby is behind X amount of WAL from that of the logical
slot on primary. The standby needs to retain more WAL for sure. IIUC,
primary doesn't need to retain the WAL required for a logical slot on
standby, no?
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-20 11:34 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-21 06:05 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-21 11:46 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-24 02:32 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-24 03:30 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-28 14:39 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
@ 2023-08-04 09:14 ` Drouvot, Bertrand <[email protected]>
2023-08-04 11:32 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Drouvot, Bertrand @ 2023-08-04 09:14 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; +Cc: shveta malik <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
Hi,
On 7/28/23 4:39 PM, Bharath Rupireddy wrote:
> On Mon, Jul 24, 2023 at 9:00 AM Amit Kapila <[email protected]> wrote:
>>
>>> 2. All candidate standbys will start one slot sync worker per logical
>>> slot which might not be scalable.
>>
>> Yeah, that doesn't sound like a good idea but IIRC, the proposed patch
>> is using one worker per database (for all slots corresponding to a
>> database).
>
> Right. It's based on one worker for each database.
>
>>> Is having one (or a few more - not
>>> necessarily one for each logical slot) worker for all logical slots
>>> enough?
>>
>> I guess for a large number of slots the is a possibility of a large
>> gap in syncing the slots which probably means we need to retain
>> corresponding WAL for a much longer time on the primary. If we can
>> prove that the gap won't be large enough to matter then this would be
>> probably worth considering otherwise, I think we should find a way to
>> scale the number of workers to avoid the large gap.
>
> I think the gap is largely determined by the time taken to advance
> each slot and the amount of WAL that each logical slot moves ahead on
> primary.
Sorry to be late, but I gave a second thought and I wonder if we really need this design.
(i.e start a logical replication background worker on the standby to sync the slots).
Wouldn't that be simpler to "just" update the sync slots "metadata"
as the https://github.com/EnterpriseDB/pg_failover_slots module (mentioned by Peter
up-thread) is doing?
(making use of LogicalConfirmReceivedLocation(), LogicalIncreaseXminForSlot()
and LogicalIncreaseRestartDecodingForSlot(), If I read synchronize_one_slot() correctly).
> I've measured the time it takes for
> pg_logical_replication_slot_advance with different amounts WAL on my
> system. It took 2595ms/5091ms/31238ms to advance the slot by
> 3.7GB/7.3GB/13GB respectively. To put things into perspective here,
> imagine there are 3 logical slots to sync for a single slot sync
> worker and each of them are in need of advancing the slot by
> 3.7GB/7.3GB/13GB of WAL. The slot sync worker gets to slot 1 again
> after 2595ms+5091ms+31238ms (~40sec), gets to slot 2 again after
> advance time of slot 1 with amount of WAL that the slot has moved
> ahead on primary during 40sec, gets to slot 3 again after advance time
> of slot 1 and slot 2 with amount of WAL that the slot has moved ahead
> on primary and so on. If WAL generation on the primary is pretty fast,
> and if the logical slot moves pretty fast on the primary, the time it
> takes for a single sync worker to sync a slot can increase.
That would be way "faster" and we would probably not need to
worry that much about the number of "sync" workers (if it/they "just" has/have
to sync slot's "metadata") as proposed above.
Thoughts?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-20 11:34 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-21 06:05 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-21 11:46 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-24 02:32 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-24 03:30 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-28 14:39 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-08-04 09:14 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
@ 2023-08-04 11:32 ` shveta malik <[email protected]>
2023-08-07 09:47 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: shveta malik @ 2023-08-04 11:32 UTC (permalink / raw)
To: Drouvot, Bertrand <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; shveta malik <[email protected]>
On Fri, Aug 4, 2023 at 2:44 PM Drouvot, Bertrand
<[email protected]> wrote:
>
> Hi,
>
> On 7/28/23 4:39 PM, Bharath Rupireddy wrote:
> > On Mon, Jul 24, 2023 at 9:00 AM Amit Kapila <[email protected]> wrote:
> >>
> >>> 2. All candidate standbys will start one slot sync worker per logical
> >>> slot which might not be scalable.
> >>
> >> Yeah, that doesn't sound like a good idea but IIRC, the proposed patch
> >> is using one worker per database (for all slots corresponding to a
> >> database).
> >
> > Right. It's based on one worker for each database.
> >
> >>> Is having one (or a few more - not
> >>> necessarily one for each logical slot) worker for all logical slots
> >>> enough?
> >>
> >> I guess for a large number of slots the is a possibility of a large
> >> gap in syncing the slots which probably means we need to retain
> >> corresponding WAL for a much longer time on the primary. If we can
> >> prove that the gap won't be large enough to matter then this would be
> >> probably worth considering otherwise, I think we should find a way to
> >> scale the number of workers to avoid the large gap.
> >
> > I think the gap is largely determined by the time taken to advance
> > each slot and the amount of WAL that each logical slot moves ahead on
> > primary.
>
> Sorry to be late, but I gave a second thought and I wonder if we really need this design.
> (i.e start a logical replication background worker on the standby to sync the slots).
>
> Wouldn't that be simpler to "just" update the sync slots "metadata"
> as the https://github.com/EnterpriseDB/pg_failover_slots module (mentioned by Peter
> up-thread) is doing?
> (making use of LogicalConfirmReceivedLocation(), LogicalIncreaseXminForSlot()
> and LogicalIncreaseRestartDecodingForSlot(), If I read synchronize_one_slot() correctly).
>
Agreed. It would be simpler to just update the metadata. I think you
have not got chance to review the latest posted patch ('v10-0003')
yet, it does the same.
But I do not quite get it as in how can we do it w/o starting a
background worker? Even the failover-slots extension starts one
background worker. The question here is how many background workers we
need to have. Will one be sufficient or do we need one per db (as done
earlier by the original patches in this thread) or are we good with
dividing work among some limited number of workers?
I feel syncing all slots in one worker may increase the lag between
subsequent syncs for a particular slot and if the number of slots are
huge, the chances of losing the slot-data is more in case of failure.
Starting one worker per db also might not be that efficient as it will
increase load on the system (both in terms of background worker and
network traffic) especially for a case where the number of dbs are
more. Thus starting max 'n' number of workers where 'n' is decided by
GUC and dividing the work/DBs among these looks a better option to me.
Please see the discussion in and around the email at [1]
[1]: https://www.postgresql.org/message-id/CAJpy0uCT%2BnpL4eUvCWiV_MBEri9ixcUgJVDdsBCJSqLd0oD1fQ%40mail.g...
> > I've measured the time it takes for
> > pg_logical_replication_slot_advance with different amounts WAL on my
> > system. It took 2595ms/5091ms/31238ms to advance the slot by
> > 3.7GB/7.3GB/13GB respectively. To put things into perspective here,
> > imagine there are 3 logical slots to sync for a single slot sync
> > worker and each of them are in need of advancing the slot by
> > 3.7GB/7.3GB/13GB of WAL. The slot sync worker gets to slot 1 again
> > after 2595ms+5091ms+31238ms (~40sec), gets to slot 2 again after
> > advance time of slot 1 with amount of WAL that the slot has moved
> > ahead on primary during 40sec, gets to slot 3 again after advance time
> > of slot 1 and slot 2 with amount of WAL that the slot has moved ahead
> > on primary and so on. If WAL generation on the primary is pretty fast,
> > and if the logical slot moves pretty fast on the primary, the time it
> > takes for a single sync worker to sync a slot can increase.
>
> That would be way "faster" and we would probably not need to
> worry that much about the number of "sync" workers (if it/they "just" has/have
> to sync slot's "metadata") as proposed above.
>
Agreed, we need not to worry about delay due to
pg_logical_replication_slot_advance if we are only going to update a
few simple things using the function calls as mentioned above.
thanks
Shveta
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-20 11:34 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-21 06:05 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-21 11:46 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-24 02:32 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-24 03:30 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-28 14:39 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-08-04 09:14 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-08-04 11:32 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
@ 2023-08-07 09:47 ` Drouvot, Bertrand <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Drouvot, Bertrand @ 2023-08-07 09:47 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
Hi,
On 8/4/23 1:32 PM, shveta malik wrote:
> On Fri, Aug 4, 2023 at 2:44 PM Drouvot, Bertrand
> <[email protected]> wrote:
>> On 7/28/23 4:39 PM, Bharath Rupireddy wrote:
>> Sorry to be late, but I gave a second thought and I wonder if we really need this design.
>> (i.e start a logical replication background worker on the standby to sync the slots).
>>
>> Wouldn't that be simpler to "just" update the sync slots "metadata"
>> as the https://github.com/EnterpriseDB/pg_failover_slots module (mentioned by Peter
>> up-thread) is doing?
>> (making use of LogicalConfirmReceivedLocation(), LogicalIncreaseXminForSlot()
>> and LogicalIncreaseRestartDecodingForSlot(), If I read synchronize_one_slot() correctly).
>>
>
> Agreed. It would be simpler to just update the metadata. I think you
> have not got chance to review the latest posted patch ('v10-0003')
> yet, it does the same.
Thanks for the feedback! Yeah, I did not look at v10 in details and was
looking at the email thread only.
Indeed, I now see that 0003 does update the metadata in local_slot_advance(),
that's great!
>
> But I do not quite get it as in how can we do it w/o starting a
> background worker?
Yeah, agree that we still need background workers.
What I meant was to avoid to use "logical replication background worker"
(aka through logicalrep_worker_launch()) to sync the slots.
> The question here is how many background workers we
> need to have. Will one be sufficient or do we need one per db (as done
> earlier by the original patches in this thread) or are we good with
> dividing work among some limited number of workers?
>
> I feel syncing all slots in one worker may increase the lag between
> subsequent syncs for a particular slot and if the number of slots are
> huge, the chances of losing the slot-data is more in case of failure.
> Starting one worker per db also might not be that efficient as it will
> increase load on the system (both in terms of background worker and
> network traffic) especially for a case where the number of dbs are
> more. Thus starting max 'n' number of workers where 'n' is decided by
> GUC and dividing the work/DBs among these looks a better option to me.
> Please see the discussion in and around the email at [1]
>
> [1]: https://www.postgresql.org/message-id/CAJpy0uCT%2BnpL4eUvCWiV_MBEri9ixcUgJVDdsBCJSqLd0oD1fQ%40mail.g...
Thanks for the link! If I read the email thread correctly, this discussion
was before V10 (which is the first version making use of LogicalConfirmReceivedLocation(),
LogicalIncreaseXminForSlot(), LogicalIncreaseRestartDecodingForSlot()) means
before the metadata sync only has been implemented.
While I agree that the approach to split the sync load among workers with the new
max_slot_sync_workers GUC seems reasonable without the sync only feature (pre V10),
I'm not sure that with the metadata sync only in place the extra complexity to manage multiple
sync workers is needed.
Maybe we should start some tests/benchmark with only one sync worker to get numbers
and start from there?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-20 11:34 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-21 06:05 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-21 11:46 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-24 02:32 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
@ 2023-07-26 12:25 ` shveta malik <[email protected]>
2023-07-27 06:54 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2 siblings, 1 reply; 59+ messages in thread
From: shveta malik @ 2023-07-26 12:25 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Amit Kapila <[email protected]>; Drouvot, Bertrand <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; shveta malik <[email protected]>
On Mon, Jul 24, 2023 at 8:03 AM Bharath Rupireddy
<[email protected]> wrote:
>
> On Fri, Jul 21, 2023 at 5:16 PM shveta malik <[email protected]> wrote:
> >
> > Thanks Bharat for letting us know. It is okay to split the patch, it
> > may definitely help to understand the modules better but shall we take
> > a step back and try to reevaluate the design first before moving to
> > other tasks?
>
> Agree that design comes first. FWIW, I'm attaching the v9 patch set
> that I have with me. It can't be a perfect patch set unless the design
> is finalized.
>
Thanks for the patch and summarizing all the issues here. I was going
through the patch and found that now we need to maintain
'synchronize_slot_names' on both primary and standby unlike the old
way where it was maintained only on standby. I am aware of the problem
in earlier implementation where each logical walsender/slot needed to
wait for all standbys to catch-up before sending changes to logical
subscribers even though that particular slot is not even needed to be
synced by any of the standbys. Now it is more restrictive. But now, is
this 'synchronize_slot_names' per standby? If there are multiple
standbys each having different 'synchronize_slot_names' requirements,
then how primary is going to keep track of that?
Please let me know if that scenario can never arise where standbys can
have different 'synchronize_slot_names'.
thanks
Shveta
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-20 11:34 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-21 06:05 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-21 11:46 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-24 02:32 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-26 12:25 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
@ 2023-07-27 06:54 ` Amit Kapila <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Amit Kapila @ 2023-07-27 06:54 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; Drouvot, Bertrand <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
On Wed, Jul 26, 2023 at 5:55 PM shveta malik <[email protected]> wrote:
>
> On Mon, Jul 24, 2023 at 8:03 AM Bharath Rupireddy
> <[email protected]> wrote:
> >
> > On Fri, Jul 21, 2023 at 5:16 PM shveta malik <[email protected]> wrote:
> > >
> > > Thanks Bharat for letting us know. It is okay to split the patch, it
> > > may definitely help to understand the modules better but shall we take
> > > a step back and try to reevaluate the design first before moving to
> > > other tasks?
> >
> > Agree that design comes first. FWIW, I'm attaching the v9 patch set
> > that I have with me. It can't be a perfect patch set unless the design
> > is finalized.
> >
>
> Thanks for the patch and summarizing all the issues here. I was going
> through the patch and found that now we need to maintain
> 'synchronize_slot_names' on both primary and standby unlike the old
> way where it was maintained only on standby. I am aware of the problem
> in earlier implementation where each logical walsender/slot needed to
> wait for all standbys to catch-up before sending changes to logical
> subscribers even though that particular slot is not even needed to be
> synced by any of the standbys. Now it is more restrictive. But now, is
> this 'synchronize_slot_names' per standby? If there are multiple
> standbys each having different 'synchronize_slot_names' requirements,
> then how primary is going to keep track of that?
> Please let me know if that scenario can never arise where standbys can
> have different 'synchronize_slot_names'.
>
Can we think of sending 'synchronize_slot_names' from standby to
primary at the time of connection? I think we also need to ensure that
if the user changes this value then we need to restart the sync slot
worker to allow this information to be sent to the primary. We do
something similar for apply worker in logical replication.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-07-20 11:34 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-21 06:05 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
2023-07-21 11:46 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]>
2023-07-24 02:32 ` Re: Synchronizing slots from primary to standby Bharath Rupireddy <[email protected]>
@ 2023-08-04 07:40 ` Drouvot, Bertrand <[email protected]>
2 siblings, 0 replies; 59+ messages in thread
From: Drouvot, Bertrand @ 2023-08-04 07:40 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; shveta malik <[email protected]>; +Cc: Amit Kapila <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
Hi,
On 7/24/23 4:32 AM, Bharath Rupireddy wrote:
> On Fri, Jul 21, 2023 at 5:16 PM shveta malik <[email protected]> wrote:
> Here are my thoughts about this feature:
Thanks for looking at it!
>
> Important considerations:
>
> 1. Does this design guarantee the row versions required by subscribers
> aren't removed on candidate standbys as raised here -
> https://www.postgresql.org/message-id/20220218222319.yozkbhren7vkjbi5%40alap3.anarazel.de?
>
> It seems safe with logical decoding on standbys feature. Also, a
> test-case from upthread is already in patch sets (in v9 too)
> https://www.postgresql.org/message-id/CAAaqYe9FdKODa1a9n%3Dqj%2Bw3NiB9gkwvhRHhcJNginuYYRCnLrg%40mail....
> However, we need to verify the use cases extensively.
Agree. We also discussed up-thread that we'd have to drop any "sync" slots if they
are invalidated. And they should be re-created based on the synchronize_slot_names.
> Please feel free to add the list if I'm missing anything.
>
We'd also have to ensure that "sync" slots can't be consumed on the standby (this has been
discussed up-thread).
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* RE: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
@ 2023-06-29 10:22 ` Hayato Kuroda (Fujitsu) <[email protected]>
2023-06-29 10:45 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
2023-06-30 06:09 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-07-01 01:42 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]>
1 sibling, 3 replies; 59+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2023-06-29 10:22 UTC (permalink / raw)
To: 'Drouvot, Bertrand' <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>
Dear Drouvot,
Hi, I'm also interested in the feature. Followings are my high-level comments.
I did not mention some detailed notations because this patch is not at the stage.
And very sorry that I could not follow all of this discussions.
1. I thought that we should not reuse logical replication launcher for another purpose.
The background worker should have only one task. I wanted to ask opinions some other people...
2. I want to confirm the reason why new replication command is added. IIUC the
launcher connects to primary by using primary_conninfo connection string, but
it establishes the physical replication connection so that any SQL cannot be executed.
Is it right? Another approach not to use is to specify the target database via
GUC, whereas not smart. How do you think?
3. You chose the per-db worker approach, however, it is difficult to extend the
feature to support physical slots. This may be problematic. Was there any
reasons for that? I doubted ReplicationSlotCreate() or advance functions might
not be used from other databases and these may be reasons, but not sure.
If these operations can do without connecting to specific database, I think
the architecture can be changed.
4. Currently the launcher establishes the connection every time. Isn't it better
to reuse the same one instead?
Following comments are assumed the configuration, maybe the straightfoward:
primary->standby
|->subscriber
5. After constructing the system, I dropped the subscription on the subscriber.
In this case the logical slot on primary was removed, but that was not replicated
to standby server. Did you support the workload or not?
```
$ psql -U postgres -p $port_sub -c "DROP SUBSCRIPTION sub"
NOTICE: dropped replication slot "sub" on publisher
DROP SUBSCRIPTION
$ psql -U postgres -p $port_primary -c "SELECT * FROM pg_replication_slots"
slot_name | plugin | slot_type | datoid | database |...
-----------+----------+-----------+--------+----------+...
(0 rows)
$ psql -U postgres -p $port_standby -c "SELECT * FROM pg_replication_slots"
slot_name | plugin | slot_type | datoid | database |...
-----------+----------+-----------+--------+----------+...
sub | pgoutput | logical | 5 | postgres |...
(1 row)
```
6. Current approach may delay the startpoint of sync.
Assuming that physical replication system is created first, and then the
subscriber connects to the publisher node. In this case the launcher connects to
primary earlier than the apply worker, and reads the slot. At that time there are
no slots on primary, so launcher disconnects from primary and waits a time period (up to 3min).
Even if the apply worker creates the slot on publisher, but the launcher on standby
cannot notice that. The synchronization may start 3 min later.
I'm not sure how to fix or it could be acceptable. Thought?
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-29 10:22 ` RE: Synchronizing slots from primary to standby Hayato Kuroda (Fujitsu) <[email protected]>
@ 2023-06-29 10:45 ` Amit Kapila <[email protected]>
2 siblings, 0 replies; 59+ messages in thread
From: Amit Kapila @ 2023-06-29 10:45 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: Drouvot, Bertrand <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>
On Thu, Jun 29, 2023 at 3:52 PM Hayato Kuroda (Fujitsu)
<[email protected]> wrote:
>
> Dear Drouvot,
>
> Hi, I'm also interested in the feature. Followings are my high-level comments.
> I did not mention some detailed notations because this patch is not at the stage.
> And very sorry that I could not follow all of this discussions.
>
> 1. I thought that we should not reuse logical replication launcher for another purpose.
> The background worker should have only one task. I wanted to ask opinions some other people...
>
IIUC, the launcher will launch the sync slot workers corresponding to
slots that need sync on standby and apply workers for active
subscriptions on primary (which will be a subscriber in this context).
If this is correct, then do you expect to launch a separate kind of
standby launcher for sync slots?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-29 10:22 ` RE: Synchronizing slots from primary to standby Hayato Kuroda (Fujitsu) <[email protected]>
@ 2023-06-30 06:09 ` Drouvot, Bertrand <[email protected]>
2 siblings, 0 replies; 59+ messages in thread
From: Drouvot, Bertrand @ 2023-06-30 06:09 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Amit Kapila <[email protected]>
Hi Kuroda-san,
On 6/29/23 12:22 PM, Hayato Kuroda (Fujitsu) wrote:
> Dear Drouvot,
>
> Hi, I'm also interested in the feature. Followings are my high-level comments.
> I did not mention some detailed notations because this patch is not at the stage.
> And very sorry that I could not follow all of this discussions.
>
Thanks for looking at it and your feedback!
All I've done so far is to provide a re-based version in April of the existing patch.
I'll have a closer look at the code, at your feedback and Amit's one while working on the new version that will:
- take care of slot invalidation
- ensure that synchronized slot cant' be consumed until the standby gets promoted
as discussed up-thread.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: Synchronizing slots from primary to standby
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Re: Synchronizing slots from primary to standby Andres Freund <[email protected]>
2022-02-08 14:57 ` Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Re: Synchronizing slots from primary to standby Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Re: Synchronizing slots from primary to standby Peter Eisentraut <[email protected]>
2022-11-15 09:02 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Re: Synchronizing slots from primary to standby Drouvot, Bertrand <[email protected]>
2023-06-29 10:22 ` RE: Synchronizing slots from primary to standby Hayato Kuroda (Fujitsu) <[email protected]>
@ 2023-07-01 01:42 ` Amit Kapila <[email protected]>
2 siblings, 0 replies; 59+ messages in thread
From: Amit Kapila @ 2023-07-01 01:42 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: Drouvot, Bertrand <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>
On Thu, Jun 29, 2023 at 3:52 PM Hayato Kuroda (Fujitsu)
<[email protected]> wrote:
>
> 2. I want to confirm the reason why new replication command is added.
>
Are you referring LIST_SLOTS command? If so, I don't think that is
required and instead, we can use a query to fetch the required
information.
> IIUC the
> launcher connects to primary by using primary_conninfo connection string, but
> it establishes the physical replication connection so that any SQL cannot be executed.
> Is it right? Another approach not to use is to specify the target database via
> GUC, whereas not smart. How do you think?
> 3. You chose the per-db worker approach, however, it is difficult to extend the
> feature to support physical slots. This may be problematic. Was there any
> reasons for that? I doubted ReplicationSlotCreate() or advance functions might
> not be used from other databases and these may be reasons, but not sure.
> If these operations can do without connecting to specific database, I think
> the architecture can be changed.
>
I think this point needs some investigation but do we want just one
worker that syncs all the slots? That may lead to lag in keeping the
slots up-to-date. We probably need some tests.
> 4. Currently the launcher establishes the connection every time. Isn't it better
> to reuse the same one instead?
>
I feel it is not the launcher but a separate sync slot worker that
establishes the connection. It is not clear to me what exactly you
have in mind. Can you please explain a bit more?
> Following comments are assumed the configuration, maybe the straightfoward:
>
> primary->standby
> |->subscriber
>
> 5. After constructing the system, I dropped the subscription on the subscriber.
> In this case the logical slot on primary was removed, but that was not replicated
> to standby server. Did you support the workload or not?
>
This should work.
>
> 6. Current approach may delay the startpoint of sync.
>
> Assuming that physical replication system is created first, and then the
> subscriber connects to the publisher node. In this case the launcher connects to
> primary earlier than the apply worker, and reads the slot. At that time there are
> no slots on primary, so launcher disconnects from primary and waits a time period (up to 3min).
> Even if the apply worker creates the slot on publisher, but the launcher on standby
> cannot notice that. The synchronization may start 3 min later.
>
I feel this should be based on some GUC like
'wal_retrieve_retry_interval' which we are already using in the
launcher or probably a new one if that doesn't seem to match.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 59+ messages in thread
* [PATCH v7 11/13] Separate TBM[Shared]Iterator and TBMIterateResult
@ 2024-02-16 02:23 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Melanie Plageman @ 2024-02-16 02:23 UTC (permalink / raw)
Remove the TBMIterateResult from the TBMIterator and TBMSharedIterator
and have tbm_[shared_]iterate() take a TBMIterateResult as a parameter.
This will allow multiple TBMIterateResults to exist concurrently
allowing asynchronous use of the TIDBitmap for prefetching, for example.
tbm_[shared]_iterate() now sets blockno to InvalidBlockNumber when the
bitmap is exhausted instead of returning NULL.
BitmapHeapScan callers of tbm_iterate make a TBMIterateResult locally
and pass it in.
Because GIN only needs a single TBMIterateResult, inline the matchResult
in the GinScanEntry to avoid having to separately manage memory for the
TBMIterateResult.
---
src/backend/access/gin/ginget.c | 48 +++++++++------
src/backend/access/gin/ginscan.c | 2 +-
src/backend/access/heap/heapam_handler.c | 32 +++++-----
src/backend/executor/nodeBitmapHeapscan.c | 38 ++++++------
src/backend/nodes/tidbitmap.c | 73 ++++++++++++-----------
src/include/access/gin_private.h | 2 +-
src/include/nodes/tidbitmap.h | 4 +-
7 files changed, 108 insertions(+), 91 deletions(-)
diff --git a/src/backend/access/gin/ginget.c b/src/backend/access/gin/ginget.c
index 0b4f2ebadb6..3aa457a29e1 100644
--- a/src/backend/access/gin/ginget.c
+++ b/src/backend/access/gin/ginget.c
@@ -332,10 +332,22 @@ restartScanEntry:
entry->list = NULL;
entry->nlist = 0;
entry->matchBitmap = NULL;
- entry->matchResult = NULL;
entry->reduceResult = false;
entry->predictNumberResult = 0;
+ /*
+ * MTODO: is it enough to set blockno to InvalidBlockNumber? In all the
+ * places were we previously set matchResult to NULL, I just set blockno
+ * to InvalidBlockNumber. It seems like this should be okay because that
+ * is usually what we check before using the matchResult members. But it
+ * might be safer to zero out the offsets array. But that is expensive.
+ */
+ entry->matchResult.blockno = InvalidBlockNumber;
+ entry->matchResult.ntuples = 0;
+ entry->matchResult.recheck = true;
+ memset(entry->matchResult.offsets, 0,
+ sizeof(OffsetNumber) * MaxHeapTuplesPerPage);
+
/*
* we should find entry, and begin scan of posting tree or just store
* posting list in memory
@@ -374,6 +386,7 @@ restartScanEntry:
{
if (entry->matchIterator)
tbm_end_iterate(entry->matchIterator);
+ entry->matchResult.blockno = InvalidBlockNumber;
entry->matchIterator = NULL;
tbm_free(entry->matchBitmap);
entry->matchBitmap = NULL;
@@ -823,18 +836,19 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
{
/*
* If we've exhausted all items on this block, move to next block
- * in the bitmap.
+ * in the bitmap. tbm_iterate() sets matchResult->blockno to
+ * InvalidBlockNumber when the bitmap is exhausted.
*/
- while (entry->matchResult == NULL ||
- (entry->matchResult->ntuples >= 0 &&
- entry->offset >= entry->matchResult->ntuples) ||
- entry->matchResult->blockno < advancePastBlk ||
+ while ((!BlockNumberIsValid(entry->matchResult.blockno)) ||
+ (entry->matchResult.ntuples >= 0 &&
+ entry->offset >= entry->matchResult.ntuples) ||
+ entry->matchResult.blockno < advancePastBlk ||
(ItemPointerIsLossyPage(&advancePast) &&
- entry->matchResult->blockno == advancePastBlk))
+ entry->matchResult.blockno == advancePastBlk))
{
- entry->matchResult = tbm_iterate(entry->matchIterator);
+ tbm_iterate(entry->matchIterator, &entry->matchResult);
- if (entry->matchResult == NULL)
+ if (!BlockNumberIsValid(entry->matchResult.blockno))
{
ItemPointerSetInvalid(&entry->curItem);
tbm_end_iterate(entry->matchIterator);
@@ -858,10 +872,10 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* We're now on the first page after advancePast which has any
* items on it. If it's a lossy result, return that.
*/
- if (entry->matchResult->ntuples < 0)
+ if (entry->matchResult.ntuples < 0)
{
ItemPointerSetLossyPage(&entry->curItem,
- entry->matchResult->blockno);
+ entry->matchResult.blockno);
/*
* We might as well fall out of the loop; we could not
@@ -875,27 +889,27 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* Not a lossy page. Skip over any offsets <= advancePast, and
* return that.
*/
- if (entry->matchResult->blockno == advancePastBlk)
+ if (entry->matchResult.blockno == advancePastBlk)
{
/*
* First, do a quick check against the last offset on the
* page. If that's > advancePast, so are all the other
* offsets, so just go back to the top to get the next page.
*/
- if (entry->matchResult->offsets[entry->matchResult->ntuples - 1] <= advancePastOff)
+ if (entry->matchResult.offsets[entry->matchResult.ntuples - 1] <= advancePastOff)
{
- entry->offset = entry->matchResult->ntuples;
+ entry->offset = entry->matchResult.ntuples;
continue;
}
/* Otherwise scan to find the first item > advancePast */
- while (entry->matchResult->offsets[entry->offset] <= advancePastOff)
+ while (entry->matchResult.offsets[entry->offset] <= advancePastOff)
entry->offset++;
}
ItemPointerSet(&entry->curItem,
- entry->matchResult->blockno,
- entry->matchResult->offsets[entry->offset]);
+ entry->matchResult.blockno,
+ entry->matchResult.offsets[entry->offset]);
entry->offset++;
/* Done unless we need to reduce the result */
diff --git a/src/backend/access/gin/ginscan.c b/src/backend/access/gin/ginscan.c
index af24d38544e..033d5253394 100644
--- a/src/backend/access/gin/ginscan.c
+++ b/src/backend/access/gin/ginscan.c
@@ -106,7 +106,7 @@ ginFillScanEntry(GinScanOpaque so, OffsetNumber attnum,
ItemPointerSetMin(&scanEntry->curItem);
scanEntry->matchBitmap = NULL;
scanEntry->matchIterator = NULL;
- scanEntry->matchResult = NULL;
+ scanEntry->matchResult.blockno = InvalidBlockNumber;
scanEntry->list = NULL;
scanEntry->nlist = 0;
scanEntry->offset = InvalidOffsetNumber;
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index e038e60cd8f..022753e203a 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2119,7 +2119,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Buffer buffer;
Snapshot snapshot;
int ntup;
- TBMIterateResult *tbmres;
+ TBMIterateResult tbmres;
hscan->rs_cindex = 0;
hscan->rs_ntuples = 0;
@@ -2132,11 +2132,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
CHECK_FOR_INTERRUPTS();
if (scan->shared_tbmiterator)
- tbmres = tbm_shared_iterate(scan->shared_tbmiterator);
+ tbm_shared_iterate(scan->shared_tbmiterator, &tbmres);
else
- tbmres = tbm_iterate(scan->tbmiterator);
+ tbm_iterate(scan->tbmiterator, &tbmres);
- if (tbmres == NULL)
+ if (!BlockNumberIsValid(tbmres.blockno))
{
/* no more entries in the bitmap */
Assert(hscan->rs_empty_tuples_pending == 0);
@@ -2151,11 +2151,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* isolation though, as we need to examine all invisible tuples
* reachable by the index.
*/
- } while (!IsolationIsSerializable() && tbmres->blockno >= hscan->rs_nblocks);
+ } while (!IsolationIsSerializable() && tbmres.blockno >= hscan->rs_nblocks);
/* Got a valid block */
- *blockno = tbmres->blockno;
- *recheck = tbmres->recheck;
+ *blockno = tbmres.blockno;
+ *recheck = tbmres.recheck;
/*
* We can skip fetching the heap page if we don't need any fields from the
@@ -2163,19 +2163,19 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* the page are visible to our transaction.
*/
if (scan->rs_flags & SO_CAN_SKIP_FETCH &&
- !tbmres->recheck &&
- VM_ALL_VISIBLE(scan->rs_rd, tbmres->blockno, &hscan->rs_vmbuffer))
+ !tbmres.recheck &&
+ VM_ALL_VISIBLE(scan->rs_rd, tbmres.blockno, &hscan->rs_vmbuffer))
{
/* can't be lossy in the skip_fetch case */
- Assert(tbmres->ntuples >= 0);
+ Assert(tbmres.ntuples >= 0);
Assert(hscan->rs_empty_tuples_pending >= 0);
- hscan->rs_empty_tuples_pending += tbmres->ntuples;
+ hscan->rs_empty_tuples_pending += tbmres.ntuples;
return true;
}
- block = tbmres->blockno;
+ block = tbmres.blockno;
/*
* Acquire pin on the target heap page, trading in any pin we held before.
@@ -2204,7 +2204,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
/*
* We need two separate strategies for lossy and non-lossy cases.
*/
- if (tbmres->ntuples >= 0)
+ if (tbmres.ntuples >= 0)
{
/*
* Bitmap is non-lossy, so we just look through the offsets listed in
@@ -2213,9 +2213,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
*/
int curslot;
- for (curslot = 0; curslot < tbmres->ntuples; curslot++)
+ for (curslot = 0; curslot < tbmres.ntuples; curslot++)
{
- OffsetNumber offnum = tbmres->offsets[curslot];
+ OffsetNumber offnum = tbmres.offsets[curslot];
ItemPointerData tid;
HeapTupleData heapTuple;
@@ -2265,7 +2265,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
- *lossy = tbmres->ntuples < 0;
+ *lossy = tbmres.ntuples < 0;
/*
* Return true to indicate that a valid block was found and the bitmap is
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index b1dfa582c7d..43df0ac9964 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -321,7 +321,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
{
#ifdef USE_PREFETCH
ParallelBitmapHeapState *pstate = node->pstate;
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
if (pstate == NULL)
{
@@ -335,8 +335,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
else if (prefetch_iterator)
{
/* Do not let the prefetch iterator get behind the main one */
- tbmpre = tbm_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
return;
}
@@ -371,8 +371,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
*/
if (prefetch_iterator)
{
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
}
}
@@ -440,10 +440,12 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (node->prefetch_pages < node->prefetch_target)
{
- TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator);
+ TBMIterateResult tbmpre;
bool skip_fetch;
- if (tbmpre == NULL)
+ tbm_iterate(prefetch_iterator, &tbmpre);
+
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_iterate(prefetch_iterator);
@@ -451,7 +453,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
node->prefetch_pages++;
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/*
* If we expect not to have to actually read this heap page,
@@ -460,13 +462,13 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
* prefetch_pages?)
*/
skip_fetch = (scan->rs_flags & SO_CAN_SKIP_FETCH &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
@@ -481,7 +483,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (1)
{
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
bool do_prefetch = false;
bool skip_fetch;
@@ -500,8 +502,8 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
if (!do_prefetch)
return;
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- if (tbmpre == NULL)
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_shared_iterate(prefetch_iterator);
@@ -509,17 +511,17 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/* As above, skip prefetch if we expect not to need page */
skip_fetch = (scan->rs_flags & SO_CAN_SKIP_FETCH &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
}
diff --git a/src/backend/nodes/tidbitmap.c b/src/backend/nodes/tidbitmap.c
index d2bf8f44d50..7d038c2018d 100644
--- a/src/backend/nodes/tidbitmap.c
+++ b/src/backend/nodes/tidbitmap.c
@@ -172,7 +172,6 @@ struct TBMIterator
int spageptr; /* next spages index */
int schunkptr; /* next schunks index */
int schunkbit; /* next bit to check in current schunk */
- TBMIterateResult output;
};
/*
@@ -213,7 +212,6 @@ struct TBMSharedIterator
PTEntryArray *ptbase; /* pagetable element array */
PTIterationArray *ptpages; /* sorted exact page index list */
PTIterationArray *ptchunks; /* sorted lossy page index list */
- TBMIterateResult output;
};
/* Local function prototypes */
@@ -944,20 +942,21 @@ tbm_advance_schunkbit(PagetableEntry *chunk, int *schunkbitp)
/*
* tbm_iterate - scan through next page of a TIDBitmap
*
- * Returns a TBMIterateResult representing one page, or NULL if there are
- * no more pages to scan. Pages are guaranteed to be delivered in numerical
- * order. If result->ntuples < 0, then the bitmap is "lossy" and failed to
- * remember the exact tuples to look at on this page --- the caller must
- * examine all tuples on the page and check if they meet the intended
- * condition. If result->recheck is true, only the indicated tuples need
- * be examined, but the condition must be rechecked anyway. (For ease of
- * testing, recheck is always set true when ntuples < 0.)
+ * Caller must pass in a TBMIterateResult to be filled.
+ *
+ * Pages are guaranteed to be delivered in numerical order. tbmres->blockno is
+ * set to InvalidBlockNumber when there are no more pages to scan. If
+ * tbmres->ntuples < 0, then the bitmap is "lossy" and failed to remember the
+ * exact tuples to look at on this page --- the caller must examine all tuples
+ * on the page and check if they meet the intended condition. If
+ * tbmres->recheck is true, only the indicated tuples need be examined, but the
+ * condition must be rechecked anyway. (For ease of testing, recheck is always
+ * set true when ntuples < 0.)
*/
-TBMIterateResult *
-tbm_iterate(TBMIterator *iterator)
+void
+tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres)
{
TIDBitmap *tbm = iterator->tbm;
- TBMIterateResult *output = &(iterator->output);
Assert(tbm->iterating == TBM_ITERATING_PRIVATE);
@@ -985,6 +984,7 @@ tbm_iterate(TBMIterator *iterator)
* If both chunk and per-page data remain, must output the numerically
* earlier page.
*/
+ Assert(tbmres);
if (iterator->schunkptr < tbm->nchunks)
{
PagetableEntry *chunk = tbm->schunks[iterator->schunkptr];
@@ -995,11 +995,11 @@ tbm_iterate(TBMIterator *iterator)
chunk_blockno < tbm->spages[iterator->spageptr]->blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
iterator->schunkbit++;
- return output;
+ return;
}
}
@@ -1015,16 +1015,17 @@ tbm_iterate(TBMIterator *iterator)
page = tbm->spages[iterator->spageptr];
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
iterator->spageptr++;
- return output;
+ return;
}
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
@@ -1034,10 +1035,9 @@ tbm_iterate(TBMIterator *iterator)
* across multiple processes. We need to acquire the iterator LWLock,
* before accessing the shared members.
*/
-TBMIterateResult *
-tbm_shared_iterate(TBMSharedIterator *iterator)
+void
+tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres)
{
- TBMIterateResult *output = &iterator->output;
TBMSharedIteratorState *istate = iterator->state;
PagetableEntry *ptbase = NULL;
int *idxpages = NULL;
@@ -1088,13 +1088,13 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
chunk_blockno < ptbase[idxpages[istate->spageptr]].blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
istate->schunkbit++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
}
@@ -1104,21 +1104,22 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
int ntuples;
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
istate->spageptr++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
LWLockRelease(&istate->lock);
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 3013a44bae1..3b432263bb0 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -353,7 +353,7 @@ typedef struct GinScanEntryData
/* for a partial-match or full-scan query, we accumulate all TIDs here */
TIDBitmap *matchBitmap;
TBMIterator *matchIterator;
- TBMIterateResult *matchResult;
+ TBMIterateResult matchResult;
/* used for Posting list and one page in Posting tree */
ItemPointerData *list;
diff --git a/src/include/nodes/tidbitmap.h b/src/include/nodes/tidbitmap.h
index 432fae52962..f000c1af28f 100644
--- a/src/include/nodes/tidbitmap.h
+++ b/src/include/nodes/tidbitmap.h
@@ -72,8 +72,8 @@ extern bool tbm_is_empty(const TIDBitmap *tbm);
extern TBMIterator *tbm_begin_iterate(TIDBitmap *tbm);
extern dsa_pointer tbm_prepare_shared_iterate(TIDBitmap *tbm);
-extern TBMIterateResult *tbm_iterate(TBMIterator *iterator);
-extern TBMIterateResult *tbm_shared_iterate(TBMSharedIterator *iterator);
+extern void tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres);
+extern void tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres);
extern void tbm_end_iterate(TBMIterator *iterator);
extern void tbm_end_shared_iterate(TBMSharedIterator *iterator);
extern TBMSharedIterator *tbm_attach_shared_iterate(dsa_area *dsa,
--
2.40.1
--kqqpqghcwbcc3dt5
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0012-v7-Streaming-Read-API.patch"
^ permalink raw reply [nested|flat] 59+ messages in thread
* [PATCH v8 12/17] Separate TBM[Shared]Iterator and TBMIterateResult
@ 2024-02-16 02:23 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Melanie Plageman @ 2024-02-16 02:23 UTC (permalink / raw)
Remove the TBMIterateResult from the TBMIterator and TBMSharedIterator
and have tbm_[shared_]iterate() take a TBMIterateResult as a parameter.
This will allow multiple TBMIterateResults to exist concurrently
allowing asynchronous use of the TIDBitmap for prefetching, for example.
tbm_[shared]_iterate() now sets blockno to InvalidBlockNumber when the
bitmap is exhausted instead of returning NULL.
BitmapHeapScan callers of tbm_iterate make a TBMIterateResult locally
and pass it in.
Because GIN only needs a single TBMIterateResult, inline the matchResult
in the GinScanEntry to avoid having to separately manage memory for the
TBMIterateResult.
---
src/backend/access/gin/ginget.c | 48 +++++++++------
src/backend/access/gin/ginscan.c | 2 +-
src/backend/access/heap/heapam_handler.c | 32 +++++-----
src/backend/executor/nodeBitmapHeapscan.c | 38 ++++++------
src/backend/nodes/tidbitmap.c | 73 ++++++++++++-----------
src/include/access/gin_private.h | 2 +-
src/include/nodes/tidbitmap.h | 4 +-
7 files changed, 108 insertions(+), 91 deletions(-)
diff --git a/src/backend/access/gin/ginget.c b/src/backend/access/gin/ginget.c
index 0b4f2ebadb..3aa457a29e 100644
--- a/src/backend/access/gin/ginget.c
+++ b/src/backend/access/gin/ginget.c
@@ -332,10 +332,22 @@ restartScanEntry:
entry->list = NULL;
entry->nlist = 0;
entry->matchBitmap = NULL;
- entry->matchResult = NULL;
entry->reduceResult = false;
entry->predictNumberResult = 0;
+ /*
+ * MTODO: is it enough to set blockno to InvalidBlockNumber? In all the
+ * places were we previously set matchResult to NULL, I just set blockno
+ * to InvalidBlockNumber. It seems like this should be okay because that
+ * is usually what we check before using the matchResult members. But it
+ * might be safer to zero out the offsets array. But that is expensive.
+ */
+ entry->matchResult.blockno = InvalidBlockNumber;
+ entry->matchResult.ntuples = 0;
+ entry->matchResult.recheck = true;
+ memset(entry->matchResult.offsets, 0,
+ sizeof(OffsetNumber) * MaxHeapTuplesPerPage);
+
/*
* we should find entry, and begin scan of posting tree or just store
* posting list in memory
@@ -374,6 +386,7 @@ restartScanEntry:
{
if (entry->matchIterator)
tbm_end_iterate(entry->matchIterator);
+ entry->matchResult.blockno = InvalidBlockNumber;
entry->matchIterator = NULL;
tbm_free(entry->matchBitmap);
entry->matchBitmap = NULL;
@@ -823,18 +836,19 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
{
/*
* If we've exhausted all items on this block, move to next block
- * in the bitmap.
+ * in the bitmap. tbm_iterate() sets matchResult->blockno to
+ * InvalidBlockNumber when the bitmap is exhausted.
*/
- while (entry->matchResult == NULL ||
- (entry->matchResult->ntuples >= 0 &&
- entry->offset >= entry->matchResult->ntuples) ||
- entry->matchResult->blockno < advancePastBlk ||
+ while ((!BlockNumberIsValid(entry->matchResult.blockno)) ||
+ (entry->matchResult.ntuples >= 0 &&
+ entry->offset >= entry->matchResult.ntuples) ||
+ entry->matchResult.blockno < advancePastBlk ||
(ItemPointerIsLossyPage(&advancePast) &&
- entry->matchResult->blockno == advancePastBlk))
+ entry->matchResult.blockno == advancePastBlk))
{
- entry->matchResult = tbm_iterate(entry->matchIterator);
+ tbm_iterate(entry->matchIterator, &entry->matchResult);
- if (entry->matchResult == NULL)
+ if (!BlockNumberIsValid(entry->matchResult.blockno))
{
ItemPointerSetInvalid(&entry->curItem);
tbm_end_iterate(entry->matchIterator);
@@ -858,10 +872,10 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* We're now on the first page after advancePast which has any
* items on it. If it's a lossy result, return that.
*/
- if (entry->matchResult->ntuples < 0)
+ if (entry->matchResult.ntuples < 0)
{
ItemPointerSetLossyPage(&entry->curItem,
- entry->matchResult->blockno);
+ entry->matchResult.blockno);
/*
* We might as well fall out of the loop; we could not
@@ -875,27 +889,27 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* Not a lossy page. Skip over any offsets <= advancePast, and
* return that.
*/
- if (entry->matchResult->blockno == advancePastBlk)
+ if (entry->matchResult.blockno == advancePastBlk)
{
/*
* First, do a quick check against the last offset on the
* page. If that's > advancePast, so are all the other
* offsets, so just go back to the top to get the next page.
*/
- if (entry->matchResult->offsets[entry->matchResult->ntuples - 1] <= advancePastOff)
+ if (entry->matchResult.offsets[entry->matchResult.ntuples - 1] <= advancePastOff)
{
- entry->offset = entry->matchResult->ntuples;
+ entry->offset = entry->matchResult.ntuples;
continue;
}
/* Otherwise scan to find the first item > advancePast */
- while (entry->matchResult->offsets[entry->offset] <= advancePastOff)
+ while (entry->matchResult.offsets[entry->offset] <= advancePastOff)
entry->offset++;
}
ItemPointerSet(&entry->curItem,
- entry->matchResult->blockno,
- entry->matchResult->offsets[entry->offset]);
+ entry->matchResult.blockno,
+ entry->matchResult.offsets[entry->offset]);
entry->offset++;
/* Done unless we need to reduce the result */
diff --git a/src/backend/access/gin/ginscan.c b/src/backend/access/gin/ginscan.c
index af24d38544..033d525339 100644
--- a/src/backend/access/gin/ginscan.c
+++ b/src/backend/access/gin/ginscan.c
@@ -106,7 +106,7 @@ ginFillScanEntry(GinScanOpaque so, OffsetNumber attnum,
ItemPointerSetMin(&scanEntry->curItem);
scanEntry->matchBitmap = NULL;
scanEntry->matchIterator = NULL;
- scanEntry->matchResult = NULL;
+ scanEntry->matchResult.blockno = InvalidBlockNumber;
scanEntry->list = NULL;
scanEntry->nlist = 0;
scanEntry->offset = InvalidOffsetNumber;
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 266b34fe6b..adfc77684a 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2122,7 +2122,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Buffer buffer;
Snapshot snapshot;
int ntup;
- TBMIterateResult *tbmres;
+ TBMIterateResult tbmres;
hscan->rs_cindex = 0;
hscan->rs_ntuples = 0;
@@ -2135,11 +2135,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
CHECK_FOR_INTERRUPTS();
if (scan->shared_tbmiterator)
- tbmres = tbm_shared_iterate(scan->shared_tbmiterator);
+ tbm_shared_iterate(scan->shared_tbmiterator, &tbmres);
else
- tbmres = tbm_iterate(scan->tbmiterator);
+ tbm_iterate(scan->tbmiterator, &tbmres);
- if (tbmres == NULL)
+ if (!BlockNumberIsValid(tbmres.blockno))
{
/* no more entries in the bitmap */
Assert(hscan->rs_empty_tuples_pending == 0);
@@ -2154,11 +2154,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* isolation though, as we need to examine all invisible tuples
* reachable by the index.
*/
- } while (!IsolationIsSerializable() && tbmres->blockno >= hscan->rs_nblocks);
+ } while (!IsolationIsSerializable() && tbmres.blockno >= hscan->rs_nblocks);
/* Got a valid block */
- *blockno = tbmres->blockno;
- *recheck = tbmres->recheck;
+ *blockno = tbmres.blockno;
+ *recheck = tbmres.recheck;
/*
* We can skip fetching the heap page if we don't need any fields from the
@@ -2166,19 +2166,19 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* page are visible to our transaction.
*/
if (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmres->recheck &&
- VM_ALL_VISIBLE(scan->rs_rd, tbmres->blockno, &hscan->rs_vmbuffer))
+ !tbmres.recheck &&
+ VM_ALL_VISIBLE(scan->rs_rd, tbmres.blockno, &hscan->rs_vmbuffer))
{
/* can't be lossy in the skip_fetch case */
- Assert(tbmres->ntuples >= 0);
+ Assert(tbmres.ntuples >= 0);
Assert(hscan->rs_empty_tuples_pending >= 0);
- hscan->rs_empty_tuples_pending += tbmres->ntuples;
+ hscan->rs_empty_tuples_pending += tbmres.ntuples;
return true;
}
- block = tbmres->blockno;
+ block = tbmres.blockno;
/*
* Acquire pin on the target heap page, trading in any pin we held before.
@@ -2207,7 +2207,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
/*
* We need two separate strategies for lossy and non-lossy cases.
*/
- if (tbmres->ntuples >= 0)
+ if (tbmres.ntuples >= 0)
{
/*
* Bitmap is non-lossy, so we just look through the offsets listed in
@@ -2216,9 +2216,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
*/
int curslot;
- for (curslot = 0; curslot < tbmres->ntuples; curslot++)
+ for (curslot = 0; curslot < tbmres.ntuples; curslot++)
{
- OffsetNumber offnum = tbmres->offsets[curslot];
+ OffsetNumber offnum = tbmres.offsets[curslot];
ItemPointerData tid;
HeapTupleData heapTuple;
@@ -2268,7 +2268,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
- if (tbmres->ntuples < 0)
+ if (tbmres.ntuples < 0)
(*lossy_pages)++;
else
(*exact_pages)++;
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 96b55507a3..51c4360205 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -316,7 +316,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
{
#ifdef USE_PREFETCH
ParallelBitmapHeapState *pstate = node->pstate;
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
if (pstate == NULL)
{
@@ -330,8 +330,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
else if (prefetch_iterator)
{
/* Do not let the prefetch iterator get behind the main one */
- tbmpre = tbm_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
return;
}
@@ -366,8 +366,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
*/
if (prefetch_iterator)
{
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
}
}
@@ -435,10 +435,12 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (node->prefetch_pages < node->prefetch_target)
{
- TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator);
+ TBMIterateResult tbmpre;
bool skip_fetch;
- if (tbmpre == NULL)
+ tbm_iterate(prefetch_iterator, &tbmpre);
+
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_iterate(prefetch_iterator);
@@ -446,7 +448,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
node->prefetch_pages++;
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/*
* If we expect not to have to actually read this heap page,
@@ -455,13 +457,13 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
* prefetch_pages?)
*/
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
@@ -476,7 +478,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (1)
{
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
bool do_prefetch = false;
bool skip_fetch;
@@ -495,8 +497,8 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
if (!do_prefetch)
return;
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- if (tbmpre == NULL)
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_shared_iterate(prefetch_iterator);
@@ -504,17 +506,17 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/* As above, skip prefetch if we expect not to need page */
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
}
diff --git a/src/backend/nodes/tidbitmap.c b/src/backend/nodes/tidbitmap.c
index 1dc4c99bf9..309a44bdb8 100644
--- a/src/backend/nodes/tidbitmap.c
+++ b/src/backend/nodes/tidbitmap.c
@@ -172,7 +172,6 @@ struct TBMIterator
int spageptr; /* next spages index */
int schunkptr; /* next schunks index */
int schunkbit; /* next bit to check in current schunk */
- TBMIterateResult output;
};
/*
@@ -213,7 +212,6 @@ struct TBMSharedIterator
PTEntryArray *ptbase; /* pagetable element array */
PTIterationArray *ptpages; /* sorted exact page index list */
PTIterationArray *ptchunks; /* sorted lossy page index list */
- TBMIterateResult output;
};
/* Local function prototypes */
@@ -944,20 +942,21 @@ tbm_advance_schunkbit(PagetableEntry *chunk, int *schunkbitp)
/*
* tbm_iterate - scan through next page of a TIDBitmap
*
- * Returns a TBMIterateResult representing one page, or NULL if there are
- * no more pages to scan. Pages are guaranteed to be delivered in numerical
- * order. If result->ntuples < 0, then the bitmap is "lossy" and failed to
- * remember the exact tuples to look at on this page --- the caller must
- * examine all tuples on the page and check if they meet the intended
- * condition. If result->recheck is true, only the indicated tuples need
- * be examined, but the condition must be rechecked anyway. (For ease of
- * testing, recheck is always set true when ntuples < 0.)
+ * Caller must pass in a TBMIterateResult to be filled.
+ *
+ * Pages are guaranteed to be delivered in numerical order. tbmres->blockno is
+ * set to InvalidBlockNumber when there are no more pages to scan. If
+ * tbmres->ntuples < 0, then the bitmap is "lossy" and failed to remember the
+ * exact tuples to look at on this page --- the caller must examine all tuples
+ * on the page and check if they meet the intended condition. If
+ * tbmres->recheck is true, only the indicated tuples need be examined, but the
+ * condition must be rechecked anyway. (For ease of testing, recheck is always
+ * set true when ntuples < 0.)
*/
-TBMIterateResult *
-tbm_iterate(TBMIterator *iterator)
+void
+tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres)
{
TIDBitmap *tbm = iterator->tbm;
- TBMIterateResult *output = &(iterator->output);
Assert(tbm->iterating == TBM_ITERATING_PRIVATE);
@@ -985,6 +984,7 @@ tbm_iterate(TBMIterator *iterator)
* If both chunk and per-page data remain, must output the numerically
* earlier page.
*/
+ Assert(tbmres);
if (iterator->schunkptr < tbm->nchunks)
{
PagetableEntry *chunk = tbm->schunks[iterator->schunkptr];
@@ -995,11 +995,11 @@ tbm_iterate(TBMIterator *iterator)
chunk_blockno < tbm->spages[iterator->spageptr]->blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
iterator->schunkbit++;
- return output;
+ return;
}
}
@@ -1015,16 +1015,17 @@ tbm_iterate(TBMIterator *iterator)
page = tbm->spages[iterator->spageptr];
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
iterator->spageptr++;
- return output;
+ return;
}
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
@@ -1034,10 +1035,9 @@ tbm_iterate(TBMIterator *iterator)
* across multiple processes. We need to acquire the iterator LWLock,
* before accessing the shared members.
*/
-TBMIterateResult *
-tbm_shared_iterate(TBMSharedIterator *iterator)
+void
+tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres)
{
- TBMIterateResult *output = &iterator->output;
TBMSharedIteratorState *istate = iterator->state;
PagetableEntry *ptbase = NULL;
int *idxpages = NULL;
@@ -1088,13 +1088,13 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
chunk_blockno < ptbase[idxpages[istate->spageptr]].blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
istate->schunkbit++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
}
@@ -1104,21 +1104,22 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
int ntuples;
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
istate->spageptr++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
LWLockRelease(&istate->lock);
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 3013a44bae..3b432263bb 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -353,7 +353,7 @@ typedef struct GinScanEntryData
/* for a partial-match or full-scan query, we accumulate all TIDs here */
TIDBitmap *matchBitmap;
TBMIterator *matchIterator;
- TBMIterateResult *matchResult;
+ TBMIterateResult matchResult;
/* used for Posting list and one page in Posting tree */
ItemPointerData *list;
diff --git a/src/include/nodes/tidbitmap.h b/src/include/nodes/tidbitmap.h
index 432fae5296..f000c1af28 100644
--- a/src/include/nodes/tidbitmap.h
+++ b/src/include/nodes/tidbitmap.h
@@ -72,8 +72,8 @@ extern bool tbm_is_empty(const TIDBitmap *tbm);
extern TBMIterator *tbm_begin_iterate(TIDBitmap *tbm);
extern dsa_pointer tbm_prepare_shared_iterate(TIDBitmap *tbm);
-extern TBMIterateResult *tbm_iterate(TBMIterator *iterator);
-extern TBMIterateResult *tbm_shared_iterate(TBMSharedIterator *iterator);
+extern void tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres);
+extern void tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres);
extern void tbm_end_iterate(TBMIterator *iterator);
extern void tbm_end_shared_iterate(TBMSharedIterator *iterator);
extern TBMSharedIterator *tbm_attach_shared_iterate(dsa_area *dsa,
--
2.40.1
--xqq4defy3uncu6k6
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v8-0013-Push-BitmapHeapScan-prefetch-code-into-heapam.c.patch"
^ permalink raw reply [nested|flat] 59+ messages in thread
* [PATCH v7 11/13] Separate TBM[Shared]Iterator and TBMIterateResult
@ 2024-02-16 02:23 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Melanie Plageman @ 2024-02-16 02:23 UTC (permalink / raw)
Remove the TBMIterateResult from the TBMIterator and TBMSharedIterator
and have tbm_[shared_]iterate() take a TBMIterateResult as a parameter.
This will allow multiple TBMIterateResults to exist concurrently
allowing asynchronous use of the TIDBitmap for prefetching, for example.
tbm_[shared]_iterate() now sets blockno to InvalidBlockNumber when the
bitmap is exhausted instead of returning NULL.
BitmapHeapScan callers of tbm_iterate make a TBMIterateResult locally
and pass it in.
Because GIN only needs a single TBMIterateResult, inline the matchResult
in the GinScanEntry to avoid having to separately manage memory for the
TBMIterateResult.
---
src/backend/access/gin/ginget.c | 48 +++++++++------
src/backend/access/gin/ginscan.c | 2 +-
src/backend/access/heap/heapam_handler.c | 32 +++++-----
src/backend/executor/nodeBitmapHeapscan.c | 38 ++++++------
src/backend/nodes/tidbitmap.c | 73 ++++++++++++-----------
src/include/access/gin_private.h | 2 +-
src/include/nodes/tidbitmap.h | 4 +-
7 files changed, 108 insertions(+), 91 deletions(-)
diff --git a/src/backend/access/gin/ginget.c b/src/backend/access/gin/ginget.c
index 0b4f2ebadb6..3aa457a29e1 100644
--- a/src/backend/access/gin/ginget.c
+++ b/src/backend/access/gin/ginget.c
@@ -332,10 +332,22 @@ restartScanEntry:
entry->list = NULL;
entry->nlist = 0;
entry->matchBitmap = NULL;
- entry->matchResult = NULL;
entry->reduceResult = false;
entry->predictNumberResult = 0;
+ /*
+ * MTODO: is it enough to set blockno to InvalidBlockNumber? In all the
+ * places were we previously set matchResult to NULL, I just set blockno
+ * to InvalidBlockNumber. It seems like this should be okay because that
+ * is usually what we check before using the matchResult members. But it
+ * might be safer to zero out the offsets array. But that is expensive.
+ */
+ entry->matchResult.blockno = InvalidBlockNumber;
+ entry->matchResult.ntuples = 0;
+ entry->matchResult.recheck = true;
+ memset(entry->matchResult.offsets, 0,
+ sizeof(OffsetNumber) * MaxHeapTuplesPerPage);
+
/*
* we should find entry, and begin scan of posting tree or just store
* posting list in memory
@@ -374,6 +386,7 @@ restartScanEntry:
{
if (entry->matchIterator)
tbm_end_iterate(entry->matchIterator);
+ entry->matchResult.blockno = InvalidBlockNumber;
entry->matchIterator = NULL;
tbm_free(entry->matchBitmap);
entry->matchBitmap = NULL;
@@ -823,18 +836,19 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
{
/*
* If we've exhausted all items on this block, move to next block
- * in the bitmap.
+ * in the bitmap. tbm_iterate() sets matchResult->blockno to
+ * InvalidBlockNumber when the bitmap is exhausted.
*/
- while (entry->matchResult == NULL ||
- (entry->matchResult->ntuples >= 0 &&
- entry->offset >= entry->matchResult->ntuples) ||
- entry->matchResult->blockno < advancePastBlk ||
+ while ((!BlockNumberIsValid(entry->matchResult.blockno)) ||
+ (entry->matchResult.ntuples >= 0 &&
+ entry->offset >= entry->matchResult.ntuples) ||
+ entry->matchResult.blockno < advancePastBlk ||
(ItemPointerIsLossyPage(&advancePast) &&
- entry->matchResult->blockno == advancePastBlk))
+ entry->matchResult.blockno == advancePastBlk))
{
- entry->matchResult = tbm_iterate(entry->matchIterator);
+ tbm_iterate(entry->matchIterator, &entry->matchResult);
- if (entry->matchResult == NULL)
+ if (!BlockNumberIsValid(entry->matchResult.blockno))
{
ItemPointerSetInvalid(&entry->curItem);
tbm_end_iterate(entry->matchIterator);
@@ -858,10 +872,10 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* We're now on the first page after advancePast which has any
* items on it. If it's a lossy result, return that.
*/
- if (entry->matchResult->ntuples < 0)
+ if (entry->matchResult.ntuples < 0)
{
ItemPointerSetLossyPage(&entry->curItem,
- entry->matchResult->blockno);
+ entry->matchResult.blockno);
/*
* We might as well fall out of the loop; we could not
@@ -875,27 +889,27 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* Not a lossy page. Skip over any offsets <= advancePast, and
* return that.
*/
- if (entry->matchResult->blockno == advancePastBlk)
+ if (entry->matchResult.blockno == advancePastBlk)
{
/*
* First, do a quick check against the last offset on the
* page. If that's > advancePast, so are all the other
* offsets, so just go back to the top to get the next page.
*/
- if (entry->matchResult->offsets[entry->matchResult->ntuples - 1] <= advancePastOff)
+ if (entry->matchResult.offsets[entry->matchResult.ntuples - 1] <= advancePastOff)
{
- entry->offset = entry->matchResult->ntuples;
+ entry->offset = entry->matchResult.ntuples;
continue;
}
/* Otherwise scan to find the first item > advancePast */
- while (entry->matchResult->offsets[entry->offset] <= advancePastOff)
+ while (entry->matchResult.offsets[entry->offset] <= advancePastOff)
entry->offset++;
}
ItemPointerSet(&entry->curItem,
- entry->matchResult->blockno,
- entry->matchResult->offsets[entry->offset]);
+ entry->matchResult.blockno,
+ entry->matchResult.offsets[entry->offset]);
entry->offset++;
/* Done unless we need to reduce the result */
diff --git a/src/backend/access/gin/ginscan.c b/src/backend/access/gin/ginscan.c
index af24d38544e..033d5253394 100644
--- a/src/backend/access/gin/ginscan.c
+++ b/src/backend/access/gin/ginscan.c
@@ -106,7 +106,7 @@ ginFillScanEntry(GinScanOpaque so, OffsetNumber attnum,
ItemPointerSetMin(&scanEntry->curItem);
scanEntry->matchBitmap = NULL;
scanEntry->matchIterator = NULL;
- scanEntry->matchResult = NULL;
+ scanEntry->matchResult.blockno = InvalidBlockNumber;
scanEntry->list = NULL;
scanEntry->nlist = 0;
scanEntry->offset = InvalidOffsetNumber;
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index e038e60cd8f..022753e203a 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2119,7 +2119,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Buffer buffer;
Snapshot snapshot;
int ntup;
- TBMIterateResult *tbmres;
+ TBMIterateResult tbmres;
hscan->rs_cindex = 0;
hscan->rs_ntuples = 0;
@@ -2132,11 +2132,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
CHECK_FOR_INTERRUPTS();
if (scan->shared_tbmiterator)
- tbmres = tbm_shared_iterate(scan->shared_tbmiterator);
+ tbm_shared_iterate(scan->shared_tbmiterator, &tbmres);
else
- tbmres = tbm_iterate(scan->tbmiterator);
+ tbm_iterate(scan->tbmiterator, &tbmres);
- if (tbmres == NULL)
+ if (!BlockNumberIsValid(tbmres.blockno))
{
/* no more entries in the bitmap */
Assert(hscan->rs_empty_tuples_pending == 0);
@@ -2151,11 +2151,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* isolation though, as we need to examine all invisible tuples
* reachable by the index.
*/
- } while (!IsolationIsSerializable() && tbmres->blockno >= hscan->rs_nblocks);
+ } while (!IsolationIsSerializable() && tbmres.blockno >= hscan->rs_nblocks);
/* Got a valid block */
- *blockno = tbmres->blockno;
- *recheck = tbmres->recheck;
+ *blockno = tbmres.blockno;
+ *recheck = tbmres.recheck;
/*
* We can skip fetching the heap page if we don't need any fields from the
@@ -2163,19 +2163,19 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* the page are visible to our transaction.
*/
if (scan->rs_flags & SO_CAN_SKIP_FETCH &&
- !tbmres->recheck &&
- VM_ALL_VISIBLE(scan->rs_rd, tbmres->blockno, &hscan->rs_vmbuffer))
+ !tbmres.recheck &&
+ VM_ALL_VISIBLE(scan->rs_rd, tbmres.blockno, &hscan->rs_vmbuffer))
{
/* can't be lossy in the skip_fetch case */
- Assert(tbmres->ntuples >= 0);
+ Assert(tbmres.ntuples >= 0);
Assert(hscan->rs_empty_tuples_pending >= 0);
- hscan->rs_empty_tuples_pending += tbmres->ntuples;
+ hscan->rs_empty_tuples_pending += tbmres.ntuples;
return true;
}
- block = tbmres->blockno;
+ block = tbmres.blockno;
/*
* Acquire pin on the target heap page, trading in any pin we held before.
@@ -2204,7 +2204,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
/*
* We need two separate strategies for lossy and non-lossy cases.
*/
- if (tbmres->ntuples >= 0)
+ if (tbmres.ntuples >= 0)
{
/*
* Bitmap is non-lossy, so we just look through the offsets listed in
@@ -2213,9 +2213,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
*/
int curslot;
- for (curslot = 0; curslot < tbmres->ntuples; curslot++)
+ for (curslot = 0; curslot < tbmres.ntuples; curslot++)
{
- OffsetNumber offnum = tbmres->offsets[curslot];
+ OffsetNumber offnum = tbmres.offsets[curslot];
ItemPointerData tid;
HeapTupleData heapTuple;
@@ -2265,7 +2265,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
- *lossy = tbmres->ntuples < 0;
+ *lossy = tbmres.ntuples < 0;
/*
* Return true to indicate that a valid block was found and the bitmap is
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index b1dfa582c7d..43df0ac9964 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -321,7 +321,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
{
#ifdef USE_PREFETCH
ParallelBitmapHeapState *pstate = node->pstate;
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
if (pstate == NULL)
{
@@ -335,8 +335,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
else if (prefetch_iterator)
{
/* Do not let the prefetch iterator get behind the main one */
- tbmpre = tbm_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
return;
}
@@ -371,8 +371,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
*/
if (prefetch_iterator)
{
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
}
}
@@ -440,10 +440,12 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (node->prefetch_pages < node->prefetch_target)
{
- TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator);
+ TBMIterateResult tbmpre;
bool skip_fetch;
- if (tbmpre == NULL)
+ tbm_iterate(prefetch_iterator, &tbmpre);
+
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_iterate(prefetch_iterator);
@@ -451,7 +453,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
node->prefetch_pages++;
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/*
* If we expect not to have to actually read this heap page,
@@ -460,13 +462,13 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
* prefetch_pages?)
*/
skip_fetch = (scan->rs_flags & SO_CAN_SKIP_FETCH &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
@@ -481,7 +483,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (1)
{
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
bool do_prefetch = false;
bool skip_fetch;
@@ -500,8 +502,8 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
if (!do_prefetch)
return;
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- if (tbmpre == NULL)
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_shared_iterate(prefetch_iterator);
@@ -509,17 +511,17 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/* As above, skip prefetch if we expect not to need page */
skip_fetch = (scan->rs_flags & SO_CAN_SKIP_FETCH &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
}
diff --git a/src/backend/nodes/tidbitmap.c b/src/backend/nodes/tidbitmap.c
index d2bf8f44d50..7d038c2018d 100644
--- a/src/backend/nodes/tidbitmap.c
+++ b/src/backend/nodes/tidbitmap.c
@@ -172,7 +172,6 @@ struct TBMIterator
int spageptr; /* next spages index */
int schunkptr; /* next schunks index */
int schunkbit; /* next bit to check in current schunk */
- TBMIterateResult output;
};
/*
@@ -213,7 +212,6 @@ struct TBMSharedIterator
PTEntryArray *ptbase; /* pagetable element array */
PTIterationArray *ptpages; /* sorted exact page index list */
PTIterationArray *ptchunks; /* sorted lossy page index list */
- TBMIterateResult output;
};
/* Local function prototypes */
@@ -944,20 +942,21 @@ tbm_advance_schunkbit(PagetableEntry *chunk, int *schunkbitp)
/*
* tbm_iterate - scan through next page of a TIDBitmap
*
- * Returns a TBMIterateResult representing one page, or NULL if there are
- * no more pages to scan. Pages are guaranteed to be delivered in numerical
- * order. If result->ntuples < 0, then the bitmap is "lossy" and failed to
- * remember the exact tuples to look at on this page --- the caller must
- * examine all tuples on the page and check if they meet the intended
- * condition. If result->recheck is true, only the indicated tuples need
- * be examined, but the condition must be rechecked anyway. (For ease of
- * testing, recheck is always set true when ntuples < 0.)
+ * Caller must pass in a TBMIterateResult to be filled.
+ *
+ * Pages are guaranteed to be delivered in numerical order. tbmres->blockno is
+ * set to InvalidBlockNumber when there are no more pages to scan. If
+ * tbmres->ntuples < 0, then the bitmap is "lossy" and failed to remember the
+ * exact tuples to look at on this page --- the caller must examine all tuples
+ * on the page and check if they meet the intended condition. If
+ * tbmres->recheck is true, only the indicated tuples need be examined, but the
+ * condition must be rechecked anyway. (For ease of testing, recheck is always
+ * set true when ntuples < 0.)
*/
-TBMIterateResult *
-tbm_iterate(TBMIterator *iterator)
+void
+tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres)
{
TIDBitmap *tbm = iterator->tbm;
- TBMIterateResult *output = &(iterator->output);
Assert(tbm->iterating == TBM_ITERATING_PRIVATE);
@@ -985,6 +984,7 @@ tbm_iterate(TBMIterator *iterator)
* If both chunk and per-page data remain, must output the numerically
* earlier page.
*/
+ Assert(tbmres);
if (iterator->schunkptr < tbm->nchunks)
{
PagetableEntry *chunk = tbm->schunks[iterator->schunkptr];
@@ -995,11 +995,11 @@ tbm_iterate(TBMIterator *iterator)
chunk_blockno < tbm->spages[iterator->spageptr]->blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
iterator->schunkbit++;
- return output;
+ return;
}
}
@@ -1015,16 +1015,17 @@ tbm_iterate(TBMIterator *iterator)
page = tbm->spages[iterator->spageptr];
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
iterator->spageptr++;
- return output;
+ return;
}
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
@@ -1034,10 +1035,9 @@ tbm_iterate(TBMIterator *iterator)
* across multiple processes. We need to acquire the iterator LWLock,
* before accessing the shared members.
*/
-TBMIterateResult *
-tbm_shared_iterate(TBMSharedIterator *iterator)
+void
+tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres)
{
- TBMIterateResult *output = &iterator->output;
TBMSharedIteratorState *istate = iterator->state;
PagetableEntry *ptbase = NULL;
int *idxpages = NULL;
@@ -1088,13 +1088,13 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
chunk_blockno < ptbase[idxpages[istate->spageptr]].blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
istate->schunkbit++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
}
@@ -1104,21 +1104,22 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
int ntuples;
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
istate->spageptr++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
LWLockRelease(&istate->lock);
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 3013a44bae1..3b432263bb0 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -353,7 +353,7 @@ typedef struct GinScanEntryData
/* for a partial-match or full-scan query, we accumulate all TIDs here */
TIDBitmap *matchBitmap;
TBMIterator *matchIterator;
- TBMIterateResult *matchResult;
+ TBMIterateResult matchResult;
/* used for Posting list and one page in Posting tree */
ItemPointerData *list;
diff --git a/src/include/nodes/tidbitmap.h b/src/include/nodes/tidbitmap.h
index 432fae52962..f000c1af28f 100644
--- a/src/include/nodes/tidbitmap.h
+++ b/src/include/nodes/tidbitmap.h
@@ -72,8 +72,8 @@ extern bool tbm_is_empty(const TIDBitmap *tbm);
extern TBMIterator *tbm_begin_iterate(TIDBitmap *tbm);
extern dsa_pointer tbm_prepare_shared_iterate(TIDBitmap *tbm);
-extern TBMIterateResult *tbm_iterate(TBMIterator *iterator);
-extern TBMIterateResult *tbm_shared_iterate(TBMSharedIterator *iterator);
+extern void tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres);
+extern void tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres);
extern void tbm_end_iterate(TBMIterator *iterator);
extern void tbm_end_shared_iterate(TBMSharedIterator *iterator);
extern TBMSharedIterator *tbm_attach_shared_iterate(dsa_area *dsa,
--
2.40.1
--kqqpqghcwbcc3dt5
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0012-v7-Streaming-Read-API.patch"
^ permalink raw reply [nested|flat] 59+ messages in thread
* [PATCH v8 12/17] Separate TBM[Shared]Iterator and TBMIterateResult
@ 2024-02-16 02:23 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Melanie Plageman @ 2024-02-16 02:23 UTC (permalink / raw)
Remove the TBMIterateResult from the TBMIterator and TBMSharedIterator
and have tbm_[shared_]iterate() take a TBMIterateResult as a parameter.
This will allow multiple TBMIterateResults to exist concurrently
allowing asynchronous use of the TIDBitmap for prefetching, for example.
tbm_[shared]_iterate() now sets blockno to InvalidBlockNumber when the
bitmap is exhausted instead of returning NULL.
BitmapHeapScan callers of tbm_iterate make a TBMIterateResult locally
and pass it in.
Because GIN only needs a single TBMIterateResult, inline the matchResult
in the GinScanEntry to avoid having to separately manage memory for the
TBMIterateResult.
---
src/backend/access/gin/ginget.c | 48 +++++++++------
src/backend/access/gin/ginscan.c | 2 +-
src/backend/access/heap/heapam_handler.c | 32 +++++-----
src/backend/executor/nodeBitmapHeapscan.c | 38 ++++++------
src/backend/nodes/tidbitmap.c | 73 ++++++++++++-----------
src/include/access/gin_private.h | 2 +-
src/include/nodes/tidbitmap.h | 4 +-
7 files changed, 108 insertions(+), 91 deletions(-)
diff --git a/src/backend/access/gin/ginget.c b/src/backend/access/gin/ginget.c
index 0b4f2ebadb..3aa457a29e 100644
--- a/src/backend/access/gin/ginget.c
+++ b/src/backend/access/gin/ginget.c
@@ -332,10 +332,22 @@ restartScanEntry:
entry->list = NULL;
entry->nlist = 0;
entry->matchBitmap = NULL;
- entry->matchResult = NULL;
entry->reduceResult = false;
entry->predictNumberResult = 0;
+ /*
+ * MTODO: is it enough to set blockno to InvalidBlockNumber? In all the
+ * places were we previously set matchResult to NULL, I just set blockno
+ * to InvalidBlockNumber. It seems like this should be okay because that
+ * is usually what we check before using the matchResult members. But it
+ * might be safer to zero out the offsets array. But that is expensive.
+ */
+ entry->matchResult.blockno = InvalidBlockNumber;
+ entry->matchResult.ntuples = 0;
+ entry->matchResult.recheck = true;
+ memset(entry->matchResult.offsets, 0,
+ sizeof(OffsetNumber) * MaxHeapTuplesPerPage);
+
/*
* we should find entry, and begin scan of posting tree or just store
* posting list in memory
@@ -374,6 +386,7 @@ restartScanEntry:
{
if (entry->matchIterator)
tbm_end_iterate(entry->matchIterator);
+ entry->matchResult.blockno = InvalidBlockNumber;
entry->matchIterator = NULL;
tbm_free(entry->matchBitmap);
entry->matchBitmap = NULL;
@@ -823,18 +836,19 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
{
/*
* If we've exhausted all items on this block, move to next block
- * in the bitmap.
+ * in the bitmap. tbm_iterate() sets matchResult->blockno to
+ * InvalidBlockNumber when the bitmap is exhausted.
*/
- while (entry->matchResult == NULL ||
- (entry->matchResult->ntuples >= 0 &&
- entry->offset >= entry->matchResult->ntuples) ||
- entry->matchResult->blockno < advancePastBlk ||
+ while ((!BlockNumberIsValid(entry->matchResult.blockno)) ||
+ (entry->matchResult.ntuples >= 0 &&
+ entry->offset >= entry->matchResult.ntuples) ||
+ entry->matchResult.blockno < advancePastBlk ||
(ItemPointerIsLossyPage(&advancePast) &&
- entry->matchResult->blockno == advancePastBlk))
+ entry->matchResult.blockno == advancePastBlk))
{
- entry->matchResult = tbm_iterate(entry->matchIterator);
+ tbm_iterate(entry->matchIterator, &entry->matchResult);
- if (entry->matchResult == NULL)
+ if (!BlockNumberIsValid(entry->matchResult.blockno))
{
ItemPointerSetInvalid(&entry->curItem);
tbm_end_iterate(entry->matchIterator);
@@ -858,10 +872,10 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* We're now on the first page after advancePast which has any
* items on it. If it's a lossy result, return that.
*/
- if (entry->matchResult->ntuples < 0)
+ if (entry->matchResult.ntuples < 0)
{
ItemPointerSetLossyPage(&entry->curItem,
- entry->matchResult->blockno);
+ entry->matchResult.blockno);
/*
* We might as well fall out of the loop; we could not
@@ -875,27 +889,27 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* Not a lossy page. Skip over any offsets <= advancePast, and
* return that.
*/
- if (entry->matchResult->blockno == advancePastBlk)
+ if (entry->matchResult.blockno == advancePastBlk)
{
/*
* First, do a quick check against the last offset on the
* page. If that's > advancePast, so are all the other
* offsets, so just go back to the top to get the next page.
*/
- if (entry->matchResult->offsets[entry->matchResult->ntuples - 1] <= advancePastOff)
+ if (entry->matchResult.offsets[entry->matchResult.ntuples - 1] <= advancePastOff)
{
- entry->offset = entry->matchResult->ntuples;
+ entry->offset = entry->matchResult.ntuples;
continue;
}
/* Otherwise scan to find the first item > advancePast */
- while (entry->matchResult->offsets[entry->offset] <= advancePastOff)
+ while (entry->matchResult.offsets[entry->offset] <= advancePastOff)
entry->offset++;
}
ItemPointerSet(&entry->curItem,
- entry->matchResult->blockno,
- entry->matchResult->offsets[entry->offset]);
+ entry->matchResult.blockno,
+ entry->matchResult.offsets[entry->offset]);
entry->offset++;
/* Done unless we need to reduce the result */
diff --git a/src/backend/access/gin/ginscan.c b/src/backend/access/gin/ginscan.c
index af24d38544..033d525339 100644
--- a/src/backend/access/gin/ginscan.c
+++ b/src/backend/access/gin/ginscan.c
@@ -106,7 +106,7 @@ ginFillScanEntry(GinScanOpaque so, OffsetNumber attnum,
ItemPointerSetMin(&scanEntry->curItem);
scanEntry->matchBitmap = NULL;
scanEntry->matchIterator = NULL;
- scanEntry->matchResult = NULL;
+ scanEntry->matchResult.blockno = InvalidBlockNumber;
scanEntry->list = NULL;
scanEntry->nlist = 0;
scanEntry->offset = InvalidOffsetNumber;
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 266b34fe6b..adfc77684a 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2122,7 +2122,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Buffer buffer;
Snapshot snapshot;
int ntup;
- TBMIterateResult *tbmres;
+ TBMIterateResult tbmres;
hscan->rs_cindex = 0;
hscan->rs_ntuples = 0;
@@ -2135,11 +2135,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
CHECK_FOR_INTERRUPTS();
if (scan->shared_tbmiterator)
- tbmres = tbm_shared_iterate(scan->shared_tbmiterator);
+ tbm_shared_iterate(scan->shared_tbmiterator, &tbmres);
else
- tbmres = tbm_iterate(scan->tbmiterator);
+ tbm_iterate(scan->tbmiterator, &tbmres);
- if (tbmres == NULL)
+ if (!BlockNumberIsValid(tbmres.blockno))
{
/* no more entries in the bitmap */
Assert(hscan->rs_empty_tuples_pending == 0);
@@ -2154,11 +2154,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* isolation though, as we need to examine all invisible tuples
* reachable by the index.
*/
- } while (!IsolationIsSerializable() && tbmres->blockno >= hscan->rs_nblocks);
+ } while (!IsolationIsSerializable() && tbmres.blockno >= hscan->rs_nblocks);
/* Got a valid block */
- *blockno = tbmres->blockno;
- *recheck = tbmres->recheck;
+ *blockno = tbmres.blockno;
+ *recheck = tbmres.recheck;
/*
* We can skip fetching the heap page if we don't need any fields from the
@@ -2166,19 +2166,19 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* page are visible to our transaction.
*/
if (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmres->recheck &&
- VM_ALL_VISIBLE(scan->rs_rd, tbmres->blockno, &hscan->rs_vmbuffer))
+ !tbmres.recheck &&
+ VM_ALL_VISIBLE(scan->rs_rd, tbmres.blockno, &hscan->rs_vmbuffer))
{
/* can't be lossy in the skip_fetch case */
- Assert(tbmres->ntuples >= 0);
+ Assert(tbmres.ntuples >= 0);
Assert(hscan->rs_empty_tuples_pending >= 0);
- hscan->rs_empty_tuples_pending += tbmres->ntuples;
+ hscan->rs_empty_tuples_pending += tbmres.ntuples;
return true;
}
- block = tbmres->blockno;
+ block = tbmres.blockno;
/*
* Acquire pin on the target heap page, trading in any pin we held before.
@@ -2207,7 +2207,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
/*
* We need two separate strategies for lossy and non-lossy cases.
*/
- if (tbmres->ntuples >= 0)
+ if (tbmres.ntuples >= 0)
{
/*
* Bitmap is non-lossy, so we just look through the offsets listed in
@@ -2216,9 +2216,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
*/
int curslot;
- for (curslot = 0; curslot < tbmres->ntuples; curslot++)
+ for (curslot = 0; curslot < tbmres.ntuples; curslot++)
{
- OffsetNumber offnum = tbmres->offsets[curslot];
+ OffsetNumber offnum = tbmres.offsets[curslot];
ItemPointerData tid;
HeapTupleData heapTuple;
@@ -2268,7 +2268,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
- if (tbmres->ntuples < 0)
+ if (tbmres.ntuples < 0)
(*lossy_pages)++;
else
(*exact_pages)++;
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 96b55507a3..51c4360205 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -316,7 +316,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
{
#ifdef USE_PREFETCH
ParallelBitmapHeapState *pstate = node->pstate;
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
if (pstate == NULL)
{
@@ -330,8 +330,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
else if (prefetch_iterator)
{
/* Do not let the prefetch iterator get behind the main one */
- tbmpre = tbm_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
return;
}
@@ -366,8 +366,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
*/
if (prefetch_iterator)
{
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
}
}
@@ -435,10 +435,12 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (node->prefetch_pages < node->prefetch_target)
{
- TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator);
+ TBMIterateResult tbmpre;
bool skip_fetch;
- if (tbmpre == NULL)
+ tbm_iterate(prefetch_iterator, &tbmpre);
+
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_iterate(prefetch_iterator);
@@ -446,7 +448,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
node->prefetch_pages++;
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/*
* If we expect not to have to actually read this heap page,
@@ -455,13 +457,13 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
* prefetch_pages?)
*/
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
@@ -476,7 +478,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (1)
{
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
bool do_prefetch = false;
bool skip_fetch;
@@ -495,8 +497,8 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
if (!do_prefetch)
return;
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- if (tbmpre == NULL)
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_shared_iterate(prefetch_iterator);
@@ -504,17 +506,17 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/* As above, skip prefetch if we expect not to need page */
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
}
diff --git a/src/backend/nodes/tidbitmap.c b/src/backend/nodes/tidbitmap.c
index 1dc4c99bf9..309a44bdb8 100644
--- a/src/backend/nodes/tidbitmap.c
+++ b/src/backend/nodes/tidbitmap.c
@@ -172,7 +172,6 @@ struct TBMIterator
int spageptr; /* next spages index */
int schunkptr; /* next schunks index */
int schunkbit; /* next bit to check in current schunk */
- TBMIterateResult output;
};
/*
@@ -213,7 +212,6 @@ struct TBMSharedIterator
PTEntryArray *ptbase; /* pagetable element array */
PTIterationArray *ptpages; /* sorted exact page index list */
PTIterationArray *ptchunks; /* sorted lossy page index list */
- TBMIterateResult output;
};
/* Local function prototypes */
@@ -944,20 +942,21 @@ tbm_advance_schunkbit(PagetableEntry *chunk, int *schunkbitp)
/*
* tbm_iterate - scan through next page of a TIDBitmap
*
- * Returns a TBMIterateResult representing one page, or NULL if there are
- * no more pages to scan. Pages are guaranteed to be delivered in numerical
- * order. If result->ntuples < 0, then the bitmap is "lossy" and failed to
- * remember the exact tuples to look at on this page --- the caller must
- * examine all tuples on the page and check if they meet the intended
- * condition. If result->recheck is true, only the indicated tuples need
- * be examined, but the condition must be rechecked anyway. (For ease of
- * testing, recheck is always set true when ntuples < 0.)
+ * Caller must pass in a TBMIterateResult to be filled.
+ *
+ * Pages are guaranteed to be delivered in numerical order. tbmres->blockno is
+ * set to InvalidBlockNumber when there are no more pages to scan. If
+ * tbmres->ntuples < 0, then the bitmap is "lossy" and failed to remember the
+ * exact tuples to look at on this page --- the caller must examine all tuples
+ * on the page and check if they meet the intended condition. If
+ * tbmres->recheck is true, only the indicated tuples need be examined, but the
+ * condition must be rechecked anyway. (For ease of testing, recheck is always
+ * set true when ntuples < 0.)
*/
-TBMIterateResult *
-tbm_iterate(TBMIterator *iterator)
+void
+tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres)
{
TIDBitmap *tbm = iterator->tbm;
- TBMIterateResult *output = &(iterator->output);
Assert(tbm->iterating == TBM_ITERATING_PRIVATE);
@@ -985,6 +984,7 @@ tbm_iterate(TBMIterator *iterator)
* If both chunk and per-page data remain, must output the numerically
* earlier page.
*/
+ Assert(tbmres);
if (iterator->schunkptr < tbm->nchunks)
{
PagetableEntry *chunk = tbm->schunks[iterator->schunkptr];
@@ -995,11 +995,11 @@ tbm_iterate(TBMIterator *iterator)
chunk_blockno < tbm->spages[iterator->spageptr]->blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
iterator->schunkbit++;
- return output;
+ return;
}
}
@@ -1015,16 +1015,17 @@ tbm_iterate(TBMIterator *iterator)
page = tbm->spages[iterator->spageptr];
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
iterator->spageptr++;
- return output;
+ return;
}
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
@@ -1034,10 +1035,9 @@ tbm_iterate(TBMIterator *iterator)
* across multiple processes. We need to acquire the iterator LWLock,
* before accessing the shared members.
*/
-TBMIterateResult *
-tbm_shared_iterate(TBMSharedIterator *iterator)
+void
+tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres)
{
- TBMIterateResult *output = &iterator->output;
TBMSharedIteratorState *istate = iterator->state;
PagetableEntry *ptbase = NULL;
int *idxpages = NULL;
@@ -1088,13 +1088,13 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
chunk_blockno < ptbase[idxpages[istate->spageptr]].blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
istate->schunkbit++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
}
@@ -1104,21 +1104,22 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
int ntuples;
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
istate->spageptr++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
LWLockRelease(&istate->lock);
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 3013a44bae..3b432263bb 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -353,7 +353,7 @@ typedef struct GinScanEntryData
/* for a partial-match or full-scan query, we accumulate all TIDs here */
TIDBitmap *matchBitmap;
TBMIterator *matchIterator;
- TBMIterateResult *matchResult;
+ TBMIterateResult matchResult;
/* used for Posting list and one page in Posting tree */
ItemPointerData *list;
diff --git a/src/include/nodes/tidbitmap.h b/src/include/nodes/tidbitmap.h
index 432fae5296..f000c1af28 100644
--- a/src/include/nodes/tidbitmap.h
+++ b/src/include/nodes/tidbitmap.h
@@ -72,8 +72,8 @@ extern bool tbm_is_empty(const TIDBitmap *tbm);
extern TBMIterator *tbm_begin_iterate(TIDBitmap *tbm);
extern dsa_pointer tbm_prepare_shared_iterate(TIDBitmap *tbm);
-extern TBMIterateResult *tbm_iterate(TBMIterator *iterator);
-extern TBMIterateResult *tbm_shared_iterate(TBMSharedIterator *iterator);
+extern void tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres);
+extern void tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres);
extern void tbm_end_iterate(TBMIterator *iterator);
extern void tbm_end_shared_iterate(TBMSharedIterator *iterator);
extern TBMSharedIterator *tbm_attach_shared_iterate(dsa_area *dsa,
--
2.40.1
--xqq4defy3uncu6k6
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v8-0013-Push-BitmapHeapScan-prefetch-code-into-heapam.c.patch"
^ permalink raw reply [nested|flat] 59+ messages in thread
* [PATCH v7 11/13] Separate TBM[Shared]Iterator and TBMIterateResult
@ 2024-02-16 02:23 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Melanie Plageman @ 2024-02-16 02:23 UTC (permalink / raw)
Remove the TBMIterateResult from the TBMIterator and TBMSharedIterator
and have tbm_[shared_]iterate() take a TBMIterateResult as a parameter.
This will allow multiple TBMIterateResults to exist concurrently
allowing asynchronous use of the TIDBitmap for prefetching, for example.
tbm_[shared]_iterate() now sets blockno to InvalidBlockNumber when the
bitmap is exhausted instead of returning NULL.
BitmapHeapScan callers of tbm_iterate make a TBMIterateResult locally
and pass it in.
Because GIN only needs a single TBMIterateResult, inline the matchResult
in the GinScanEntry to avoid having to separately manage memory for the
TBMIterateResult.
---
src/backend/access/gin/ginget.c | 48 +++++++++------
src/backend/access/gin/ginscan.c | 2 +-
src/backend/access/heap/heapam_handler.c | 32 +++++-----
src/backend/executor/nodeBitmapHeapscan.c | 38 ++++++------
src/backend/nodes/tidbitmap.c | 73 ++++++++++++-----------
src/include/access/gin_private.h | 2 +-
src/include/nodes/tidbitmap.h | 4 +-
7 files changed, 108 insertions(+), 91 deletions(-)
diff --git a/src/backend/access/gin/ginget.c b/src/backend/access/gin/ginget.c
index 0b4f2ebadb6..3aa457a29e1 100644
--- a/src/backend/access/gin/ginget.c
+++ b/src/backend/access/gin/ginget.c
@@ -332,10 +332,22 @@ restartScanEntry:
entry->list = NULL;
entry->nlist = 0;
entry->matchBitmap = NULL;
- entry->matchResult = NULL;
entry->reduceResult = false;
entry->predictNumberResult = 0;
+ /*
+ * MTODO: is it enough to set blockno to InvalidBlockNumber? In all the
+ * places were we previously set matchResult to NULL, I just set blockno
+ * to InvalidBlockNumber. It seems like this should be okay because that
+ * is usually what we check before using the matchResult members. But it
+ * might be safer to zero out the offsets array. But that is expensive.
+ */
+ entry->matchResult.blockno = InvalidBlockNumber;
+ entry->matchResult.ntuples = 0;
+ entry->matchResult.recheck = true;
+ memset(entry->matchResult.offsets, 0,
+ sizeof(OffsetNumber) * MaxHeapTuplesPerPage);
+
/*
* we should find entry, and begin scan of posting tree or just store
* posting list in memory
@@ -374,6 +386,7 @@ restartScanEntry:
{
if (entry->matchIterator)
tbm_end_iterate(entry->matchIterator);
+ entry->matchResult.blockno = InvalidBlockNumber;
entry->matchIterator = NULL;
tbm_free(entry->matchBitmap);
entry->matchBitmap = NULL;
@@ -823,18 +836,19 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
{
/*
* If we've exhausted all items on this block, move to next block
- * in the bitmap.
+ * in the bitmap. tbm_iterate() sets matchResult->blockno to
+ * InvalidBlockNumber when the bitmap is exhausted.
*/
- while (entry->matchResult == NULL ||
- (entry->matchResult->ntuples >= 0 &&
- entry->offset >= entry->matchResult->ntuples) ||
- entry->matchResult->blockno < advancePastBlk ||
+ while ((!BlockNumberIsValid(entry->matchResult.blockno)) ||
+ (entry->matchResult.ntuples >= 0 &&
+ entry->offset >= entry->matchResult.ntuples) ||
+ entry->matchResult.blockno < advancePastBlk ||
(ItemPointerIsLossyPage(&advancePast) &&
- entry->matchResult->blockno == advancePastBlk))
+ entry->matchResult.blockno == advancePastBlk))
{
- entry->matchResult = tbm_iterate(entry->matchIterator);
+ tbm_iterate(entry->matchIterator, &entry->matchResult);
- if (entry->matchResult == NULL)
+ if (!BlockNumberIsValid(entry->matchResult.blockno))
{
ItemPointerSetInvalid(&entry->curItem);
tbm_end_iterate(entry->matchIterator);
@@ -858,10 +872,10 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* We're now on the first page after advancePast which has any
* items on it. If it's a lossy result, return that.
*/
- if (entry->matchResult->ntuples < 0)
+ if (entry->matchResult.ntuples < 0)
{
ItemPointerSetLossyPage(&entry->curItem,
- entry->matchResult->blockno);
+ entry->matchResult.blockno);
/*
* We might as well fall out of the loop; we could not
@@ -875,27 +889,27 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* Not a lossy page. Skip over any offsets <= advancePast, and
* return that.
*/
- if (entry->matchResult->blockno == advancePastBlk)
+ if (entry->matchResult.blockno == advancePastBlk)
{
/*
* First, do a quick check against the last offset on the
* page. If that's > advancePast, so are all the other
* offsets, so just go back to the top to get the next page.
*/
- if (entry->matchResult->offsets[entry->matchResult->ntuples - 1] <= advancePastOff)
+ if (entry->matchResult.offsets[entry->matchResult.ntuples - 1] <= advancePastOff)
{
- entry->offset = entry->matchResult->ntuples;
+ entry->offset = entry->matchResult.ntuples;
continue;
}
/* Otherwise scan to find the first item > advancePast */
- while (entry->matchResult->offsets[entry->offset] <= advancePastOff)
+ while (entry->matchResult.offsets[entry->offset] <= advancePastOff)
entry->offset++;
}
ItemPointerSet(&entry->curItem,
- entry->matchResult->blockno,
- entry->matchResult->offsets[entry->offset]);
+ entry->matchResult.blockno,
+ entry->matchResult.offsets[entry->offset]);
entry->offset++;
/* Done unless we need to reduce the result */
diff --git a/src/backend/access/gin/ginscan.c b/src/backend/access/gin/ginscan.c
index af24d38544e..033d5253394 100644
--- a/src/backend/access/gin/ginscan.c
+++ b/src/backend/access/gin/ginscan.c
@@ -106,7 +106,7 @@ ginFillScanEntry(GinScanOpaque so, OffsetNumber attnum,
ItemPointerSetMin(&scanEntry->curItem);
scanEntry->matchBitmap = NULL;
scanEntry->matchIterator = NULL;
- scanEntry->matchResult = NULL;
+ scanEntry->matchResult.blockno = InvalidBlockNumber;
scanEntry->list = NULL;
scanEntry->nlist = 0;
scanEntry->offset = InvalidOffsetNumber;
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index e038e60cd8f..022753e203a 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2119,7 +2119,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Buffer buffer;
Snapshot snapshot;
int ntup;
- TBMIterateResult *tbmres;
+ TBMIterateResult tbmres;
hscan->rs_cindex = 0;
hscan->rs_ntuples = 0;
@@ -2132,11 +2132,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
CHECK_FOR_INTERRUPTS();
if (scan->shared_tbmiterator)
- tbmres = tbm_shared_iterate(scan->shared_tbmiterator);
+ tbm_shared_iterate(scan->shared_tbmiterator, &tbmres);
else
- tbmres = tbm_iterate(scan->tbmiterator);
+ tbm_iterate(scan->tbmiterator, &tbmres);
- if (tbmres == NULL)
+ if (!BlockNumberIsValid(tbmres.blockno))
{
/* no more entries in the bitmap */
Assert(hscan->rs_empty_tuples_pending == 0);
@@ -2151,11 +2151,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* isolation though, as we need to examine all invisible tuples
* reachable by the index.
*/
- } while (!IsolationIsSerializable() && tbmres->blockno >= hscan->rs_nblocks);
+ } while (!IsolationIsSerializable() && tbmres.blockno >= hscan->rs_nblocks);
/* Got a valid block */
- *blockno = tbmres->blockno;
- *recheck = tbmres->recheck;
+ *blockno = tbmres.blockno;
+ *recheck = tbmres.recheck;
/*
* We can skip fetching the heap page if we don't need any fields from the
@@ -2163,19 +2163,19 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* the page are visible to our transaction.
*/
if (scan->rs_flags & SO_CAN_SKIP_FETCH &&
- !tbmres->recheck &&
- VM_ALL_VISIBLE(scan->rs_rd, tbmres->blockno, &hscan->rs_vmbuffer))
+ !tbmres.recheck &&
+ VM_ALL_VISIBLE(scan->rs_rd, tbmres.blockno, &hscan->rs_vmbuffer))
{
/* can't be lossy in the skip_fetch case */
- Assert(tbmres->ntuples >= 0);
+ Assert(tbmres.ntuples >= 0);
Assert(hscan->rs_empty_tuples_pending >= 0);
- hscan->rs_empty_tuples_pending += tbmres->ntuples;
+ hscan->rs_empty_tuples_pending += tbmres.ntuples;
return true;
}
- block = tbmres->blockno;
+ block = tbmres.blockno;
/*
* Acquire pin on the target heap page, trading in any pin we held before.
@@ -2204,7 +2204,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
/*
* We need two separate strategies for lossy and non-lossy cases.
*/
- if (tbmres->ntuples >= 0)
+ if (tbmres.ntuples >= 0)
{
/*
* Bitmap is non-lossy, so we just look through the offsets listed in
@@ -2213,9 +2213,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
*/
int curslot;
- for (curslot = 0; curslot < tbmres->ntuples; curslot++)
+ for (curslot = 0; curslot < tbmres.ntuples; curslot++)
{
- OffsetNumber offnum = tbmres->offsets[curslot];
+ OffsetNumber offnum = tbmres.offsets[curslot];
ItemPointerData tid;
HeapTupleData heapTuple;
@@ -2265,7 +2265,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
- *lossy = tbmres->ntuples < 0;
+ *lossy = tbmres.ntuples < 0;
/*
* Return true to indicate that a valid block was found and the bitmap is
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index b1dfa582c7d..43df0ac9964 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -321,7 +321,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
{
#ifdef USE_PREFETCH
ParallelBitmapHeapState *pstate = node->pstate;
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
if (pstate == NULL)
{
@@ -335,8 +335,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
else if (prefetch_iterator)
{
/* Do not let the prefetch iterator get behind the main one */
- tbmpre = tbm_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
return;
}
@@ -371,8 +371,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
*/
if (prefetch_iterator)
{
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
}
}
@@ -440,10 +440,12 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (node->prefetch_pages < node->prefetch_target)
{
- TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator);
+ TBMIterateResult tbmpre;
bool skip_fetch;
- if (tbmpre == NULL)
+ tbm_iterate(prefetch_iterator, &tbmpre);
+
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_iterate(prefetch_iterator);
@@ -451,7 +453,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
node->prefetch_pages++;
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/*
* If we expect not to have to actually read this heap page,
@@ -460,13 +462,13 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
* prefetch_pages?)
*/
skip_fetch = (scan->rs_flags & SO_CAN_SKIP_FETCH &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
@@ -481,7 +483,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (1)
{
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
bool do_prefetch = false;
bool skip_fetch;
@@ -500,8 +502,8 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
if (!do_prefetch)
return;
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- if (tbmpre == NULL)
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_shared_iterate(prefetch_iterator);
@@ -509,17 +511,17 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/* As above, skip prefetch if we expect not to need page */
skip_fetch = (scan->rs_flags & SO_CAN_SKIP_FETCH &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
}
diff --git a/src/backend/nodes/tidbitmap.c b/src/backend/nodes/tidbitmap.c
index d2bf8f44d50..7d038c2018d 100644
--- a/src/backend/nodes/tidbitmap.c
+++ b/src/backend/nodes/tidbitmap.c
@@ -172,7 +172,6 @@ struct TBMIterator
int spageptr; /* next spages index */
int schunkptr; /* next schunks index */
int schunkbit; /* next bit to check in current schunk */
- TBMIterateResult output;
};
/*
@@ -213,7 +212,6 @@ struct TBMSharedIterator
PTEntryArray *ptbase; /* pagetable element array */
PTIterationArray *ptpages; /* sorted exact page index list */
PTIterationArray *ptchunks; /* sorted lossy page index list */
- TBMIterateResult output;
};
/* Local function prototypes */
@@ -944,20 +942,21 @@ tbm_advance_schunkbit(PagetableEntry *chunk, int *schunkbitp)
/*
* tbm_iterate - scan through next page of a TIDBitmap
*
- * Returns a TBMIterateResult representing one page, or NULL if there are
- * no more pages to scan. Pages are guaranteed to be delivered in numerical
- * order. If result->ntuples < 0, then the bitmap is "lossy" and failed to
- * remember the exact tuples to look at on this page --- the caller must
- * examine all tuples on the page and check if they meet the intended
- * condition. If result->recheck is true, only the indicated tuples need
- * be examined, but the condition must be rechecked anyway. (For ease of
- * testing, recheck is always set true when ntuples < 0.)
+ * Caller must pass in a TBMIterateResult to be filled.
+ *
+ * Pages are guaranteed to be delivered in numerical order. tbmres->blockno is
+ * set to InvalidBlockNumber when there are no more pages to scan. If
+ * tbmres->ntuples < 0, then the bitmap is "lossy" and failed to remember the
+ * exact tuples to look at on this page --- the caller must examine all tuples
+ * on the page and check if they meet the intended condition. If
+ * tbmres->recheck is true, only the indicated tuples need be examined, but the
+ * condition must be rechecked anyway. (For ease of testing, recheck is always
+ * set true when ntuples < 0.)
*/
-TBMIterateResult *
-tbm_iterate(TBMIterator *iterator)
+void
+tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres)
{
TIDBitmap *tbm = iterator->tbm;
- TBMIterateResult *output = &(iterator->output);
Assert(tbm->iterating == TBM_ITERATING_PRIVATE);
@@ -985,6 +984,7 @@ tbm_iterate(TBMIterator *iterator)
* If both chunk and per-page data remain, must output the numerically
* earlier page.
*/
+ Assert(tbmres);
if (iterator->schunkptr < tbm->nchunks)
{
PagetableEntry *chunk = tbm->schunks[iterator->schunkptr];
@@ -995,11 +995,11 @@ tbm_iterate(TBMIterator *iterator)
chunk_blockno < tbm->spages[iterator->spageptr]->blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
iterator->schunkbit++;
- return output;
+ return;
}
}
@@ -1015,16 +1015,17 @@ tbm_iterate(TBMIterator *iterator)
page = tbm->spages[iterator->spageptr];
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
iterator->spageptr++;
- return output;
+ return;
}
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
@@ -1034,10 +1035,9 @@ tbm_iterate(TBMIterator *iterator)
* across multiple processes. We need to acquire the iterator LWLock,
* before accessing the shared members.
*/
-TBMIterateResult *
-tbm_shared_iterate(TBMSharedIterator *iterator)
+void
+tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres)
{
- TBMIterateResult *output = &iterator->output;
TBMSharedIteratorState *istate = iterator->state;
PagetableEntry *ptbase = NULL;
int *idxpages = NULL;
@@ -1088,13 +1088,13 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
chunk_blockno < ptbase[idxpages[istate->spageptr]].blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
istate->schunkbit++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
}
@@ -1104,21 +1104,22 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
int ntuples;
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
istate->spageptr++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
LWLockRelease(&istate->lock);
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 3013a44bae1..3b432263bb0 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -353,7 +353,7 @@ typedef struct GinScanEntryData
/* for a partial-match or full-scan query, we accumulate all TIDs here */
TIDBitmap *matchBitmap;
TBMIterator *matchIterator;
- TBMIterateResult *matchResult;
+ TBMIterateResult matchResult;
/* used for Posting list and one page in Posting tree */
ItemPointerData *list;
diff --git a/src/include/nodes/tidbitmap.h b/src/include/nodes/tidbitmap.h
index 432fae52962..f000c1af28f 100644
--- a/src/include/nodes/tidbitmap.h
+++ b/src/include/nodes/tidbitmap.h
@@ -72,8 +72,8 @@ extern bool tbm_is_empty(const TIDBitmap *tbm);
extern TBMIterator *tbm_begin_iterate(TIDBitmap *tbm);
extern dsa_pointer tbm_prepare_shared_iterate(TIDBitmap *tbm);
-extern TBMIterateResult *tbm_iterate(TBMIterator *iterator);
-extern TBMIterateResult *tbm_shared_iterate(TBMSharedIterator *iterator);
+extern void tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres);
+extern void tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres);
extern void tbm_end_iterate(TBMIterator *iterator);
extern void tbm_end_shared_iterate(TBMSharedIterator *iterator);
extern TBMSharedIterator *tbm_attach_shared_iterate(dsa_area *dsa,
--
2.40.1
--kqqpqghcwbcc3dt5
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0012-v7-Streaming-Read-API.patch"
^ permalink raw reply [nested|flat] 59+ messages in thread
* [PATCH v12 13/17] Separate TBM[Shared]Iterator and TBMIterateResult
@ 2024-02-16 02:23 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Melanie Plageman @ 2024-02-16 02:23 UTC (permalink / raw)
Remove the TBMIterateResult from the TBMIterator and TBMSharedIterator
and have tbm_[shared_]iterate() take a TBMIterateResult as a parameter.
This will allow multiple TBMIterateResults to exist concurrently
allowing asynchronous use of the TIDBitmap for prefetching, for example.
tbm_[shared]_iterate() now sets blockno to InvalidBlockNumber when the
bitmap is exhausted instead of returning NULL.
BitmapHeapScan callers of tbm_iterate make a TBMIterateResult locally
and pass it in.
Because GIN only needs a single TBMIterateResult, inline the matchResult
in the GinScanEntry to avoid having to separately manage memory for the
TBMIterateResult.
---
src/backend/access/gin/ginget.c | 48 +++++++++------
src/backend/access/gin/ginscan.c | 2 +-
src/backend/access/heap/heapam_handler.c | 30 +++++-----
src/backend/executor/nodeBitmapHeapscan.c | 47 ++++++++-------
src/backend/nodes/tidbitmap.c | 73 ++++++++++++-----------
src/include/access/gin_private.h | 2 +-
src/include/executor/nodeBitmapHeapscan.h | 2 +-
src/include/nodes/tidbitmap.h | 4 +-
8 files changed, 113 insertions(+), 95 deletions(-)
diff --git a/src/backend/access/gin/ginget.c b/src/backend/access/gin/ginget.c
index 0b4f2ebadb6..3aa457a29e1 100644
--- a/src/backend/access/gin/ginget.c
+++ b/src/backend/access/gin/ginget.c
@@ -332,10 +332,22 @@ restartScanEntry:
entry->list = NULL;
entry->nlist = 0;
entry->matchBitmap = NULL;
- entry->matchResult = NULL;
entry->reduceResult = false;
entry->predictNumberResult = 0;
+ /*
+ * MTODO: is it enough to set blockno to InvalidBlockNumber? In all the
+ * places were we previously set matchResult to NULL, I just set blockno
+ * to InvalidBlockNumber. It seems like this should be okay because that
+ * is usually what we check before using the matchResult members. But it
+ * might be safer to zero out the offsets array. But that is expensive.
+ */
+ entry->matchResult.blockno = InvalidBlockNumber;
+ entry->matchResult.ntuples = 0;
+ entry->matchResult.recheck = true;
+ memset(entry->matchResult.offsets, 0,
+ sizeof(OffsetNumber) * MaxHeapTuplesPerPage);
+
/*
* we should find entry, and begin scan of posting tree or just store
* posting list in memory
@@ -374,6 +386,7 @@ restartScanEntry:
{
if (entry->matchIterator)
tbm_end_iterate(entry->matchIterator);
+ entry->matchResult.blockno = InvalidBlockNumber;
entry->matchIterator = NULL;
tbm_free(entry->matchBitmap);
entry->matchBitmap = NULL;
@@ -823,18 +836,19 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
{
/*
* If we've exhausted all items on this block, move to next block
- * in the bitmap.
+ * in the bitmap. tbm_iterate() sets matchResult->blockno to
+ * InvalidBlockNumber when the bitmap is exhausted.
*/
- while (entry->matchResult == NULL ||
- (entry->matchResult->ntuples >= 0 &&
- entry->offset >= entry->matchResult->ntuples) ||
- entry->matchResult->blockno < advancePastBlk ||
+ while ((!BlockNumberIsValid(entry->matchResult.blockno)) ||
+ (entry->matchResult.ntuples >= 0 &&
+ entry->offset >= entry->matchResult.ntuples) ||
+ entry->matchResult.blockno < advancePastBlk ||
(ItemPointerIsLossyPage(&advancePast) &&
- entry->matchResult->blockno == advancePastBlk))
+ entry->matchResult.blockno == advancePastBlk))
{
- entry->matchResult = tbm_iterate(entry->matchIterator);
+ tbm_iterate(entry->matchIterator, &entry->matchResult);
- if (entry->matchResult == NULL)
+ if (!BlockNumberIsValid(entry->matchResult.blockno))
{
ItemPointerSetInvalid(&entry->curItem);
tbm_end_iterate(entry->matchIterator);
@@ -858,10 +872,10 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* We're now on the first page after advancePast which has any
* items on it. If it's a lossy result, return that.
*/
- if (entry->matchResult->ntuples < 0)
+ if (entry->matchResult.ntuples < 0)
{
ItemPointerSetLossyPage(&entry->curItem,
- entry->matchResult->blockno);
+ entry->matchResult.blockno);
/*
* We might as well fall out of the loop; we could not
@@ -875,27 +889,27 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* Not a lossy page. Skip over any offsets <= advancePast, and
* return that.
*/
- if (entry->matchResult->blockno == advancePastBlk)
+ if (entry->matchResult.blockno == advancePastBlk)
{
/*
* First, do a quick check against the last offset on the
* page. If that's > advancePast, so are all the other
* offsets, so just go back to the top to get the next page.
*/
- if (entry->matchResult->offsets[entry->matchResult->ntuples - 1] <= advancePastOff)
+ if (entry->matchResult.offsets[entry->matchResult.ntuples - 1] <= advancePastOff)
{
- entry->offset = entry->matchResult->ntuples;
+ entry->offset = entry->matchResult.ntuples;
continue;
}
/* Otherwise scan to find the first item > advancePast */
- while (entry->matchResult->offsets[entry->offset] <= advancePastOff)
+ while (entry->matchResult.offsets[entry->offset] <= advancePastOff)
entry->offset++;
}
ItemPointerSet(&entry->curItem,
- entry->matchResult->blockno,
- entry->matchResult->offsets[entry->offset]);
+ entry->matchResult.blockno,
+ entry->matchResult.offsets[entry->offset]);
entry->offset++;
/* Done unless we need to reduce the result */
diff --git a/src/backend/access/gin/ginscan.c b/src/backend/access/gin/ginscan.c
index af24d38544e..033d5253394 100644
--- a/src/backend/access/gin/ginscan.c
+++ b/src/backend/access/gin/ginscan.c
@@ -106,7 +106,7 @@ ginFillScanEntry(GinScanOpaque so, OffsetNumber attnum,
ItemPointerSetMin(&scanEntry->curItem);
scanEntry->matchBitmap = NULL;
scanEntry->matchIterator = NULL;
- scanEntry->matchResult = NULL;
+ scanEntry->matchResult.blockno = InvalidBlockNumber;
scanEntry->list = NULL;
scanEntry->nlist = 0;
scanEntry->offset = InvalidOffsetNumber;
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 46f1a374252..bd630e38fa8 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2207,7 +2207,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Buffer buffer;
Snapshot snapshot;
int ntup;
- TBMIterateResult *tbmres;
+ TBMIterateResult tbmres;
hscan->rs_cindex = 0;
hscan->rs_ntuples = 0;
@@ -2219,9 +2219,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
{
CHECK_FOR_INTERRUPTS();
- tbmres = bhs_iterate(scan->rs_bhs_iterator);
+ bhs_iterate(scan->rs_bhs_iterator, &tbmres);
- if (tbmres == NULL)
+ if (!BlockNumberIsValid(tbmres.blockno))
{
/* no more entries in the bitmap */
Assert(hscan->rs_empty_tuples_pending == 0);
@@ -2236,11 +2236,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* isolation though, as we need to examine all invisible tuples
* reachable by the index.
*/
- } while (!IsolationIsSerializable() && tbmres->blockno >= hscan->rs_nblocks);
+ } while (!IsolationIsSerializable() && tbmres.blockno >= hscan->rs_nblocks);
/* Got a valid block */
- *blockno = tbmres->blockno;
- *recheck = tbmres->recheck;
+ *blockno = tbmres.blockno;
+ *recheck = tbmres.recheck;
/*
* We can skip fetching the heap page if we don't need any fields from the
@@ -2248,19 +2248,19 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* page are visible to our transaction.
*/
if (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmres->recheck &&
- VM_ALL_VISIBLE(scan->rs_rd, tbmres->blockno, &hscan->rs_vmbuffer))
+ !tbmres.recheck &&
+ VM_ALL_VISIBLE(scan->rs_rd, tbmres.blockno, &hscan->rs_vmbuffer))
{
/* can't be lossy in the skip_fetch case */
- Assert(tbmres->ntuples >= 0);
+ Assert(tbmres.ntuples >= 0);
Assert(hscan->rs_empty_tuples_pending >= 0);
- hscan->rs_empty_tuples_pending += tbmres->ntuples;
+ hscan->rs_empty_tuples_pending += tbmres.ntuples;
return true;
}
- block = tbmres->blockno;
+ block = tbmres.blockno;
/*
* Acquire pin on the target heap page, trading in any pin we held before.
@@ -2289,7 +2289,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
/*
* We need two separate strategies for lossy and non-lossy cases.
*/
- if (tbmres->ntuples >= 0)
+ if (tbmres.ntuples >= 0)
{
/*
* Bitmap is non-lossy, so we just look through the offsets listed in
@@ -2298,9 +2298,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
*/
int curslot;
- for (curslot = 0; curslot < tbmres->ntuples; curslot++)
+ for (curslot = 0; curslot < tbmres.ntuples; curslot++)
{
- OffsetNumber offnum = tbmres->offsets[curslot];
+ OffsetNumber offnum = tbmres.offsets[curslot];
ItemPointerData tid;
HeapTupleData heapTuple;
@@ -2350,7 +2350,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
- if (tbmres->ntuples < 0)
+ if (tbmres.ntuples < 0)
(*lossy_pages)++;
else
(*exact_pages)++;
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index fb79f57d7a6..d61965a2761 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -77,15 +77,16 @@ bhs_begin_iterate(TIDBitmap *tbm, dsa_pointer shared_area, dsa_area *personal_ar
return result;
}
-TBMIterateResult *
-bhs_iterate(BitmapHeapIterator *iterator)
+void
+bhs_iterate(BitmapHeapIterator *iterator, TBMIterateResult *result)
{
Assert(iterator);
+ Assert(result);
if (iterator->serial)
- return tbm_iterate(iterator->serial);
+ tbm_iterate(iterator->serial, result);
else
- return tbm_shared_iterate(iterator->parallel);
+ tbm_shared_iterate(iterator->parallel, result);
}
void
@@ -348,7 +349,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
#ifdef USE_PREFETCH
ParallelBitmapHeapState *pstate = node->pstate;
BitmapHeapIterator *prefetch_iterator = node->pf_iterator;
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
if (pstate == NULL)
{
@@ -360,8 +361,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
else if (prefetch_iterator)
{
/* Do not let the prefetch iterator get behind the main one */
- tbmpre = bhs_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ bhs_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
return;
}
@@ -394,8 +395,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
*/
if (prefetch_iterator)
{
- tbmpre = bhs_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ bhs_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
}
}
@@ -462,10 +463,12 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (node->prefetch_pages < node->prefetch_target)
{
- TBMIterateResult *tbmpre = bhs_iterate(prefetch_iterator);
+ TBMIterateResult tbmpre;
bool skip_fetch;
- if (tbmpre == NULL)
+ bhs_iterate(prefetch_iterator, &tbmpre);
+
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
bhs_end_iterate(prefetch_iterator);
@@ -473,7 +476,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
node->prefetch_pages++;
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/*
* If we expect not to have to actually read this heap page,
@@ -482,13 +485,13 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
* prefetch_pages?)
*/
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
@@ -501,7 +504,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (1)
{
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
bool do_prefetch = false;
bool skip_fetch;
@@ -520,8 +523,8 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
if (!do_prefetch)
return;
- tbmpre = bhs_iterate(prefetch_iterator);
- if (tbmpre == NULL)
+ bhs_iterate(prefetch_iterator, &tbmpre);
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
bhs_end_iterate(prefetch_iterator);
@@ -529,17 +532,17 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/* As above, skip prefetch if we expect not to need page */
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
}
diff --git a/src/backend/nodes/tidbitmap.c b/src/backend/nodes/tidbitmap.c
index 1dc4c99bf99..309a44bdb84 100644
--- a/src/backend/nodes/tidbitmap.c
+++ b/src/backend/nodes/tidbitmap.c
@@ -172,7 +172,6 @@ struct TBMIterator
int spageptr; /* next spages index */
int schunkptr; /* next schunks index */
int schunkbit; /* next bit to check in current schunk */
- TBMIterateResult output;
};
/*
@@ -213,7 +212,6 @@ struct TBMSharedIterator
PTEntryArray *ptbase; /* pagetable element array */
PTIterationArray *ptpages; /* sorted exact page index list */
PTIterationArray *ptchunks; /* sorted lossy page index list */
- TBMIterateResult output;
};
/* Local function prototypes */
@@ -944,20 +942,21 @@ tbm_advance_schunkbit(PagetableEntry *chunk, int *schunkbitp)
/*
* tbm_iterate - scan through next page of a TIDBitmap
*
- * Returns a TBMIterateResult representing one page, or NULL if there are
- * no more pages to scan. Pages are guaranteed to be delivered in numerical
- * order. If result->ntuples < 0, then the bitmap is "lossy" and failed to
- * remember the exact tuples to look at on this page --- the caller must
- * examine all tuples on the page and check if they meet the intended
- * condition. If result->recheck is true, only the indicated tuples need
- * be examined, but the condition must be rechecked anyway. (For ease of
- * testing, recheck is always set true when ntuples < 0.)
+ * Caller must pass in a TBMIterateResult to be filled.
+ *
+ * Pages are guaranteed to be delivered in numerical order. tbmres->blockno is
+ * set to InvalidBlockNumber when there are no more pages to scan. If
+ * tbmres->ntuples < 0, then the bitmap is "lossy" and failed to remember the
+ * exact tuples to look at on this page --- the caller must examine all tuples
+ * on the page and check if they meet the intended condition. If
+ * tbmres->recheck is true, only the indicated tuples need be examined, but the
+ * condition must be rechecked anyway. (For ease of testing, recheck is always
+ * set true when ntuples < 0.)
*/
-TBMIterateResult *
-tbm_iterate(TBMIterator *iterator)
+void
+tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres)
{
TIDBitmap *tbm = iterator->tbm;
- TBMIterateResult *output = &(iterator->output);
Assert(tbm->iterating == TBM_ITERATING_PRIVATE);
@@ -985,6 +984,7 @@ tbm_iterate(TBMIterator *iterator)
* If both chunk and per-page data remain, must output the numerically
* earlier page.
*/
+ Assert(tbmres);
if (iterator->schunkptr < tbm->nchunks)
{
PagetableEntry *chunk = tbm->schunks[iterator->schunkptr];
@@ -995,11 +995,11 @@ tbm_iterate(TBMIterator *iterator)
chunk_blockno < tbm->spages[iterator->spageptr]->blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
iterator->schunkbit++;
- return output;
+ return;
}
}
@@ -1015,16 +1015,17 @@ tbm_iterate(TBMIterator *iterator)
page = tbm->spages[iterator->spageptr];
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
iterator->spageptr++;
- return output;
+ return;
}
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
@@ -1034,10 +1035,9 @@ tbm_iterate(TBMIterator *iterator)
* across multiple processes. We need to acquire the iterator LWLock,
* before accessing the shared members.
*/
-TBMIterateResult *
-tbm_shared_iterate(TBMSharedIterator *iterator)
+void
+tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres)
{
- TBMIterateResult *output = &iterator->output;
TBMSharedIteratorState *istate = iterator->state;
PagetableEntry *ptbase = NULL;
int *idxpages = NULL;
@@ -1088,13 +1088,13 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
chunk_blockno < ptbase[idxpages[istate->spageptr]].blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
istate->schunkbit++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
}
@@ -1104,21 +1104,22 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
int ntuples;
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
istate->spageptr++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
LWLockRelease(&istate->lock);
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 3013a44bae1..3b432263bb0 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -353,7 +353,7 @@ typedef struct GinScanEntryData
/* for a partial-match or full-scan query, we accumulate all TIDs here */
TIDBitmap *matchBitmap;
TBMIterator *matchIterator;
- TBMIterateResult *matchResult;
+ TBMIterateResult matchResult;
/* used for Posting list and one page in Posting tree */
ItemPointerData *list;
diff --git a/src/include/executor/nodeBitmapHeapscan.h b/src/include/executor/nodeBitmapHeapscan.h
index cb56d20dc6f..3c330f86e62 100644
--- a/src/include/executor/nodeBitmapHeapscan.h
+++ b/src/include/executor/nodeBitmapHeapscan.h
@@ -34,7 +34,7 @@ typedef struct BitmapHeapIterator
struct TBMSharedIterator *parallel;
} BitmapHeapIterator;
-extern TBMIterateResult *bhs_iterate(BitmapHeapIterator *iterator);
+extern void bhs_iterate(BitmapHeapIterator *iterator, TBMIterateResult *result);
extern void bhs_end_iterate(BitmapHeapIterator *iterator);
diff --git a/src/include/nodes/tidbitmap.h b/src/include/nodes/tidbitmap.h
index 432fae52962..f000c1af28f 100644
--- a/src/include/nodes/tidbitmap.h
+++ b/src/include/nodes/tidbitmap.h
@@ -72,8 +72,8 @@ extern bool tbm_is_empty(const TIDBitmap *tbm);
extern TBMIterator *tbm_begin_iterate(TIDBitmap *tbm);
extern dsa_pointer tbm_prepare_shared_iterate(TIDBitmap *tbm);
-extern TBMIterateResult *tbm_iterate(TBMIterator *iterator);
-extern TBMIterateResult *tbm_shared_iterate(TBMSharedIterator *iterator);
+extern void tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres);
+extern void tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres);
extern void tbm_end_iterate(TBMIterator *iterator);
extern void tbm_end_shared_iterate(TBMSharedIterator *iterator);
extern TBMSharedIterator *tbm_attach_shared_iterate(dsa_area *dsa,
--
2.40.1
--6jpz2j246qmht4bt
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v12-0014-Push-BitmapHeapScan-prefetch-code-into-heapam.c.patch"
^ permalink raw reply [nested|flat] 59+ messages in thread
* [PATCH v6 12/14] Separate TBM[Shared]Iterator and TBMIterateResult
@ 2024-02-16 02:23 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Melanie Plageman @ 2024-02-16 02:23 UTC (permalink / raw)
Remove the TBMIterateResult from the TBMIterator and TBMSharedIterator
and have tbm_[shared_]iterate() take a TBMIterateResult as a parameter.
This will allow multiple TBMIterateResults to exist concurrently
allowing asynchronous use of the TIDBitmap for prefetching, for example.
tbm_[shared]_iterate() now sets blockno to InvalidBlockNumber when the
bitmap is exhausted instead of returning NULL.
BitmapHeapScan callers of tbm_iterate make a TBMIterateResult locally
and pass it in.
Because GIN only needs a single TBMIterateResult, inline the matchResult
in the GinScanEntry to avoid having to separately manage memory for the
TBMIterateResult.
---
src/backend/access/gin/ginget.c | 48 +++++++++------
src/backend/access/gin/ginscan.c | 2 +-
src/backend/access/heap/heapam_handler.c | 32 +++++-----
src/backend/executor/nodeBitmapHeapscan.c | 38 ++++++------
src/backend/nodes/tidbitmap.c | 73 ++++++++++++-----------
src/include/access/gin_private.h | 2 +-
src/include/nodes/tidbitmap.h | 4 +-
7 files changed, 108 insertions(+), 91 deletions(-)
diff --git a/src/backend/access/gin/ginget.c b/src/backend/access/gin/ginget.c
index 0b4f2ebadb6..3aa457a29e1 100644
--- a/src/backend/access/gin/ginget.c
+++ b/src/backend/access/gin/ginget.c
@@ -332,10 +332,22 @@ restartScanEntry:
entry->list = NULL;
entry->nlist = 0;
entry->matchBitmap = NULL;
- entry->matchResult = NULL;
entry->reduceResult = false;
entry->predictNumberResult = 0;
+ /*
+ * MTODO: is it enough to set blockno to InvalidBlockNumber? In all the
+ * places were we previously set matchResult to NULL, I just set blockno
+ * to InvalidBlockNumber. It seems like this should be okay because that
+ * is usually what we check before using the matchResult members. But it
+ * might be safer to zero out the offsets array. But that is expensive.
+ */
+ entry->matchResult.blockno = InvalidBlockNumber;
+ entry->matchResult.ntuples = 0;
+ entry->matchResult.recheck = true;
+ memset(entry->matchResult.offsets, 0,
+ sizeof(OffsetNumber) * MaxHeapTuplesPerPage);
+
/*
* we should find entry, and begin scan of posting tree or just store
* posting list in memory
@@ -374,6 +386,7 @@ restartScanEntry:
{
if (entry->matchIterator)
tbm_end_iterate(entry->matchIterator);
+ entry->matchResult.blockno = InvalidBlockNumber;
entry->matchIterator = NULL;
tbm_free(entry->matchBitmap);
entry->matchBitmap = NULL;
@@ -823,18 +836,19 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
{
/*
* If we've exhausted all items on this block, move to next block
- * in the bitmap.
+ * in the bitmap. tbm_iterate() sets matchResult->blockno to
+ * InvalidBlockNumber when the bitmap is exhausted.
*/
- while (entry->matchResult == NULL ||
- (entry->matchResult->ntuples >= 0 &&
- entry->offset >= entry->matchResult->ntuples) ||
- entry->matchResult->blockno < advancePastBlk ||
+ while ((!BlockNumberIsValid(entry->matchResult.blockno)) ||
+ (entry->matchResult.ntuples >= 0 &&
+ entry->offset >= entry->matchResult.ntuples) ||
+ entry->matchResult.blockno < advancePastBlk ||
(ItemPointerIsLossyPage(&advancePast) &&
- entry->matchResult->blockno == advancePastBlk))
+ entry->matchResult.blockno == advancePastBlk))
{
- entry->matchResult = tbm_iterate(entry->matchIterator);
+ tbm_iterate(entry->matchIterator, &entry->matchResult);
- if (entry->matchResult == NULL)
+ if (!BlockNumberIsValid(entry->matchResult.blockno))
{
ItemPointerSetInvalid(&entry->curItem);
tbm_end_iterate(entry->matchIterator);
@@ -858,10 +872,10 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* We're now on the first page after advancePast which has any
* items on it. If it's a lossy result, return that.
*/
- if (entry->matchResult->ntuples < 0)
+ if (entry->matchResult.ntuples < 0)
{
ItemPointerSetLossyPage(&entry->curItem,
- entry->matchResult->blockno);
+ entry->matchResult.blockno);
/*
* We might as well fall out of the loop; we could not
@@ -875,27 +889,27 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* Not a lossy page. Skip over any offsets <= advancePast, and
* return that.
*/
- if (entry->matchResult->blockno == advancePastBlk)
+ if (entry->matchResult.blockno == advancePastBlk)
{
/*
* First, do a quick check against the last offset on the
* page. If that's > advancePast, so are all the other
* offsets, so just go back to the top to get the next page.
*/
- if (entry->matchResult->offsets[entry->matchResult->ntuples - 1] <= advancePastOff)
+ if (entry->matchResult.offsets[entry->matchResult.ntuples - 1] <= advancePastOff)
{
- entry->offset = entry->matchResult->ntuples;
+ entry->offset = entry->matchResult.ntuples;
continue;
}
/* Otherwise scan to find the first item > advancePast */
- while (entry->matchResult->offsets[entry->offset] <= advancePastOff)
+ while (entry->matchResult.offsets[entry->offset] <= advancePastOff)
entry->offset++;
}
ItemPointerSet(&entry->curItem,
- entry->matchResult->blockno,
- entry->matchResult->offsets[entry->offset]);
+ entry->matchResult.blockno,
+ entry->matchResult.offsets[entry->offset]);
entry->offset++;
/* Done unless we need to reduce the result */
diff --git a/src/backend/access/gin/ginscan.c b/src/backend/access/gin/ginscan.c
index af24d38544e..033d5253394 100644
--- a/src/backend/access/gin/ginscan.c
+++ b/src/backend/access/gin/ginscan.c
@@ -106,7 +106,7 @@ ginFillScanEntry(GinScanOpaque so, OffsetNumber attnum,
ItemPointerSetMin(&scanEntry->curItem);
scanEntry->matchBitmap = NULL;
scanEntry->matchIterator = NULL;
- scanEntry->matchResult = NULL;
+ scanEntry->matchResult.blockno = InvalidBlockNumber;
scanEntry->list = NULL;
scanEntry->nlist = 0;
scanEntry->offset = InvalidOffsetNumber;
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index e038e60cd8f..022753e203a 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2119,7 +2119,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Buffer buffer;
Snapshot snapshot;
int ntup;
- TBMIterateResult *tbmres;
+ TBMIterateResult tbmres;
hscan->rs_cindex = 0;
hscan->rs_ntuples = 0;
@@ -2132,11 +2132,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
CHECK_FOR_INTERRUPTS();
if (scan->shared_tbmiterator)
- tbmres = tbm_shared_iterate(scan->shared_tbmiterator);
+ tbm_shared_iterate(scan->shared_tbmiterator, &tbmres);
else
- tbmres = tbm_iterate(scan->tbmiterator);
+ tbm_iterate(scan->tbmiterator, &tbmres);
- if (tbmres == NULL)
+ if (!BlockNumberIsValid(tbmres.blockno))
{
/* no more entries in the bitmap */
Assert(hscan->rs_empty_tuples_pending == 0);
@@ -2151,11 +2151,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* isolation though, as we need to examine all invisible tuples
* reachable by the index.
*/
- } while (!IsolationIsSerializable() && tbmres->blockno >= hscan->rs_nblocks);
+ } while (!IsolationIsSerializable() && tbmres.blockno >= hscan->rs_nblocks);
/* Got a valid block */
- *blockno = tbmres->blockno;
- *recheck = tbmres->recheck;
+ *blockno = tbmres.blockno;
+ *recheck = tbmres.recheck;
/*
* We can skip fetching the heap page if we don't need any fields from the
@@ -2163,19 +2163,19 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* the page are visible to our transaction.
*/
if (scan->rs_flags & SO_CAN_SKIP_FETCH &&
- !tbmres->recheck &&
- VM_ALL_VISIBLE(scan->rs_rd, tbmres->blockno, &hscan->rs_vmbuffer))
+ !tbmres.recheck &&
+ VM_ALL_VISIBLE(scan->rs_rd, tbmres.blockno, &hscan->rs_vmbuffer))
{
/* can't be lossy in the skip_fetch case */
- Assert(tbmres->ntuples >= 0);
+ Assert(tbmres.ntuples >= 0);
Assert(hscan->rs_empty_tuples_pending >= 0);
- hscan->rs_empty_tuples_pending += tbmres->ntuples;
+ hscan->rs_empty_tuples_pending += tbmres.ntuples;
return true;
}
- block = tbmres->blockno;
+ block = tbmres.blockno;
/*
* Acquire pin on the target heap page, trading in any pin we held before.
@@ -2204,7 +2204,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
/*
* We need two separate strategies for lossy and non-lossy cases.
*/
- if (tbmres->ntuples >= 0)
+ if (tbmres.ntuples >= 0)
{
/*
* Bitmap is non-lossy, so we just look through the offsets listed in
@@ -2213,9 +2213,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
*/
int curslot;
- for (curslot = 0; curslot < tbmres->ntuples; curslot++)
+ for (curslot = 0; curslot < tbmres.ntuples; curslot++)
{
- OffsetNumber offnum = tbmres->offsets[curslot];
+ OffsetNumber offnum = tbmres.offsets[curslot];
ItemPointerData tid;
HeapTupleData heapTuple;
@@ -2265,7 +2265,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
- *lossy = tbmres->ntuples < 0;
+ *lossy = tbmres.ntuples < 0;
/*
* Return true to indicate that a valid block was found and the bitmap is
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 0924613b247..b3b9448627b 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -320,7 +320,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
{
#ifdef USE_PREFETCH
ParallelBitmapHeapState *pstate = node->pstate;
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
if (pstate == NULL)
{
@@ -334,8 +334,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
else if (prefetch_iterator)
{
/* Do not let the prefetch iterator get behind the main one */
- tbmpre = tbm_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
return;
}
@@ -370,8 +370,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
*/
if (prefetch_iterator)
{
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
}
}
@@ -439,10 +439,12 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (node->prefetch_pages < node->prefetch_target)
{
- TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator);
+ TBMIterateResult tbmpre;
bool skip_fetch;
- if (tbmpre == NULL)
+ tbm_iterate(prefetch_iterator, &tbmpre);
+
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_iterate(prefetch_iterator);
@@ -450,7 +452,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
node->prefetch_pages++;
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/*
* If we expect not to have to actually read this heap page,
@@ -459,13 +461,13 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
* prefetch_pages?)
*/
skip_fetch = (scan->rs_flags & SO_CAN_SKIP_FETCH &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
@@ -480,7 +482,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (1)
{
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
bool do_prefetch = false;
bool skip_fetch;
@@ -499,8 +501,8 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
if (!do_prefetch)
return;
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- if (tbmpre == NULL)
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_shared_iterate(prefetch_iterator);
@@ -508,17 +510,17 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/* As above, skip prefetch if we expect not to need page */
skip_fetch = (scan->rs_flags & SO_CAN_SKIP_FETCH &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
}
diff --git a/src/backend/nodes/tidbitmap.c b/src/backend/nodes/tidbitmap.c
index d2bf8f44d50..7d038c2018d 100644
--- a/src/backend/nodes/tidbitmap.c
+++ b/src/backend/nodes/tidbitmap.c
@@ -172,7 +172,6 @@ struct TBMIterator
int spageptr; /* next spages index */
int schunkptr; /* next schunks index */
int schunkbit; /* next bit to check in current schunk */
- TBMIterateResult output;
};
/*
@@ -213,7 +212,6 @@ struct TBMSharedIterator
PTEntryArray *ptbase; /* pagetable element array */
PTIterationArray *ptpages; /* sorted exact page index list */
PTIterationArray *ptchunks; /* sorted lossy page index list */
- TBMIterateResult output;
};
/* Local function prototypes */
@@ -944,20 +942,21 @@ tbm_advance_schunkbit(PagetableEntry *chunk, int *schunkbitp)
/*
* tbm_iterate - scan through next page of a TIDBitmap
*
- * Returns a TBMIterateResult representing one page, or NULL if there are
- * no more pages to scan. Pages are guaranteed to be delivered in numerical
- * order. If result->ntuples < 0, then the bitmap is "lossy" and failed to
- * remember the exact tuples to look at on this page --- the caller must
- * examine all tuples on the page and check if they meet the intended
- * condition. If result->recheck is true, only the indicated tuples need
- * be examined, but the condition must be rechecked anyway. (For ease of
- * testing, recheck is always set true when ntuples < 0.)
+ * Caller must pass in a TBMIterateResult to be filled.
+ *
+ * Pages are guaranteed to be delivered in numerical order. tbmres->blockno is
+ * set to InvalidBlockNumber when there are no more pages to scan. If
+ * tbmres->ntuples < 0, then the bitmap is "lossy" and failed to remember the
+ * exact tuples to look at on this page --- the caller must examine all tuples
+ * on the page and check if they meet the intended condition. If
+ * tbmres->recheck is true, only the indicated tuples need be examined, but the
+ * condition must be rechecked anyway. (For ease of testing, recheck is always
+ * set true when ntuples < 0.)
*/
-TBMIterateResult *
-tbm_iterate(TBMIterator *iterator)
+void
+tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres)
{
TIDBitmap *tbm = iterator->tbm;
- TBMIterateResult *output = &(iterator->output);
Assert(tbm->iterating == TBM_ITERATING_PRIVATE);
@@ -985,6 +984,7 @@ tbm_iterate(TBMIterator *iterator)
* If both chunk and per-page data remain, must output the numerically
* earlier page.
*/
+ Assert(tbmres);
if (iterator->schunkptr < tbm->nchunks)
{
PagetableEntry *chunk = tbm->schunks[iterator->schunkptr];
@@ -995,11 +995,11 @@ tbm_iterate(TBMIterator *iterator)
chunk_blockno < tbm->spages[iterator->spageptr]->blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
iterator->schunkbit++;
- return output;
+ return;
}
}
@@ -1015,16 +1015,17 @@ tbm_iterate(TBMIterator *iterator)
page = tbm->spages[iterator->spageptr];
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
iterator->spageptr++;
- return output;
+ return;
}
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
@@ -1034,10 +1035,9 @@ tbm_iterate(TBMIterator *iterator)
* across multiple processes. We need to acquire the iterator LWLock,
* before accessing the shared members.
*/
-TBMIterateResult *
-tbm_shared_iterate(TBMSharedIterator *iterator)
+void
+tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres)
{
- TBMIterateResult *output = &iterator->output;
TBMSharedIteratorState *istate = iterator->state;
PagetableEntry *ptbase = NULL;
int *idxpages = NULL;
@@ -1088,13 +1088,13 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
chunk_blockno < ptbase[idxpages[istate->spageptr]].blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
istate->schunkbit++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
}
@@ -1104,21 +1104,22 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
int ntuples;
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
istate->spageptr++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
LWLockRelease(&istate->lock);
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 3013a44bae1..3b432263bb0 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -353,7 +353,7 @@ typedef struct GinScanEntryData
/* for a partial-match or full-scan query, we accumulate all TIDs here */
TIDBitmap *matchBitmap;
TBMIterator *matchIterator;
- TBMIterateResult *matchResult;
+ TBMIterateResult matchResult;
/* used for Posting list and one page in Posting tree */
ItemPointerData *list;
diff --git a/src/include/nodes/tidbitmap.h b/src/include/nodes/tidbitmap.h
index 432fae52962..f000c1af28f 100644
--- a/src/include/nodes/tidbitmap.h
+++ b/src/include/nodes/tidbitmap.h
@@ -72,8 +72,8 @@ extern bool tbm_is_empty(const TIDBitmap *tbm);
extern TBMIterator *tbm_begin_iterate(TIDBitmap *tbm);
extern dsa_pointer tbm_prepare_shared_iterate(TIDBitmap *tbm);
-extern TBMIterateResult *tbm_iterate(TBMIterator *iterator);
-extern TBMIterateResult *tbm_shared_iterate(TBMSharedIterator *iterator);
+extern void tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres);
+extern void tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres);
extern void tbm_end_iterate(TBMIterator *iterator);
extern void tbm_end_shared_iterate(TBMSharedIterator *iterator);
extern TBMSharedIterator *tbm_attach_shared_iterate(dsa_area *dsa,
--
2.40.1
--w4wcjcocxsm37usi
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0013-v7-Streaming-Read-API.patch"
^ permalink raw reply [nested|flat] 59+ messages in thread
* [PATCH v6 12/14] Separate TBM[Shared]Iterator and TBMIterateResult
@ 2024-02-16 02:23 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Melanie Plageman @ 2024-02-16 02:23 UTC (permalink / raw)
Remove the TBMIterateResult from the TBMIterator and TBMSharedIterator
and have tbm_[shared_]iterate() take a TBMIterateResult as a parameter.
This will allow multiple TBMIterateResults to exist concurrently
allowing asynchronous use of the TIDBitmap for prefetching, for example.
tbm_[shared]_iterate() now sets blockno to InvalidBlockNumber when the
bitmap is exhausted instead of returning NULL.
BitmapHeapScan callers of tbm_iterate make a TBMIterateResult locally
and pass it in.
Because GIN only needs a single TBMIterateResult, inline the matchResult
in the GinScanEntry to avoid having to separately manage memory for the
TBMIterateResult.
---
src/backend/access/gin/ginget.c | 48 +++++++++------
src/backend/access/gin/ginscan.c | 2 +-
src/backend/access/heap/heapam_handler.c | 32 +++++-----
src/backend/executor/nodeBitmapHeapscan.c | 38 ++++++------
src/backend/nodes/tidbitmap.c | 73 ++++++++++++-----------
src/include/access/gin_private.h | 2 +-
src/include/nodes/tidbitmap.h | 4 +-
7 files changed, 108 insertions(+), 91 deletions(-)
diff --git a/src/backend/access/gin/ginget.c b/src/backend/access/gin/ginget.c
index 0b4f2ebadb6..3aa457a29e1 100644
--- a/src/backend/access/gin/ginget.c
+++ b/src/backend/access/gin/ginget.c
@@ -332,10 +332,22 @@ restartScanEntry:
entry->list = NULL;
entry->nlist = 0;
entry->matchBitmap = NULL;
- entry->matchResult = NULL;
entry->reduceResult = false;
entry->predictNumberResult = 0;
+ /*
+ * MTODO: is it enough to set blockno to InvalidBlockNumber? In all the
+ * places were we previously set matchResult to NULL, I just set blockno
+ * to InvalidBlockNumber. It seems like this should be okay because that
+ * is usually what we check before using the matchResult members. But it
+ * might be safer to zero out the offsets array. But that is expensive.
+ */
+ entry->matchResult.blockno = InvalidBlockNumber;
+ entry->matchResult.ntuples = 0;
+ entry->matchResult.recheck = true;
+ memset(entry->matchResult.offsets, 0,
+ sizeof(OffsetNumber) * MaxHeapTuplesPerPage);
+
/*
* we should find entry, and begin scan of posting tree or just store
* posting list in memory
@@ -374,6 +386,7 @@ restartScanEntry:
{
if (entry->matchIterator)
tbm_end_iterate(entry->matchIterator);
+ entry->matchResult.blockno = InvalidBlockNumber;
entry->matchIterator = NULL;
tbm_free(entry->matchBitmap);
entry->matchBitmap = NULL;
@@ -823,18 +836,19 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
{
/*
* If we've exhausted all items on this block, move to next block
- * in the bitmap.
+ * in the bitmap. tbm_iterate() sets matchResult->blockno to
+ * InvalidBlockNumber when the bitmap is exhausted.
*/
- while (entry->matchResult == NULL ||
- (entry->matchResult->ntuples >= 0 &&
- entry->offset >= entry->matchResult->ntuples) ||
- entry->matchResult->blockno < advancePastBlk ||
+ while ((!BlockNumberIsValid(entry->matchResult.blockno)) ||
+ (entry->matchResult.ntuples >= 0 &&
+ entry->offset >= entry->matchResult.ntuples) ||
+ entry->matchResult.blockno < advancePastBlk ||
(ItemPointerIsLossyPage(&advancePast) &&
- entry->matchResult->blockno == advancePastBlk))
+ entry->matchResult.blockno == advancePastBlk))
{
- entry->matchResult = tbm_iterate(entry->matchIterator);
+ tbm_iterate(entry->matchIterator, &entry->matchResult);
- if (entry->matchResult == NULL)
+ if (!BlockNumberIsValid(entry->matchResult.blockno))
{
ItemPointerSetInvalid(&entry->curItem);
tbm_end_iterate(entry->matchIterator);
@@ -858,10 +872,10 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* We're now on the first page after advancePast which has any
* items on it. If it's a lossy result, return that.
*/
- if (entry->matchResult->ntuples < 0)
+ if (entry->matchResult.ntuples < 0)
{
ItemPointerSetLossyPage(&entry->curItem,
- entry->matchResult->blockno);
+ entry->matchResult.blockno);
/*
* We might as well fall out of the loop; we could not
@@ -875,27 +889,27 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* Not a lossy page. Skip over any offsets <= advancePast, and
* return that.
*/
- if (entry->matchResult->blockno == advancePastBlk)
+ if (entry->matchResult.blockno == advancePastBlk)
{
/*
* First, do a quick check against the last offset on the
* page. If that's > advancePast, so are all the other
* offsets, so just go back to the top to get the next page.
*/
- if (entry->matchResult->offsets[entry->matchResult->ntuples - 1] <= advancePastOff)
+ if (entry->matchResult.offsets[entry->matchResult.ntuples - 1] <= advancePastOff)
{
- entry->offset = entry->matchResult->ntuples;
+ entry->offset = entry->matchResult.ntuples;
continue;
}
/* Otherwise scan to find the first item > advancePast */
- while (entry->matchResult->offsets[entry->offset] <= advancePastOff)
+ while (entry->matchResult.offsets[entry->offset] <= advancePastOff)
entry->offset++;
}
ItemPointerSet(&entry->curItem,
- entry->matchResult->blockno,
- entry->matchResult->offsets[entry->offset]);
+ entry->matchResult.blockno,
+ entry->matchResult.offsets[entry->offset]);
entry->offset++;
/* Done unless we need to reduce the result */
diff --git a/src/backend/access/gin/ginscan.c b/src/backend/access/gin/ginscan.c
index af24d38544e..033d5253394 100644
--- a/src/backend/access/gin/ginscan.c
+++ b/src/backend/access/gin/ginscan.c
@@ -106,7 +106,7 @@ ginFillScanEntry(GinScanOpaque so, OffsetNumber attnum,
ItemPointerSetMin(&scanEntry->curItem);
scanEntry->matchBitmap = NULL;
scanEntry->matchIterator = NULL;
- scanEntry->matchResult = NULL;
+ scanEntry->matchResult.blockno = InvalidBlockNumber;
scanEntry->list = NULL;
scanEntry->nlist = 0;
scanEntry->offset = InvalidOffsetNumber;
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index e038e60cd8f..022753e203a 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2119,7 +2119,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Buffer buffer;
Snapshot snapshot;
int ntup;
- TBMIterateResult *tbmres;
+ TBMIterateResult tbmres;
hscan->rs_cindex = 0;
hscan->rs_ntuples = 0;
@@ -2132,11 +2132,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
CHECK_FOR_INTERRUPTS();
if (scan->shared_tbmiterator)
- tbmres = tbm_shared_iterate(scan->shared_tbmiterator);
+ tbm_shared_iterate(scan->shared_tbmiterator, &tbmres);
else
- tbmres = tbm_iterate(scan->tbmiterator);
+ tbm_iterate(scan->tbmiterator, &tbmres);
- if (tbmres == NULL)
+ if (!BlockNumberIsValid(tbmres.blockno))
{
/* no more entries in the bitmap */
Assert(hscan->rs_empty_tuples_pending == 0);
@@ -2151,11 +2151,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* isolation though, as we need to examine all invisible tuples
* reachable by the index.
*/
- } while (!IsolationIsSerializable() && tbmres->blockno >= hscan->rs_nblocks);
+ } while (!IsolationIsSerializable() && tbmres.blockno >= hscan->rs_nblocks);
/* Got a valid block */
- *blockno = tbmres->blockno;
- *recheck = tbmres->recheck;
+ *blockno = tbmres.blockno;
+ *recheck = tbmres.recheck;
/*
* We can skip fetching the heap page if we don't need any fields from the
@@ -2163,19 +2163,19 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* the page are visible to our transaction.
*/
if (scan->rs_flags & SO_CAN_SKIP_FETCH &&
- !tbmres->recheck &&
- VM_ALL_VISIBLE(scan->rs_rd, tbmres->blockno, &hscan->rs_vmbuffer))
+ !tbmres.recheck &&
+ VM_ALL_VISIBLE(scan->rs_rd, tbmres.blockno, &hscan->rs_vmbuffer))
{
/* can't be lossy in the skip_fetch case */
- Assert(tbmres->ntuples >= 0);
+ Assert(tbmres.ntuples >= 0);
Assert(hscan->rs_empty_tuples_pending >= 0);
- hscan->rs_empty_tuples_pending += tbmres->ntuples;
+ hscan->rs_empty_tuples_pending += tbmres.ntuples;
return true;
}
- block = tbmres->blockno;
+ block = tbmres.blockno;
/*
* Acquire pin on the target heap page, trading in any pin we held before.
@@ -2204,7 +2204,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
/*
* We need two separate strategies for lossy and non-lossy cases.
*/
- if (tbmres->ntuples >= 0)
+ if (tbmres.ntuples >= 0)
{
/*
* Bitmap is non-lossy, so we just look through the offsets listed in
@@ -2213,9 +2213,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
*/
int curslot;
- for (curslot = 0; curslot < tbmres->ntuples; curslot++)
+ for (curslot = 0; curslot < tbmres.ntuples; curslot++)
{
- OffsetNumber offnum = tbmres->offsets[curslot];
+ OffsetNumber offnum = tbmres.offsets[curslot];
ItemPointerData tid;
HeapTupleData heapTuple;
@@ -2265,7 +2265,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
- *lossy = tbmres->ntuples < 0;
+ *lossy = tbmres.ntuples < 0;
/*
* Return true to indicate that a valid block was found and the bitmap is
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 0924613b247..b3b9448627b 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -320,7 +320,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
{
#ifdef USE_PREFETCH
ParallelBitmapHeapState *pstate = node->pstate;
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
if (pstate == NULL)
{
@@ -334,8 +334,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
else if (prefetch_iterator)
{
/* Do not let the prefetch iterator get behind the main one */
- tbmpre = tbm_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
return;
}
@@ -370,8 +370,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
*/
if (prefetch_iterator)
{
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
}
}
@@ -439,10 +439,12 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (node->prefetch_pages < node->prefetch_target)
{
- TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator);
+ TBMIterateResult tbmpre;
bool skip_fetch;
- if (tbmpre == NULL)
+ tbm_iterate(prefetch_iterator, &tbmpre);
+
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_iterate(prefetch_iterator);
@@ -450,7 +452,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
node->prefetch_pages++;
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/*
* If we expect not to have to actually read this heap page,
@@ -459,13 +461,13 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
* prefetch_pages?)
*/
skip_fetch = (scan->rs_flags & SO_CAN_SKIP_FETCH &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
@@ -480,7 +482,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (1)
{
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
bool do_prefetch = false;
bool skip_fetch;
@@ -499,8 +501,8 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
if (!do_prefetch)
return;
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- if (tbmpre == NULL)
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_shared_iterate(prefetch_iterator);
@@ -508,17 +510,17 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/* As above, skip prefetch if we expect not to need page */
skip_fetch = (scan->rs_flags & SO_CAN_SKIP_FETCH &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
}
diff --git a/src/backend/nodes/tidbitmap.c b/src/backend/nodes/tidbitmap.c
index d2bf8f44d50..7d038c2018d 100644
--- a/src/backend/nodes/tidbitmap.c
+++ b/src/backend/nodes/tidbitmap.c
@@ -172,7 +172,6 @@ struct TBMIterator
int spageptr; /* next spages index */
int schunkptr; /* next schunks index */
int schunkbit; /* next bit to check in current schunk */
- TBMIterateResult output;
};
/*
@@ -213,7 +212,6 @@ struct TBMSharedIterator
PTEntryArray *ptbase; /* pagetable element array */
PTIterationArray *ptpages; /* sorted exact page index list */
PTIterationArray *ptchunks; /* sorted lossy page index list */
- TBMIterateResult output;
};
/* Local function prototypes */
@@ -944,20 +942,21 @@ tbm_advance_schunkbit(PagetableEntry *chunk, int *schunkbitp)
/*
* tbm_iterate - scan through next page of a TIDBitmap
*
- * Returns a TBMIterateResult representing one page, or NULL if there are
- * no more pages to scan. Pages are guaranteed to be delivered in numerical
- * order. If result->ntuples < 0, then the bitmap is "lossy" and failed to
- * remember the exact tuples to look at on this page --- the caller must
- * examine all tuples on the page and check if they meet the intended
- * condition. If result->recheck is true, only the indicated tuples need
- * be examined, but the condition must be rechecked anyway. (For ease of
- * testing, recheck is always set true when ntuples < 0.)
+ * Caller must pass in a TBMIterateResult to be filled.
+ *
+ * Pages are guaranteed to be delivered in numerical order. tbmres->blockno is
+ * set to InvalidBlockNumber when there are no more pages to scan. If
+ * tbmres->ntuples < 0, then the bitmap is "lossy" and failed to remember the
+ * exact tuples to look at on this page --- the caller must examine all tuples
+ * on the page and check if they meet the intended condition. If
+ * tbmres->recheck is true, only the indicated tuples need be examined, but the
+ * condition must be rechecked anyway. (For ease of testing, recheck is always
+ * set true when ntuples < 0.)
*/
-TBMIterateResult *
-tbm_iterate(TBMIterator *iterator)
+void
+tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres)
{
TIDBitmap *tbm = iterator->tbm;
- TBMIterateResult *output = &(iterator->output);
Assert(tbm->iterating == TBM_ITERATING_PRIVATE);
@@ -985,6 +984,7 @@ tbm_iterate(TBMIterator *iterator)
* If both chunk and per-page data remain, must output the numerically
* earlier page.
*/
+ Assert(tbmres);
if (iterator->schunkptr < tbm->nchunks)
{
PagetableEntry *chunk = tbm->schunks[iterator->schunkptr];
@@ -995,11 +995,11 @@ tbm_iterate(TBMIterator *iterator)
chunk_blockno < tbm->spages[iterator->spageptr]->blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
iterator->schunkbit++;
- return output;
+ return;
}
}
@@ -1015,16 +1015,17 @@ tbm_iterate(TBMIterator *iterator)
page = tbm->spages[iterator->spageptr];
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
iterator->spageptr++;
- return output;
+ return;
}
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
@@ -1034,10 +1035,9 @@ tbm_iterate(TBMIterator *iterator)
* across multiple processes. We need to acquire the iterator LWLock,
* before accessing the shared members.
*/
-TBMIterateResult *
-tbm_shared_iterate(TBMSharedIterator *iterator)
+void
+tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres)
{
- TBMIterateResult *output = &iterator->output;
TBMSharedIteratorState *istate = iterator->state;
PagetableEntry *ptbase = NULL;
int *idxpages = NULL;
@@ -1088,13 +1088,13 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
chunk_blockno < ptbase[idxpages[istate->spageptr]].blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
istate->schunkbit++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
}
@@ -1104,21 +1104,22 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
int ntuples;
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
istate->spageptr++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
LWLockRelease(&istate->lock);
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 3013a44bae1..3b432263bb0 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -353,7 +353,7 @@ typedef struct GinScanEntryData
/* for a partial-match or full-scan query, we accumulate all TIDs here */
TIDBitmap *matchBitmap;
TBMIterator *matchIterator;
- TBMIterateResult *matchResult;
+ TBMIterateResult matchResult;
/* used for Posting list and one page in Posting tree */
ItemPointerData *list;
diff --git a/src/include/nodes/tidbitmap.h b/src/include/nodes/tidbitmap.h
index 432fae52962..f000c1af28f 100644
--- a/src/include/nodes/tidbitmap.h
+++ b/src/include/nodes/tidbitmap.h
@@ -72,8 +72,8 @@ extern bool tbm_is_empty(const TIDBitmap *tbm);
extern TBMIterator *tbm_begin_iterate(TIDBitmap *tbm);
extern dsa_pointer tbm_prepare_shared_iterate(TIDBitmap *tbm);
-extern TBMIterateResult *tbm_iterate(TBMIterator *iterator);
-extern TBMIterateResult *tbm_shared_iterate(TBMSharedIterator *iterator);
+extern void tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres);
+extern void tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres);
extern void tbm_end_iterate(TBMIterator *iterator);
extern void tbm_end_shared_iterate(TBMSharedIterator *iterator);
extern TBMSharedIterator *tbm_attach_shared_iterate(dsa_area *dsa,
--
2.40.1
--w4wcjcocxsm37usi
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0013-v7-Streaming-Read-API.patch"
^ permalink raw reply [nested|flat] 59+ messages in thread
* [PATCH v10 13/17] Separate TBM[Shared]Iterator and TBMIterateResult
@ 2024-02-16 02:23 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Melanie Plageman @ 2024-02-16 02:23 UTC (permalink / raw)
Remove the TBMIterateResult from the TBMIterator and TBMSharedIterator
and have tbm_[shared_]iterate() take a TBMIterateResult as a parameter.
This will allow multiple TBMIterateResults to exist concurrently
allowing asynchronous use of the TIDBitmap for prefetching, for example.
tbm_[shared]_iterate() now sets blockno to InvalidBlockNumber when the
bitmap is exhausted instead of returning NULL.
BitmapHeapScan callers of tbm_iterate make a TBMIterateResult locally
and pass it in.
Because GIN only needs a single TBMIterateResult, inline the matchResult
in the GinScanEntry to avoid having to separately manage memory for the
TBMIterateResult.
---
src/backend/access/gin/ginget.c | 48 +++++++++------
src/backend/access/gin/ginscan.c | 2 +-
src/backend/access/heap/heapam_handler.c | 30 +++++-----
src/backend/executor/nodeBitmapHeapscan.c | 47 ++++++++-------
src/backend/nodes/tidbitmap.c | 73 ++++++++++++-----------
src/include/access/gin_private.h | 2 +-
src/include/executor/nodeBitmapHeapscan.h | 2 +-
src/include/nodes/tidbitmap.h | 4 +-
8 files changed, 113 insertions(+), 95 deletions(-)
diff --git a/src/backend/access/gin/ginget.c b/src/backend/access/gin/ginget.c
index 0b4f2ebadb..3aa457a29e 100644
--- a/src/backend/access/gin/ginget.c
+++ b/src/backend/access/gin/ginget.c
@@ -332,10 +332,22 @@ restartScanEntry:
entry->list = NULL;
entry->nlist = 0;
entry->matchBitmap = NULL;
- entry->matchResult = NULL;
entry->reduceResult = false;
entry->predictNumberResult = 0;
+ /*
+ * MTODO: is it enough to set blockno to InvalidBlockNumber? In all the
+ * places were we previously set matchResult to NULL, I just set blockno
+ * to InvalidBlockNumber. It seems like this should be okay because that
+ * is usually what we check before using the matchResult members. But it
+ * might be safer to zero out the offsets array. But that is expensive.
+ */
+ entry->matchResult.blockno = InvalidBlockNumber;
+ entry->matchResult.ntuples = 0;
+ entry->matchResult.recheck = true;
+ memset(entry->matchResult.offsets, 0,
+ sizeof(OffsetNumber) * MaxHeapTuplesPerPage);
+
/*
* we should find entry, and begin scan of posting tree or just store
* posting list in memory
@@ -374,6 +386,7 @@ restartScanEntry:
{
if (entry->matchIterator)
tbm_end_iterate(entry->matchIterator);
+ entry->matchResult.blockno = InvalidBlockNumber;
entry->matchIterator = NULL;
tbm_free(entry->matchBitmap);
entry->matchBitmap = NULL;
@@ -823,18 +836,19 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
{
/*
* If we've exhausted all items on this block, move to next block
- * in the bitmap.
+ * in the bitmap. tbm_iterate() sets matchResult->blockno to
+ * InvalidBlockNumber when the bitmap is exhausted.
*/
- while (entry->matchResult == NULL ||
- (entry->matchResult->ntuples >= 0 &&
- entry->offset >= entry->matchResult->ntuples) ||
- entry->matchResult->blockno < advancePastBlk ||
+ while ((!BlockNumberIsValid(entry->matchResult.blockno)) ||
+ (entry->matchResult.ntuples >= 0 &&
+ entry->offset >= entry->matchResult.ntuples) ||
+ entry->matchResult.blockno < advancePastBlk ||
(ItemPointerIsLossyPage(&advancePast) &&
- entry->matchResult->blockno == advancePastBlk))
+ entry->matchResult.blockno == advancePastBlk))
{
- entry->matchResult = tbm_iterate(entry->matchIterator);
+ tbm_iterate(entry->matchIterator, &entry->matchResult);
- if (entry->matchResult == NULL)
+ if (!BlockNumberIsValid(entry->matchResult.blockno))
{
ItemPointerSetInvalid(&entry->curItem);
tbm_end_iterate(entry->matchIterator);
@@ -858,10 +872,10 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* We're now on the first page after advancePast which has any
* items on it. If it's a lossy result, return that.
*/
- if (entry->matchResult->ntuples < 0)
+ if (entry->matchResult.ntuples < 0)
{
ItemPointerSetLossyPage(&entry->curItem,
- entry->matchResult->blockno);
+ entry->matchResult.blockno);
/*
* We might as well fall out of the loop; we could not
@@ -875,27 +889,27 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* Not a lossy page. Skip over any offsets <= advancePast, and
* return that.
*/
- if (entry->matchResult->blockno == advancePastBlk)
+ if (entry->matchResult.blockno == advancePastBlk)
{
/*
* First, do a quick check against the last offset on the
* page. If that's > advancePast, so are all the other
* offsets, so just go back to the top to get the next page.
*/
- if (entry->matchResult->offsets[entry->matchResult->ntuples - 1] <= advancePastOff)
+ if (entry->matchResult.offsets[entry->matchResult.ntuples - 1] <= advancePastOff)
{
- entry->offset = entry->matchResult->ntuples;
+ entry->offset = entry->matchResult.ntuples;
continue;
}
/* Otherwise scan to find the first item > advancePast */
- while (entry->matchResult->offsets[entry->offset] <= advancePastOff)
+ while (entry->matchResult.offsets[entry->offset] <= advancePastOff)
entry->offset++;
}
ItemPointerSet(&entry->curItem,
- entry->matchResult->blockno,
- entry->matchResult->offsets[entry->offset]);
+ entry->matchResult.blockno,
+ entry->matchResult.offsets[entry->offset]);
entry->offset++;
/* Done unless we need to reduce the result */
diff --git a/src/backend/access/gin/ginscan.c b/src/backend/access/gin/ginscan.c
index af24d38544..033d525339 100644
--- a/src/backend/access/gin/ginscan.c
+++ b/src/backend/access/gin/ginscan.c
@@ -106,7 +106,7 @@ ginFillScanEntry(GinScanOpaque so, OffsetNumber attnum,
ItemPointerSetMin(&scanEntry->curItem);
scanEntry->matchBitmap = NULL;
scanEntry->matchIterator = NULL;
- scanEntry->matchResult = NULL;
+ scanEntry->matchResult.blockno = InvalidBlockNumber;
scanEntry->list = NULL;
scanEntry->nlist = 0;
scanEntry->offset = InvalidOffsetNumber;
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index d85fee1e50..105137396b 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2122,7 +2122,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Buffer buffer;
Snapshot snapshot;
int ntup;
- TBMIterateResult *tbmres;
+ TBMIterateResult tbmres;
hscan->rs_cindex = 0;
hscan->rs_ntuples = 0;
@@ -2134,9 +2134,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
{
CHECK_FOR_INTERRUPTS();
- tbmres = bhs_iterate(scan->rs_bhs_iterator);
+ bhs_iterate(scan->rs_bhs_iterator, &tbmres);
- if (tbmres == NULL)
+ if (!BlockNumberIsValid(tbmres.blockno))
{
/* no more entries in the bitmap */
Assert(hscan->rs_empty_tuples_pending == 0);
@@ -2151,11 +2151,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* isolation though, as we need to examine all invisible tuples
* reachable by the index.
*/
- } while (!IsolationIsSerializable() && tbmres->blockno >= hscan->rs_nblocks);
+ } while (!IsolationIsSerializable() && tbmres.blockno >= hscan->rs_nblocks);
/* Got a valid block */
- *blockno = tbmres->blockno;
- *recheck = tbmres->recheck;
+ *blockno = tbmres.blockno;
+ *recheck = tbmres.recheck;
/*
* We can skip fetching the heap page if we don't need any fields from the
@@ -2163,19 +2163,19 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* page are visible to our transaction.
*/
if (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmres->recheck &&
- VM_ALL_VISIBLE(scan->rs_rd, tbmres->blockno, &hscan->rs_vmbuffer))
+ !tbmres.recheck &&
+ VM_ALL_VISIBLE(scan->rs_rd, tbmres.blockno, &hscan->rs_vmbuffer))
{
/* can't be lossy in the skip_fetch case */
- Assert(tbmres->ntuples >= 0);
+ Assert(tbmres.ntuples >= 0);
Assert(hscan->rs_empty_tuples_pending >= 0);
- hscan->rs_empty_tuples_pending += tbmres->ntuples;
+ hscan->rs_empty_tuples_pending += tbmres.ntuples;
return true;
}
- block = tbmres->blockno;
+ block = tbmres.blockno;
/*
* Acquire pin on the target heap page, trading in any pin we held before.
@@ -2204,7 +2204,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
/*
* We need two separate strategies for lossy and non-lossy cases.
*/
- if (tbmres->ntuples >= 0)
+ if (tbmres.ntuples >= 0)
{
/*
* Bitmap is non-lossy, so we just look through the offsets listed in
@@ -2213,9 +2213,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
*/
int curslot;
- for (curslot = 0; curslot < tbmres->ntuples; curslot++)
+ for (curslot = 0; curslot < tbmres.ntuples; curslot++)
{
- OffsetNumber offnum = tbmres->offsets[curslot];
+ OffsetNumber offnum = tbmres.offsets[curslot];
ItemPointerData tid;
HeapTupleData heapTuple;
@@ -2265,7 +2265,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
- if (tbmres->ntuples < 0)
+ if (tbmres.ntuples < 0)
(*lossy_pages)++;
else
(*exact_pages)++;
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 076e1ff674..78f79aafff 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -77,15 +77,16 @@ bhs_begin_iterate(TIDBitmap *tbm, dsa_pointer shared_area, dsa_area *personal_ar
return result;
}
-TBMIterateResult *
-bhs_iterate(BitmapHeapIterator *iterator)
+void
+bhs_iterate(BitmapHeapIterator *iterator, TBMIterateResult *result)
{
Assert(iterator);
+ Assert(result);
if (iterator->serial)
- return tbm_iterate(iterator->serial);
+ tbm_iterate(iterator->serial, result);
else
- return tbm_shared_iterate(iterator->parallel);
+ tbm_shared_iterate(iterator->parallel, result);
}
void
@@ -348,7 +349,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
#ifdef USE_PREFETCH
ParallelBitmapHeapState *pstate = node->pstate;
BitmapHeapIterator *prefetch_iterator = node->pf_iterator;
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
if (pstate == NULL)
{
@@ -360,8 +361,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
else if (prefetch_iterator)
{
/* Do not let the prefetch iterator get behind the main one */
- tbmpre = bhs_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ bhs_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
return;
}
@@ -394,8 +395,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
*/
if (prefetch_iterator)
{
- tbmpre = bhs_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ bhs_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
}
}
@@ -462,10 +463,12 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (node->prefetch_pages < node->prefetch_target)
{
- TBMIterateResult *tbmpre = bhs_iterate(prefetch_iterator);
+ TBMIterateResult tbmpre;
bool skip_fetch;
- if (tbmpre == NULL)
+ bhs_iterate(prefetch_iterator, &tbmpre);
+
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
bhs_end_iterate(prefetch_iterator);
@@ -473,7 +476,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
node->prefetch_pages++;
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/*
* If we expect not to have to actually read this heap page,
@@ -482,13 +485,13 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
* prefetch_pages?)
*/
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
@@ -501,7 +504,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (1)
{
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
bool do_prefetch = false;
bool skip_fetch;
@@ -520,8 +523,8 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
if (!do_prefetch)
return;
- tbmpre = bhs_iterate(prefetch_iterator);
- if (tbmpre == NULL)
+ bhs_iterate(prefetch_iterator, &tbmpre);
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
bhs_end_iterate(prefetch_iterator);
@@ -529,17 +532,17 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/* As above, skip prefetch if we expect not to need page */
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
}
diff --git a/src/backend/nodes/tidbitmap.c b/src/backend/nodes/tidbitmap.c
index 1dc4c99bf9..309a44bdb8 100644
--- a/src/backend/nodes/tidbitmap.c
+++ b/src/backend/nodes/tidbitmap.c
@@ -172,7 +172,6 @@ struct TBMIterator
int spageptr; /* next spages index */
int schunkptr; /* next schunks index */
int schunkbit; /* next bit to check in current schunk */
- TBMIterateResult output;
};
/*
@@ -213,7 +212,6 @@ struct TBMSharedIterator
PTEntryArray *ptbase; /* pagetable element array */
PTIterationArray *ptpages; /* sorted exact page index list */
PTIterationArray *ptchunks; /* sorted lossy page index list */
- TBMIterateResult output;
};
/* Local function prototypes */
@@ -944,20 +942,21 @@ tbm_advance_schunkbit(PagetableEntry *chunk, int *schunkbitp)
/*
* tbm_iterate - scan through next page of a TIDBitmap
*
- * Returns a TBMIterateResult representing one page, or NULL if there are
- * no more pages to scan. Pages are guaranteed to be delivered in numerical
- * order. If result->ntuples < 0, then the bitmap is "lossy" and failed to
- * remember the exact tuples to look at on this page --- the caller must
- * examine all tuples on the page and check if they meet the intended
- * condition. If result->recheck is true, only the indicated tuples need
- * be examined, but the condition must be rechecked anyway. (For ease of
- * testing, recheck is always set true when ntuples < 0.)
+ * Caller must pass in a TBMIterateResult to be filled.
+ *
+ * Pages are guaranteed to be delivered in numerical order. tbmres->blockno is
+ * set to InvalidBlockNumber when there are no more pages to scan. If
+ * tbmres->ntuples < 0, then the bitmap is "lossy" and failed to remember the
+ * exact tuples to look at on this page --- the caller must examine all tuples
+ * on the page and check if they meet the intended condition. If
+ * tbmres->recheck is true, only the indicated tuples need be examined, but the
+ * condition must be rechecked anyway. (For ease of testing, recheck is always
+ * set true when ntuples < 0.)
*/
-TBMIterateResult *
-tbm_iterate(TBMIterator *iterator)
+void
+tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres)
{
TIDBitmap *tbm = iterator->tbm;
- TBMIterateResult *output = &(iterator->output);
Assert(tbm->iterating == TBM_ITERATING_PRIVATE);
@@ -985,6 +984,7 @@ tbm_iterate(TBMIterator *iterator)
* If both chunk and per-page data remain, must output the numerically
* earlier page.
*/
+ Assert(tbmres);
if (iterator->schunkptr < tbm->nchunks)
{
PagetableEntry *chunk = tbm->schunks[iterator->schunkptr];
@@ -995,11 +995,11 @@ tbm_iterate(TBMIterator *iterator)
chunk_blockno < tbm->spages[iterator->spageptr]->blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
iterator->schunkbit++;
- return output;
+ return;
}
}
@@ -1015,16 +1015,17 @@ tbm_iterate(TBMIterator *iterator)
page = tbm->spages[iterator->spageptr];
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
iterator->spageptr++;
- return output;
+ return;
}
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
@@ -1034,10 +1035,9 @@ tbm_iterate(TBMIterator *iterator)
* across multiple processes. We need to acquire the iterator LWLock,
* before accessing the shared members.
*/
-TBMIterateResult *
-tbm_shared_iterate(TBMSharedIterator *iterator)
+void
+tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres)
{
- TBMIterateResult *output = &iterator->output;
TBMSharedIteratorState *istate = iterator->state;
PagetableEntry *ptbase = NULL;
int *idxpages = NULL;
@@ -1088,13 +1088,13 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
chunk_blockno < ptbase[idxpages[istate->spageptr]].blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
istate->schunkbit++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
}
@@ -1104,21 +1104,22 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
int ntuples;
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
istate->spageptr++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
LWLockRelease(&istate->lock);
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 3013a44bae..3b432263bb 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -353,7 +353,7 @@ typedef struct GinScanEntryData
/* for a partial-match or full-scan query, we accumulate all TIDs here */
TIDBitmap *matchBitmap;
TBMIterator *matchIterator;
- TBMIterateResult *matchResult;
+ TBMIterateResult matchResult;
/* used for Posting list and one page in Posting tree */
ItemPointerData *list;
diff --git a/src/include/executor/nodeBitmapHeapscan.h b/src/include/executor/nodeBitmapHeapscan.h
index cb56d20dc6..3c330f86e6 100644
--- a/src/include/executor/nodeBitmapHeapscan.h
+++ b/src/include/executor/nodeBitmapHeapscan.h
@@ -34,7 +34,7 @@ typedef struct BitmapHeapIterator
struct TBMSharedIterator *parallel;
} BitmapHeapIterator;
-extern TBMIterateResult *bhs_iterate(BitmapHeapIterator *iterator);
+extern void bhs_iterate(BitmapHeapIterator *iterator, TBMIterateResult *result);
extern void bhs_end_iterate(BitmapHeapIterator *iterator);
diff --git a/src/include/nodes/tidbitmap.h b/src/include/nodes/tidbitmap.h
index 432fae5296..f000c1af28 100644
--- a/src/include/nodes/tidbitmap.h
+++ b/src/include/nodes/tidbitmap.h
@@ -72,8 +72,8 @@ extern bool tbm_is_empty(const TIDBitmap *tbm);
extern TBMIterator *tbm_begin_iterate(TIDBitmap *tbm);
extern dsa_pointer tbm_prepare_shared_iterate(TIDBitmap *tbm);
-extern TBMIterateResult *tbm_iterate(TBMIterator *iterator);
-extern TBMIterateResult *tbm_shared_iterate(TBMSharedIterator *iterator);
+extern void tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres);
+extern void tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres);
extern void tbm_end_iterate(TBMIterator *iterator);
extern void tbm_end_shared_iterate(TBMSharedIterator *iterator);
extern TBMSharedIterator *tbm_attach_shared_iterate(dsa_area *dsa,
--
2.40.1
--3o7pc6dfau5a5hry
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v10-0014-Push-BitmapHeapScan-prefetch-code-into-heapam.c.patch"
^ permalink raw reply [nested|flat] 59+ messages in thread
* [PATCH v11 13/17] Separate TBM[Shared]Iterator and TBMIterateResult
@ 2024-02-16 02:23 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Melanie Plageman @ 2024-02-16 02:23 UTC (permalink / raw)
Remove the TBMIterateResult from the TBMIterator and TBMSharedIterator
and have tbm_[shared_]iterate() take a TBMIterateResult as a parameter.
This will allow multiple TBMIterateResults to exist concurrently
allowing asynchronous use of the TIDBitmap for prefetching, for example.
tbm_[shared]_iterate() now sets blockno to InvalidBlockNumber when the
bitmap is exhausted instead of returning NULL.
BitmapHeapScan callers of tbm_iterate make a TBMIterateResult locally
and pass it in.
Because GIN only needs a single TBMIterateResult, inline the matchResult
in the GinScanEntry to avoid having to separately manage memory for the
TBMIterateResult.
---
src/backend/access/gin/ginget.c | 48 +++++++++------
src/backend/access/gin/ginscan.c | 2 +-
src/backend/access/heap/heapam_handler.c | 30 +++++-----
src/backend/executor/nodeBitmapHeapscan.c | 47 ++++++++-------
src/backend/nodes/tidbitmap.c | 73 ++++++++++++-----------
src/include/access/gin_private.h | 2 +-
src/include/executor/nodeBitmapHeapscan.h | 2 +-
src/include/nodes/tidbitmap.h | 4 +-
8 files changed, 113 insertions(+), 95 deletions(-)
diff --git a/src/backend/access/gin/ginget.c b/src/backend/access/gin/ginget.c
index 0b4f2ebadb..3aa457a29e 100644
--- a/src/backend/access/gin/ginget.c
+++ b/src/backend/access/gin/ginget.c
@@ -332,10 +332,22 @@ restartScanEntry:
entry->list = NULL;
entry->nlist = 0;
entry->matchBitmap = NULL;
- entry->matchResult = NULL;
entry->reduceResult = false;
entry->predictNumberResult = 0;
+ /*
+ * MTODO: is it enough to set blockno to InvalidBlockNumber? In all the
+ * places were we previously set matchResult to NULL, I just set blockno
+ * to InvalidBlockNumber. It seems like this should be okay because that
+ * is usually what we check before using the matchResult members. But it
+ * might be safer to zero out the offsets array. But that is expensive.
+ */
+ entry->matchResult.blockno = InvalidBlockNumber;
+ entry->matchResult.ntuples = 0;
+ entry->matchResult.recheck = true;
+ memset(entry->matchResult.offsets, 0,
+ sizeof(OffsetNumber) * MaxHeapTuplesPerPage);
+
/*
* we should find entry, and begin scan of posting tree or just store
* posting list in memory
@@ -374,6 +386,7 @@ restartScanEntry:
{
if (entry->matchIterator)
tbm_end_iterate(entry->matchIterator);
+ entry->matchResult.blockno = InvalidBlockNumber;
entry->matchIterator = NULL;
tbm_free(entry->matchBitmap);
entry->matchBitmap = NULL;
@@ -823,18 +836,19 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
{
/*
* If we've exhausted all items on this block, move to next block
- * in the bitmap.
+ * in the bitmap. tbm_iterate() sets matchResult->blockno to
+ * InvalidBlockNumber when the bitmap is exhausted.
*/
- while (entry->matchResult == NULL ||
- (entry->matchResult->ntuples >= 0 &&
- entry->offset >= entry->matchResult->ntuples) ||
- entry->matchResult->blockno < advancePastBlk ||
+ while ((!BlockNumberIsValid(entry->matchResult.blockno)) ||
+ (entry->matchResult.ntuples >= 0 &&
+ entry->offset >= entry->matchResult.ntuples) ||
+ entry->matchResult.blockno < advancePastBlk ||
(ItemPointerIsLossyPage(&advancePast) &&
- entry->matchResult->blockno == advancePastBlk))
+ entry->matchResult.blockno == advancePastBlk))
{
- entry->matchResult = tbm_iterate(entry->matchIterator);
+ tbm_iterate(entry->matchIterator, &entry->matchResult);
- if (entry->matchResult == NULL)
+ if (!BlockNumberIsValid(entry->matchResult.blockno))
{
ItemPointerSetInvalid(&entry->curItem);
tbm_end_iterate(entry->matchIterator);
@@ -858,10 +872,10 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* We're now on the first page after advancePast which has any
* items on it. If it's a lossy result, return that.
*/
- if (entry->matchResult->ntuples < 0)
+ if (entry->matchResult.ntuples < 0)
{
ItemPointerSetLossyPage(&entry->curItem,
- entry->matchResult->blockno);
+ entry->matchResult.blockno);
/*
* We might as well fall out of the loop; we could not
@@ -875,27 +889,27 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* Not a lossy page. Skip over any offsets <= advancePast, and
* return that.
*/
- if (entry->matchResult->blockno == advancePastBlk)
+ if (entry->matchResult.blockno == advancePastBlk)
{
/*
* First, do a quick check against the last offset on the
* page. If that's > advancePast, so are all the other
* offsets, so just go back to the top to get the next page.
*/
- if (entry->matchResult->offsets[entry->matchResult->ntuples - 1] <= advancePastOff)
+ if (entry->matchResult.offsets[entry->matchResult.ntuples - 1] <= advancePastOff)
{
- entry->offset = entry->matchResult->ntuples;
+ entry->offset = entry->matchResult.ntuples;
continue;
}
/* Otherwise scan to find the first item > advancePast */
- while (entry->matchResult->offsets[entry->offset] <= advancePastOff)
+ while (entry->matchResult.offsets[entry->offset] <= advancePastOff)
entry->offset++;
}
ItemPointerSet(&entry->curItem,
- entry->matchResult->blockno,
- entry->matchResult->offsets[entry->offset]);
+ entry->matchResult.blockno,
+ entry->matchResult.offsets[entry->offset]);
entry->offset++;
/* Done unless we need to reduce the result */
diff --git a/src/backend/access/gin/ginscan.c b/src/backend/access/gin/ginscan.c
index af24d38544..033d525339 100644
--- a/src/backend/access/gin/ginscan.c
+++ b/src/backend/access/gin/ginscan.c
@@ -106,7 +106,7 @@ ginFillScanEntry(GinScanOpaque so, OffsetNumber attnum,
ItemPointerSetMin(&scanEntry->curItem);
scanEntry->matchBitmap = NULL;
scanEntry->matchIterator = NULL;
- scanEntry->matchResult = NULL;
+ scanEntry->matchResult.blockno = InvalidBlockNumber;
scanEntry->list = NULL;
scanEntry->nlist = 0;
scanEntry->offset = InvalidOffsetNumber;
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 81a7488007..f7e4d1094d 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2180,7 +2180,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Buffer buffer;
Snapshot snapshot;
int ntup;
- TBMIterateResult *tbmres;
+ TBMIterateResult tbmres;
hscan->rs_cindex = 0;
hscan->rs_ntuples = 0;
@@ -2192,9 +2192,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
{
CHECK_FOR_INTERRUPTS();
- tbmres = bhs_iterate(scan->rs_bhs_iterator);
+ bhs_iterate(scan->rs_bhs_iterator, &tbmres);
- if (tbmres == NULL)
+ if (!BlockNumberIsValid(tbmres.blockno))
{
/* no more entries in the bitmap */
Assert(hscan->rs_empty_tuples_pending == 0);
@@ -2209,11 +2209,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* isolation though, as we need to examine all invisible tuples
* reachable by the index.
*/
- } while (!IsolationIsSerializable() && tbmres->blockno >= hscan->rs_nblocks);
+ } while (!IsolationIsSerializable() && tbmres.blockno >= hscan->rs_nblocks);
/* Got a valid block */
- *blockno = tbmres->blockno;
- *recheck = tbmres->recheck;
+ *blockno = tbmres.blockno;
+ *recheck = tbmres.recheck;
/*
* We can skip fetching the heap page if we don't need any fields from the
@@ -2221,19 +2221,19 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* page are visible to our transaction.
*/
if (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmres->recheck &&
- VM_ALL_VISIBLE(scan->rs_rd, tbmres->blockno, &hscan->rs_vmbuffer))
+ !tbmres.recheck &&
+ VM_ALL_VISIBLE(scan->rs_rd, tbmres.blockno, &hscan->rs_vmbuffer))
{
/* can't be lossy in the skip_fetch case */
- Assert(tbmres->ntuples >= 0);
+ Assert(tbmres.ntuples >= 0);
Assert(hscan->rs_empty_tuples_pending >= 0);
- hscan->rs_empty_tuples_pending += tbmres->ntuples;
+ hscan->rs_empty_tuples_pending += tbmres.ntuples;
return true;
}
- block = tbmres->blockno;
+ block = tbmres.blockno;
/*
* Acquire pin on the target heap page, trading in any pin we held before.
@@ -2262,7 +2262,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
/*
* We need two separate strategies for lossy and non-lossy cases.
*/
- if (tbmres->ntuples >= 0)
+ if (tbmres.ntuples >= 0)
{
/*
* Bitmap is non-lossy, so we just look through the offsets listed in
@@ -2271,9 +2271,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
*/
int curslot;
- for (curslot = 0; curslot < tbmres->ntuples; curslot++)
+ for (curslot = 0; curslot < tbmres.ntuples; curslot++)
{
- OffsetNumber offnum = tbmres->offsets[curslot];
+ OffsetNumber offnum = tbmres.offsets[curslot];
ItemPointerData tid;
HeapTupleData heapTuple;
@@ -2323,7 +2323,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
- if (tbmres->ntuples < 0)
+ if (tbmres.ntuples < 0)
(*lossy_pages)++;
else
(*exact_pages)++;
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 076e1ff674..78f79aafff 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -77,15 +77,16 @@ bhs_begin_iterate(TIDBitmap *tbm, dsa_pointer shared_area, dsa_area *personal_ar
return result;
}
-TBMIterateResult *
-bhs_iterate(BitmapHeapIterator *iterator)
+void
+bhs_iterate(BitmapHeapIterator *iterator, TBMIterateResult *result)
{
Assert(iterator);
+ Assert(result);
if (iterator->serial)
- return tbm_iterate(iterator->serial);
+ tbm_iterate(iterator->serial, result);
else
- return tbm_shared_iterate(iterator->parallel);
+ tbm_shared_iterate(iterator->parallel, result);
}
void
@@ -348,7 +349,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
#ifdef USE_PREFETCH
ParallelBitmapHeapState *pstate = node->pstate;
BitmapHeapIterator *prefetch_iterator = node->pf_iterator;
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
if (pstate == NULL)
{
@@ -360,8 +361,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
else if (prefetch_iterator)
{
/* Do not let the prefetch iterator get behind the main one */
- tbmpre = bhs_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ bhs_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
return;
}
@@ -394,8 +395,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
*/
if (prefetch_iterator)
{
- tbmpre = bhs_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ bhs_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
}
}
@@ -462,10 +463,12 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (node->prefetch_pages < node->prefetch_target)
{
- TBMIterateResult *tbmpre = bhs_iterate(prefetch_iterator);
+ TBMIterateResult tbmpre;
bool skip_fetch;
- if (tbmpre == NULL)
+ bhs_iterate(prefetch_iterator, &tbmpre);
+
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
bhs_end_iterate(prefetch_iterator);
@@ -473,7 +476,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
node->prefetch_pages++;
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/*
* If we expect not to have to actually read this heap page,
@@ -482,13 +485,13 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
* prefetch_pages?)
*/
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
@@ -501,7 +504,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (1)
{
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
bool do_prefetch = false;
bool skip_fetch;
@@ -520,8 +523,8 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
if (!do_prefetch)
return;
- tbmpre = bhs_iterate(prefetch_iterator);
- if (tbmpre == NULL)
+ bhs_iterate(prefetch_iterator, &tbmpre);
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
bhs_end_iterate(prefetch_iterator);
@@ -529,17 +532,17 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/* As above, skip prefetch if we expect not to need page */
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
}
diff --git a/src/backend/nodes/tidbitmap.c b/src/backend/nodes/tidbitmap.c
index 1dc4c99bf9..309a44bdb8 100644
--- a/src/backend/nodes/tidbitmap.c
+++ b/src/backend/nodes/tidbitmap.c
@@ -172,7 +172,6 @@ struct TBMIterator
int spageptr; /* next spages index */
int schunkptr; /* next schunks index */
int schunkbit; /* next bit to check in current schunk */
- TBMIterateResult output;
};
/*
@@ -213,7 +212,6 @@ struct TBMSharedIterator
PTEntryArray *ptbase; /* pagetable element array */
PTIterationArray *ptpages; /* sorted exact page index list */
PTIterationArray *ptchunks; /* sorted lossy page index list */
- TBMIterateResult output;
};
/* Local function prototypes */
@@ -944,20 +942,21 @@ tbm_advance_schunkbit(PagetableEntry *chunk, int *schunkbitp)
/*
* tbm_iterate - scan through next page of a TIDBitmap
*
- * Returns a TBMIterateResult representing one page, or NULL if there are
- * no more pages to scan. Pages are guaranteed to be delivered in numerical
- * order. If result->ntuples < 0, then the bitmap is "lossy" and failed to
- * remember the exact tuples to look at on this page --- the caller must
- * examine all tuples on the page and check if they meet the intended
- * condition. If result->recheck is true, only the indicated tuples need
- * be examined, but the condition must be rechecked anyway. (For ease of
- * testing, recheck is always set true when ntuples < 0.)
+ * Caller must pass in a TBMIterateResult to be filled.
+ *
+ * Pages are guaranteed to be delivered in numerical order. tbmres->blockno is
+ * set to InvalidBlockNumber when there are no more pages to scan. If
+ * tbmres->ntuples < 0, then the bitmap is "lossy" and failed to remember the
+ * exact tuples to look at on this page --- the caller must examine all tuples
+ * on the page and check if they meet the intended condition. If
+ * tbmres->recheck is true, only the indicated tuples need be examined, but the
+ * condition must be rechecked anyway. (For ease of testing, recheck is always
+ * set true when ntuples < 0.)
*/
-TBMIterateResult *
-tbm_iterate(TBMIterator *iterator)
+void
+tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres)
{
TIDBitmap *tbm = iterator->tbm;
- TBMIterateResult *output = &(iterator->output);
Assert(tbm->iterating == TBM_ITERATING_PRIVATE);
@@ -985,6 +984,7 @@ tbm_iterate(TBMIterator *iterator)
* If both chunk and per-page data remain, must output the numerically
* earlier page.
*/
+ Assert(tbmres);
if (iterator->schunkptr < tbm->nchunks)
{
PagetableEntry *chunk = tbm->schunks[iterator->schunkptr];
@@ -995,11 +995,11 @@ tbm_iterate(TBMIterator *iterator)
chunk_blockno < tbm->spages[iterator->spageptr]->blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
iterator->schunkbit++;
- return output;
+ return;
}
}
@@ -1015,16 +1015,17 @@ tbm_iterate(TBMIterator *iterator)
page = tbm->spages[iterator->spageptr];
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
iterator->spageptr++;
- return output;
+ return;
}
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
@@ -1034,10 +1035,9 @@ tbm_iterate(TBMIterator *iterator)
* across multiple processes. We need to acquire the iterator LWLock,
* before accessing the shared members.
*/
-TBMIterateResult *
-tbm_shared_iterate(TBMSharedIterator *iterator)
+void
+tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres)
{
- TBMIterateResult *output = &iterator->output;
TBMSharedIteratorState *istate = iterator->state;
PagetableEntry *ptbase = NULL;
int *idxpages = NULL;
@@ -1088,13 +1088,13 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
chunk_blockno < ptbase[idxpages[istate->spageptr]].blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
istate->schunkbit++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
}
@@ -1104,21 +1104,22 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
int ntuples;
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
istate->spageptr++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
LWLockRelease(&istate->lock);
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 3013a44bae..3b432263bb 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -353,7 +353,7 @@ typedef struct GinScanEntryData
/* for a partial-match or full-scan query, we accumulate all TIDs here */
TIDBitmap *matchBitmap;
TBMIterator *matchIterator;
- TBMIterateResult *matchResult;
+ TBMIterateResult matchResult;
/* used for Posting list and one page in Posting tree */
ItemPointerData *list;
diff --git a/src/include/executor/nodeBitmapHeapscan.h b/src/include/executor/nodeBitmapHeapscan.h
index cb56d20dc6..3c330f86e6 100644
--- a/src/include/executor/nodeBitmapHeapscan.h
+++ b/src/include/executor/nodeBitmapHeapscan.h
@@ -34,7 +34,7 @@ typedef struct BitmapHeapIterator
struct TBMSharedIterator *parallel;
} BitmapHeapIterator;
-extern TBMIterateResult *bhs_iterate(BitmapHeapIterator *iterator);
+extern void bhs_iterate(BitmapHeapIterator *iterator, TBMIterateResult *result);
extern void bhs_end_iterate(BitmapHeapIterator *iterator);
diff --git a/src/include/nodes/tidbitmap.h b/src/include/nodes/tidbitmap.h
index 432fae5296..f000c1af28 100644
--- a/src/include/nodes/tidbitmap.h
+++ b/src/include/nodes/tidbitmap.h
@@ -72,8 +72,8 @@ extern bool tbm_is_empty(const TIDBitmap *tbm);
extern TBMIterator *tbm_begin_iterate(TIDBitmap *tbm);
extern dsa_pointer tbm_prepare_shared_iterate(TIDBitmap *tbm);
-extern TBMIterateResult *tbm_iterate(TBMIterator *iterator);
-extern TBMIterateResult *tbm_shared_iterate(TBMSharedIterator *iterator);
+extern void tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres);
+extern void tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres);
extern void tbm_end_iterate(TBMIterator *iterator);
extern void tbm_end_shared_iterate(TBMSharedIterator *iterator);
extern TBMSharedIterator *tbm_attach_shared_iterate(dsa_area *dsa,
--
2.40.1
--owzzsiozz6hgpp7e
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v11-0014-Push-BitmapHeapScan-prefetch-code-into-heapam.c.patch"
^ permalink raw reply [nested|flat] 59+ messages in thread
* [PATCH v6 12/14] Separate TBM[Shared]Iterator and TBMIterateResult
@ 2024-02-16 02:23 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Melanie Plageman @ 2024-02-16 02:23 UTC (permalink / raw)
Remove the TBMIterateResult from the TBMIterator and TBMSharedIterator
and have tbm_[shared_]iterate() take a TBMIterateResult as a parameter.
This will allow multiple TBMIterateResults to exist concurrently
allowing asynchronous use of the TIDBitmap for prefetching, for example.
tbm_[shared]_iterate() now sets blockno to InvalidBlockNumber when the
bitmap is exhausted instead of returning NULL.
BitmapHeapScan callers of tbm_iterate make a TBMIterateResult locally
and pass it in.
Because GIN only needs a single TBMIterateResult, inline the matchResult
in the GinScanEntry to avoid having to separately manage memory for the
TBMIterateResult.
---
src/backend/access/gin/ginget.c | 48 +++++++++------
src/backend/access/gin/ginscan.c | 2 +-
src/backend/access/heap/heapam_handler.c | 32 +++++-----
src/backend/executor/nodeBitmapHeapscan.c | 38 ++++++------
src/backend/nodes/tidbitmap.c | 73 ++++++++++++-----------
src/include/access/gin_private.h | 2 +-
src/include/nodes/tidbitmap.h | 4 +-
7 files changed, 108 insertions(+), 91 deletions(-)
diff --git a/src/backend/access/gin/ginget.c b/src/backend/access/gin/ginget.c
index 0b4f2ebadb6..3aa457a29e1 100644
--- a/src/backend/access/gin/ginget.c
+++ b/src/backend/access/gin/ginget.c
@@ -332,10 +332,22 @@ restartScanEntry:
entry->list = NULL;
entry->nlist = 0;
entry->matchBitmap = NULL;
- entry->matchResult = NULL;
entry->reduceResult = false;
entry->predictNumberResult = 0;
+ /*
+ * MTODO: is it enough to set blockno to InvalidBlockNumber? In all the
+ * places were we previously set matchResult to NULL, I just set blockno
+ * to InvalidBlockNumber. It seems like this should be okay because that
+ * is usually what we check before using the matchResult members. But it
+ * might be safer to zero out the offsets array. But that is expensive.
+ */
+ entry->matchResult.blockno = InvalidBlockNumber;
+ entry->matchResult.ntuples = 0;
+ entry->matchResult.recheck = true;
+ memset(entry->matchResult.offsets, 0,
+ sizeof(OffsetNumber) * MaxHeapTuplesPerPage);
+
/*
* we should find entry, and begin scan of posting tree or just store
* posting list in memory
@@ -374,6 +386,7 @@ restartScanEntry:
{
if (entry->matchIterator)
tbm_end_iterate(entry->matchIterator);
+ entry->matchResult.blockno = InvalidBlockNumber;
entry->matchIterator = NULL;
tbm_free(entry->matchBitmap);
entry->matchBitmap = NULL;
@@ -823,18 +836,19 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
{
/*
* If we've exhausted all items on this block, move to next block
- * in the bitmap.
+ * in the bitmap. tbm_iterate() sets matchResult->blockno to
+ * InvalidBlockNumber when the bitmap is exhausted.
*/
- while (entry->matchResult == NULL ||
- (entry->matchResult->ntuples >= 0 &&
- entry->offset >= entry->matchResult->ntuples) ||
- entry->matchResult->blockno < advancePastBlk ||
+ while ((!BlockNumberIsValid(entry->matchResult.blockno)) ||
+ (entry->matchResult.ntuples >= 0 &&
+ entry->offset >= entry->matchResult.ntuples) ||
+ entry->matchResult.blockno < advancePastBlk ||
(ItemPointerIsLossyPage(&advancePast) &&
- entry->matchResult->blockno == advancePastBlk))
+ entry->matchResult.blockno == advancePastBlk))
{
- entry->matchResult = tbm_iterate(entry->matchIterator);
+ tbm_iterate(entry->matchIterator, &entry->matchResult);
- if (entry->matchResult == NULL)
+ if (!BlockNumberIsValid(entry->matchResult.blockno))
{
ItemPointerSetInvalid(&entry->curItem);
tbm_end_iterate(entry->matchIterator);
@@ -858,10 +872,10 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* We're now on the first page after advancePast which has any
* items on it. If it's a lossy result, return that.
*/
- if (entry->matchResult->ntuples < 0)
+ if (entry->matchResult.ntuples < 0)
{
ItemPointerSetLossyPage(&entry->curItem,
- entry->matchResult->blockno);
+ entry->matchResult.blockno);
/*
* We might as well fall out of the loop; we could not
@@ -875,27 +889,27 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* Not a lossy page. Skip over any offsets <= advancePast, and
* return that.
*/
- if (entry->matchResult->blockno == advancePastBlk)
+ if (entry->matchResult.blockno == advancePastBlk)
{
/*
* First, do a quick check against the last offset on the
* page. If that's > advancePast, so are all the other
* offsets, so just go back to the top to get the next page.
*/
- if (entry->matchResult->offsets[entry->matchResult->ntuples - 1] <= advancePastOff)
+ if (entry->matchResult.offsets[entry->matchResult.ntuples - 1] <= advancePastOff)
{
- entry->offset = entry->matchResult->ntuples;
+ entry->offset = entry->matchResult.ntuples;
continue;
}
/* Otherwise scan to find the first item > advancePast */
- while (entry->matchResult->offsets[entry->offset] <= advancePastOff)
+ while (entry->matchResult.offsets[entry->offset] <= advancePastOff)
entry->offset++;
}
ItemPointerSet(&entry->curItem,
- entry->matchResult->blockno,
- entry->matchResult->offsets[entry->offset]);
+ entry->matchResult.blockno,
+ entry->matchResult.offsets[entry->offset]);
entry->offset++;
/* Done unless we need to reduce the result */
diff --git a/src/backend/access/gin/ginscan.c b/src/backend/access/gin/ginscan.c
index af24d38544e..033d5253394 100644
--- a/src/backend/access/gin/ginscan.c
+++ b/src/backend/access/gin/ginscan.c
@@ -106,7 +106,7 @@ ginFillScanEntry(GinScanOpaque so, OffsetNumber attnum,
ItemPointerSetMin(&scanEntry->curItem);
scanEntry->matchBitmap = NULL;
scanEntry->matchIterator = NULL;
- scanEntry->matchResult = NULL;
+ scanEntry->matchResult.blockno = InvalidBlockNumber;
scanEntry->list = NULL;
scanEntry->nlist = 0;
scanEntry->offset = InvalidOffsetNumber;
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index e038e60cd8f..022753e203a 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2119,7 +2119,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Buffer buffer;
Snapshot snapshot;
int ntup;
- TBMIterateResult *tbmres;
+ TBMIterateResult tbmres;
hscan->rs_cindex = 0;
hscan->rs_ntuples = 0;
@@ -2132,11 +2132,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
CHECK_FOR_INTERRUPTS();
if (scan->shared_tbmiterator)
- tbmres = tbm_shared_iterate(scan->shared_tbmiterator);
+ tbm_shared_iterate(scan->shared_tbmiterator, &tbmres);
else
- tbmres = tbm_iterate(scan->tbmiterator);
+ tbm_iterate(scan->tbmiterator, &tbmres);
- if (tbmres == NULL)
+ if (!BlockNumberIsValid(tbmres.blockno))
{
/* no more entries in the bitmap */
Assert(hscan->rs_empty_tuples_pending == 0);
@@ -2151,11 +2151,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* isolation though, as we need to examine all invisible tuples
* reachable by the index.
*/
- } while (!IsolationIsSerializable() && tbmres->blockno >= hscan->rs_nblocks);
+ } while (!IsolationIsSerializable() && tbmres.blockno >= hscan->rs_nblocks);
/* Got a valid block */
- *blockno = tbmres->blockno;
- *recheck = tbmres->recheck;
+ *blockno = tbmres.blockno;
+ *recheck = tbmres.recheck;
/*
* We can skip fetching the heap page if we don't need any fields from the
@@ -2163,19 +2163,19 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* the page are visible to our transaction.
*/
if (scan->rs_flags & SO_CAN_SKIP_FETCH &&
- !tbmres->recheck &&
- VM_ALL_VISIBLE(scan->rs_rd, tbmres->blockno, &hscan->rs_vmbuffer))
+ !tbmres.recheck &&
+ VM_ALL_VISIBLE(scan->rs_rd, tbmres.blockno, &hscan->rs_vmbuffer))
{
/* can't be lossy in the skip_fetch case */
- Assert(tbmres->ntuples >= 0);
+ Assert(tbmres.ntuples >= 0);
Assert(hscan->rs_empty_tuples_pending >= 0);
- hscan->rs_empty_tuples_pending += tbmres->ntuples;
+ hscan->rs_empty_tuples_pending += tbmres.ntuples;
return true;
}
- block = tbmres->blockno;
+ block = tbmres.blockno;
/*
* Acquire pin on the target heap page, trading in any pin we held before.
@@ -2204,7 +2204,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
/*
* We need two separate strategies for lossy and non-lossy cases.
*/
- if (tbmres->ntuples >= 0)
+ if (tbmres.ntuples >= 0)
{
/*
* Bitmap is non-lossy, so we just look through the offsets listed in
@@ -2213,9 +2213,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
*/
int curslot;
- for (curslot = 0; curslot < tbmres->ntuples; curslot++)
+ for (curslot = 0; curslot < tbmres.ntuples; curslot++)
{
- OffsetNumber offnum = tbmres->offsets[curslot];
+ OffsetNumber offnum = tbmres.offsets[curslot];
ItemPointerData tid;
HeapTupleData heapTuple;
@@ -2265,7 +2265,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
- *lossy = tbmres->ntuples < 0;
+ *lossy = tbmres.ntuples < 0;
/*
* Return true to indicate that a valid block was found and the bitmap is
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 0924613b247..b3b9448627b 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -320,7 +320,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
{
#ifdef USE_PREFETCH
ParallelBitmapHeapState *pstate = node->pstate;
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
if (pstate == NULL)
{
@@ -334,8 +334,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
else if (prefetch_iterator)
{
/* Do not let the prefetch iterator get behind the main one */
- tbmpre = tbm_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
return;
}
@@ -370,8 +370,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
*/
if (prefetch_iterator)
{
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
}
}
@@ -439,10 +439,12 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (node->prefetch_pages < node->prefetch_target)
{
- TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator);
+ TBMIterateResult tbmpre;
bool skip_fetch;
- if (tbmpre == NULL)
+ tbm_iterate(prefetch_iterator, &tbmpre);
+
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_iterate(prefetch_iterator);
@@ -450,7 +452,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
node->prefetch_pages++;
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/*
* If we expect not to have to actually read this heap page,
@@ -459,13 +461,13 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
* prefetch_pages?)
*/
skip_fetch = (scan->rs_flags & SO_CAN_SKIP_FETCH &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
@@ -480,7 +482,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (1)
{
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
bool do_prefetch = false;
bool skip_fetch;
@@ -499,8 +501,8 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
if (!do_prefetch)
return;
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- if (tbmpre == NULL)
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_shared_iterate(prefetch_iterator);
@@ -508,17 +510,17 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/* As above, skip prefetch if we expect not to need page */
skip_fetch = (scan->rs_flags & SO_CAN_SKIP_FETCH &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
}
diff --git a/src/backend/nodes/tidbitmap.c b/src/backend/nodes/tidbitmap.c
index d2bf8f44d50..7d038c2018d 100644
--- a/src/backend/nodes/tidbitmap.c
+++ b/src/backend/nodes/tidbitmap.c
@@ -172,7 +172,6 @@ struct TBMIterator
int spageptr; /* next spages index */
int schunkptr; /* next schunks index */
int schunkbit; /* next bit to check in current schunk */
- TBMIterateResult output;
};
/*
@@ -213,7 +212,6 @@ struct TBMSharedIterator
PTEntryArray *ptbase; /* pagetable element array */
PTIterationArray *ptpages; /* sorted exact page index list */
PTIterationArray *ptchunks; /* sorted lossy page index list */
- TBMIterateResult output;
};
/* Local function prototypes */
@@ -944,20 +942,21 @@ tbm_advance_schunkbit(PagetableEntry *chunk, int *schunkbitp)
/*
* tbm_iterate - scan through next page of a TIDBitmap
*
- * Returns a TBMIterateResult representing one page, or NULL if there are
- * no more pages to scan. Pages are guaranteed to be delivered in numerical
- * order. If result->ntuples < 0, then the bitmap is "lossy" and failed to
- * remember the exact tuples to look at on this page --- the caller must
- * examine all tuples on the page and check if they meet the intended
- * condition. If result->recheck is true, only the indicated tuples need
- * be examined, but the condition must be rechecked anyway. (For ease of
- * testing, recheck is always set true when ntuples < 0.)
+ * Caller must pass in a TBMIterateResult to be filled.
+ *
+ * Pages are guaranteed to be delivered in numerical order. tbmres->blockno is
+ * set to InvalidBlockNumber when there are no more pages to scan. If
+ * tbmres->ntuples < 0, then the bitmap is "lossy" and failed to remember the
+ * exact tuples to look at on this page --- the caller must examine all tuples
+ * on the page and check if they meet the intended condition. If
+ * tbmres->recheck is true, only the indicated tuples need be examined, but the
+ * condition must be rechecked anyway. (For ease of testing, recheck is always
+ * set true when ntuples < 0.)
*/
-TBMIterateResult *
-tbm_iterate(TBMIterator *iterator)
+void
+tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres)
{
TIDBitmap *tbm = iterator->tbm;
- TBMIterateResult *output = &(iterator->output);
Assert(tbm->iterating == TBM_ITERATING_PRIVATE);
@@ -985,6 +984,7 @@ tbm_iterate(TBMIterator *iterator)
* If both chunk and per-page data remain, must output the numerically
* earlier page.
*/
+ Assert(tbmres);
if (iterator->schunkptr < tbm->nchunks)
{
PagetableEntry *chunk = tbm->schunks[iterator->schunkptr];
@@ -995,11 +995,11 @@ tbm_iterate(TBMIterator *iterator)
chunk_blockno < tbm->spages[iterator->spageptr]->blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
iterator->schunkbit++;
- return output;
+ return;
}
}
@@ -1015,16 +1015,17 @@ tbm_iterate(TBMIterator *iterator)
page = tbm->spages[iterator->spageptr];
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
iterator->spageptr++;
- return output;
+ return;
}
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
@@ -1034,10 +1035,9 @@ tbm_iterate(TBMIterator *iterator)
* across multiple processes. We need to acquire the iterator LWLock,
* before accessing the shared members.
*/
-TBMIterateResult *
-tbm_shared_iterate(TBMSharedIterator *iterator)
+void
+tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres)
{
- TBMIterateResult *output = &iterator->output;
TBMSharedIteratorState *istate = iterator->state;
PagetableEntry *ptbase = NULL;
int *idxpages = NULL;
@@ -1088,13 +1088,13 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
chunk_blockno < ptbase[idxpages[istate->spageptr]].blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
istate->schunkbit++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
}
@@ -1104,21 +1104,22 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
int ntuples;
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
istate->spageptr++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
LWLockRelease(&istate->lock);
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 3013a44bae1..3b432263bb0 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -353,7 +353,7 @@ typedef struct GinScanEntryData
/* for a partial-match or full-scan query, we accumulate all TIDs here */
TIDBitmap *matchBitmap;
TBMIterator *matchIterator;
- TBMIterateResult *matchResult;
+ TBMIterateResult matchResult;
/* used for Posting list and one page in Posting tree */
ItemPointerData *list;
diff --git a/src/include/nodes/tidbitmap.h b/src/include/nodes/tidbitmap.h
index 432fae52962..f000c1af28f 100644
--- a/src/include/nodes/tidbitmap.h
+++ b/src/include/nodes/tidbitmap.h
@@ -72,8 +72,8 @@ extern bool tbm_is_empty(const TIDBitmap *tbm);
extern TBMIterator *tbm_begin_iterate(TIDBitmap *tbm);
extern dsa_pointer tbm_prepare_shared_iterate(TIDBitmap *tbm);
-extern TBMIterateResult *tbm_iterate(TBMIterator *iterator);
-extern TBMIterateResult *tbm_shared_iterate(TBMSharedIterator *iterator);
+extern void tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres);
+extern void tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres);
extern void tbm_end_iterate(TBMIterator *iterator);
extern void tbm_end_shared_iterate(TBMSharedIterator *iterator);
extern TBMSharedIterator *tbm_attach_shared_iterate(dsa_area *dsa,
--
2.40.1
--w4wcjcocxsm37usi
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0013-v7-Streaming-Read-API.patch"
^ permalink raw reply [nested|flat] 59+ messages in thread
* [PATCH v8 12/17] Separate TBM[Shared]Iterator and TBMIterateResult
@ 2024-02-16 02:23 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Melanie Plageman @ 2024-02-16 02:23 UTC (permalink / raw)
Remove the TBMIterateResult from the TBMIterator and TBMSharedIterator
and have tbm_[shared_]iterate() take a TBMIterateResult as a parameter.
This will allow multiple TBMIterateResults to exist concurrently
allowing asynchronous use of the TIDBitmap for prefetching, for example.
tbm_[shared]_iterate() now sets blockno to InvalidBlockNumber when the
bitmap is exhausted instead of returning NULL.
BitmapHeapScan callers of tbm_iterate make a TBMIterateResult locally
and pass it in.
Because GIN only needs a single TBMIterateResult, inline the matchResult
in the GinScanEntry to avoid having to separately manage memory for the
TBMIterateResult.
---
src/backend/access/gin/ginget.c | 48 +++++++++------
src/backend/access/gin/ginscan.c | 2 +-
src/backend/access/heap/heapam_handler.c | 32 +++++-----
src/backend/executor/nodeBitmapHeapscan.c | 38 ++++++------
src/backend/nodes/tidbitmap.c | 73 ++++++++++++-----------
src/include/access/gin_private.h | 2 +-
src/include/nodes/tidbitmap.h | 4 +-
7 files changed, 108 insertions(+), 91 deletions(-)
diff --git a/src/backend/access/gin/ginget.c b/src/backend/access/gin/ginget.c
index 0b4f2ebadb..3aa457a29e 100644
--- a/src/backend/access/gin/ginget.c
+++ b/src/backend/access/gin/ginget.c
@@ -332,10 +332,22 @@ restartScanEntry:
entry->list = NULL;
entry->nlist = 0;
entry->matchBitmap = NULL;
- entry->matchResult = NULL;
entry->reduceResult = false;
entry->predictNumberResult = 0;
+ /*
+ * MTODO: is it enough to set blockno to InvalidBlockNumber? In all the
+ * places were we previously set matchResult to NULL, I just set blockno
+ * to InvalidBlockNumber. It seems like this should be okay because that
+ * is usually what we check before using the matchResult members. But it
+ * might be safer to zero out the offsets array. But that is expensive.
+ */
+ entry->matchResult.blockno = InvalidBlockNumber;
+ entry->matchResult.ntuples = 0;
+ entry->matchResult.recheck = true;
+ memset(entry->matchResult.offsets, 0,
+ sizeof(OffsetNumber) * MaxHeapTuplesPerPage);
+
/*
* we should find entry, and begin scan of posting tree or just store
* posting list in memory
@@ -374,6 +386,7 @@ restartScanEntry:
{
if (entry->matchIterator)
tbm_end_iterate(entry->matchIterator);
+ entry->matchResult.blockno = InvalidBlockNumber;
entry->matchIterator = NULL;
tbm_free(entry->matchBitmap);
entry->matchBitmap = NULL;
@@ -823,18 +836,19 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
{
/*
* If we've exhausted all items on this block, move to next block
- * in the bitmap.
+ * in the bitmap. tbm_iterate() sets matchResult->blockno to
+ * InvalidBlockNumber when the bitmap is exhausted.
*/
- while (entry->matchResult == NULL ||
- (entry->matchResult->ntuples >= 0 &&
- entry->offset >= entry->matchResult->ntuples) ||
- entry->matchResult->blockno < advancePastBlk ||
+ while ((!BlockNumberIsValid(entry->matchResult.blockno)) ||
+ (entry->matchResult.ntuples >= 0 &&
+ entry->offset >= entry->matchResult.ntuples) ||
+ entry->matchResult.blockno < advancePastBlk ||
(ItemPointerIsLossyPage(&advancePast) &&
- entry->matchResult->blockno == advancePastBlk))
+ entry->matchResult.blockno == advancePastBlk))
{
- entry->matchResult = tbm_iterate(entry->matchIterator);
+ tbm_iterate(entry->matchIterator, &entry->matchResult);
- if (entry->matchResult == NULL)
+ if (!BlockNumberIsValid(entry->matchResult.blockno))
{
ItemPointerSetInvalid(&entry->curItem);
tbm_end_iterate(entry->matchIterator);
@@ -858,10 +872,10 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* We're now on the first page after advancePast which has any
* items on it. If it's a lossy result, return that.
*/
- if (entry->matchResult->ntuples < 0)
+ if (entry->matchResult.ntuples < 0)
{
ItemPointerSetLossyPage(&entry->curItem,
- entry->matchResult->blockno);
+ entry->matchResult.blockno);
/*
* We might as well fall out of the loop; we could not
@@ -875,27 +889,27 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* Not a lossy page. Skip over any offsets <= advancePast, and
* return that.
*/
- if (entry->matchResult->blockno == advancePastBlk)
+ if (entry->matchResult.blockno == advancePastBlk)
{
/*
* First, do a quick check against the last offset on the
* page. If that's > advancePast, so are all the other
* offsets, so just go back to the top to get the next page.
*/
- if (entry->matchResult->offsets[entry->matchResult->ntuples - 1] <= advancePastOff)
+ if (entry->matchResult.offsets[entry->matchResult.ntuples - 1] <= advancePastOff)
{
- entry->offset = entry->matchResult->ntuples;
+ entry->offset = entry->matchResult.ntuples;
continue;
}
/* Otherwise scan to find the first item > advancePast */
- while (entry->matchResult->offsets[entry->offset] <= advancePastOff)
+ while (entry->matchResult.offsets[entry->offset] <= advancePastOff)
entry->offset++;
}
ItemPointerSet(&entry->curItem,
- entry->matchResult->blockno,
- entry->matchResult->offsets[entry->offset]);
+ entry->matchResult.blockno,
+ entry->matchResult.offsets[entry->offset]);
entry->offset++;
/* Done unless we need to reduce the result */
diff --git a/src/backend/access/gin/ginscan.c b/src/backend/access/gin/ginscan.c
index af24d38544..033d525339 100644
--- a/src/backend/access/gin/ginscan.c
+++ b/src/backend/access/gin/ginscan.c
@@ -106,7 +106,7 @@ ginFillScanEntry(GinScanOpaque so, OffsetNumber attnum,
ItemPointerSetMin(&scanEntry->curItem);
scanEntry->matchBitmap = NULL;
scanEntry->matchIterator = NULL;
- scanEntry->matchResult = NULL;
+ scanEntry->matchResult.blockno = InvalidBlockNumber;
scanEntry->list = NULL;
scanEntry->nlist = 0;
scanEntry->offset = InvalidOffsetNumber;
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 266b34fe6b..adfc77684a 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2122,7 +2122,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Buffer buffer;
Snapshot snapshot;
int ntup;
- TBMIterateResult *tbmres;
+ TBMIterateResult tbmres;
hscan->rs_cindex = 0;
hscan->rs_ntuples = 0;
@@ -2135,11 +2135,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
CHECK_FOR_INTERRUPTS();
if (scan->shared_tbmiterator)
- tbmres = tbm_shared_iterate(scan->shared_tbmiterator);
+ tbm_shared_iterate(scan->shared_tbmiterator, &tbmres);
else
- tbmres = tbm_iterate(scan->tbmiterator);
+ tbm_iterate(scan->tbmiterator, &tbmres);
- if (tbmres == NULL)
+ if (!BlockNumberIsValid(tbmres.blockno))
{
/* no more entries in the bitmap */
Assert(hscan->rs_empty_tuples_pending == 0);
@@ -2154,11 +2154,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* isolation though, as we need to examine all invisible tuples
* reachable by the index.
*/
- } while (!IsolationIsSerializable() && tbmres->blockno >= hscan->rs_nblocks);
+ } while (!IsolationIsSerializable() && tbmres.blockno >= hscan->rs_nblocks);
/* Got a valid block */
- *blockno = tbmres->blockno;
- *recheck = tbmres->recheck;
+ *blockno = tbmres.blockno;
+ *recheck = tbmres.recheck;
/*
* We can skip fetching the heap page if we don't need any fields from the
@@ -2166,19 +2166,19 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* page are visible to our transaction.
*/
if (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmres->recheck &&
- VM_ALL_VISIBLE(scan->rs_rd, tbmres->blockno, &hscan->rs_vmbuffer))
+ !tbmres.recheck &&
+ VM_ALL_VISIBLE(scan->rs_rd, tbmres.blockno, &hscan->rs_vmbuffer))
{
/* can't be lossy in the skip_fetch case */
- Assert(tbmres->ntuples >= 0);
+ Assert(tbmres.ntuples >= 0);
Assert(hscan->rs_empty_tuples_pending >= 0);
- hscan->rs_empty_tuples_pending += tbmres->ntuples;
+ hscan->rs_empty_tuples_pending += tbmres.ntuples;
return true;
}
- block = tbmres->blockno;
+ block = tbmres.blockno;
/*
* Acquire pin on the target heap page, trading in any pin we held before.
@@ -2207,7 +2207,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
/*
* We need two separate strategies for lossy and non-lossy cases.
*/
- if (tbmres->ntuples >= 0)
+ if (tbmres.ntuples >= 0)
{
/*
* Bitmap is non-lossy, so we just look through the offsets listed in
@@ -2216,9 +2216,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
*/
int curslot;
- for (curslot = 0; curslot < tbmres->ntuples; curslot++)
+ for (curslot = 0; curslot < tbmres.ntuples; curslot++)
{
- OffsetNumber offnum = tbmres->offsets[curslot];
+ OffsetNumber offnum = tbmres.offsets[curslot];
ItemPointerData tid;
HeapTupleData heapTuple;
@@ -2268,7 +2268,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
- if (tbmres->ntuples < 0)
+ if (tbmres.ntuples < 0)
(*lossy_pages)++;
else
(*exact_pages)++;
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 96b55507a3..51c4360205 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -316,7 +316,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
{
#ifdef USE_PREFETCH
ParallelBitmapHeapState *pstate = node->pstate;
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
if (pstate == NULL)
{
@@ -330,8 +330,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
else if (prefetch_iterator)
{
/* Do not let the prefetch iterator get behind the main one */
- tbmpre = tbm_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
return;
}
@@ -366,8 +366,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
*/
if (prefetch_iterator)
{
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
}
}
@@ -435,10 +435,12 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (node->prefetch_pages < node->prefetch_target)
{
- TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator);
+ TBMIterateResult tbmpre;
bool skip_fetch;
- if (tbmpre == NULL)
+ tbm_iterate(prefetch_iterator, &tbmpre);
+
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_iterate(prefetch_iterator);
@@ -446,7 +448,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
node->prefetch_pages++;
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/*
* If we expect not to have to actually read this heap page,
@@ -455,13 +457,13 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
* prefetch_pages?)
*/
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
@@ -476,7 +478,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (1)
{
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
bool do_prefetch = false;
bool skip_fetch;
@@ -495,8 +497,8 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
if (!do_prefetch)
return;
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- if (tbmpre == NULL)
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_shared_iterate(prefetch_iterator);
@@ -504,17 +506,17 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/* As above, skip prefetch if we expect not to need page */
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
}
diff --git a/src/backend/nodes/tidbitmap.c b/src/backend/nodes/tidbitmap.c
index 1dc4c99bf9..309a44bdb8 100644
--- a/src/backend/nodes/tidbitmap.c
+++ b/src/backend/nodes/tidbitmap.c
@@ -172,7 +172,6 @@ struct TBMIterator
int spageptr; /* next spages index */
int schunkptr; /* next schunks index */
int schunkbit; /* next bit to check in current schunk */
- TBMIterateResult output;
};
/*
@@ -213,7 +212,6 @@ struct TBMSharedIterator
PTEntryArray *ptbase; /* pagetable element array */
PTIterationArray *ptpages; /* sorted exact page index list */
PTIterationArray *ptchunks; /* sorted lossy page index list */
- TBMIterateResult output;
};
/* Local function prototypes */
@@ -944,20 +942,21 @@ tbm_advance_schunkbit(PagetableEntry *chunk, int *schunkbitp)
/*
* tbm_iterate - scan through next page of a TIDBitmap
*
- * Returns a TBMIterateResult representing one page, or NULL if there are
- * no more pages to scan. Pages are guaranteed to be delivered in numerical
- * order. If result->ntuples < 0, then the bitmap is "lossy" and failed to
- * remember the exact tuples to look at on this page --- the caller must
- * examine all tuples on the page and check if they meet the intended
- * condition. If result->recheck is true, only the indicated tuples need
- * be examined, but the condition must be rechecked anyway. (For ease of
- * testing, recheck is always set true when ntuples < 0.)
+ * Caller must pass in a TBMIterateResult to be filled.
+ *
+ * Pages are guaranteed to be delivered in numerical order. tbmres->blockno is
+ * set to InvalidBlockNumber when there are no more pages to scan. If
+ * tbmres->ntuples < 0, then the bitmap is "lossy" and failed to remember the
+ * exact tuples to look at on this page --- the caller must examine all tuples
+ * on the page and check if they meet the intended condition. If
+ * tbmres->recheck is true, only the indicated tuples need be examined, but the
+ * condition must be rechecked anyway. (For ease of testing, recheck is always
+ * set true when ntuples < 0.)
*/
-TBMIterateResult *
-tbm_iterate(TBMIterator *iterator)
+void
+tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres)
{
TIDBitmap *tbm = iterator->tbm;
- TBMIterateResult *output = &(iterator->output);
Assert(tbm->iterating == TBM_ITERATING_PRIVATE);
@@ -985,6 +984,7 @@ tbm_iterate(TBMIterator *iterator)
* If both chunk and per-page data remain, must output the numerically
* earlier page.
*/
+ Assert(tbmres);
if (iterator->schunkptr < tbm->nchunks)
{
PagetableEntry *chunk = tbm->schunks[iterator->schunkptr];
@@ -995,11 +995,11 @@ tbm_iterate(TBMIterator *iterator)
chunk_blockno < tbm->spages[iterator->spageptr]->blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
iterator->schunkbit++;
- return output;
+ return;
}
}
@@ -1015,16 +1015,17 @@ tbm_iterate(TBMIterator *iterator)
page = tbm->spages[iterator->spageptr];
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
iterator->spageptr++;
- return output;
+ return;
}
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
@@ -1034,10 +1035,9 @@ tbm_iterate(TBMIterator *iterator)
* across multiple processes. We need to acquire the iterator LWLock,
* before accessing the shared members.
*/
-TBMIterateResult *
-tbm_shared_iterate(TBMSharedIterator *iterator)
+void
+tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres)
{
- TBMIterateResult *output = &iterator->output;
TBMSharedIteratorState *istate = iterator->state;
PagetableEntry *ptbase = NULL;
int *idxpages = NULL;
@@ -1088,13 +1088,13 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
chunk_blockno < ptbase[idxpages[istate->spageptr]].blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
istate->schunkbit++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
}
@@ -1104,21 +1104,22 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
int ntuples;
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
istate->spageptr++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
LWLockRelease(&istate->lock);
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 3013a44bae..3b432263bb 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -353,7 +353,7 @@ typedef struct GinScanEntryData
/* for a partial-match or full-scan query, we accumulate all TIDs here */
TIDBitmap *matchBitmap;
TBMIterator *matchIterator;
- TBMIterateResult *matchResult;
+ TBMIterateResult matchResult;
/* used for Posting list and one page in Posting tree */
ItemPointerData *list;
diff --git a/src/include/nodes/tidbitmap.h b/src/include/nodes/tidbitmap.h
index 432fae5296..f000c1af28 100644
--- a/src/include/nodes/tidbitmap.h
+++ b/src/include/nodes/tidbitmap.h
@@ -72,8 +72,8 @@ extern bool tbm_is_empty(const TIDBitmap *tbm);
extern TBMIterator *tbm_begin_iterate(TIDBitmap *tbm);
extern dsa_pointer tbm_prepare_shared_iterate(TIDBitmap *tbm);
-extern TBMIterateResult *tbm_iterate(TBMIterator *iterator);
-extern TBMIterateResult *tbm_shared_iterate(TBMSharedIterator *iterator);
+extern void tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres);
+extern void tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres);
extern void tbm_end_iterate(TBMIterator *iterator);
extern void tbm_end_shared_iterate(TBMSharedIterator *iterator);
extern TBMSharedIterator *tbm_attach_shared_iterate(dsa_area *dsa,
--
2.40.1
--xqq4defy3uncu6k6
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v8-0013-Push-BitmapHeapScan-prefetch-code-into-heapam.c.patch"
^ permalink raw reply [nested|flat] 59+ messages in thread
* [PATCH v9 12/17] Separate TBM[Shared]Iterator and TBMIterateResult
@ 2024-02-16 02:23 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Melanie Plageman @ 2024-02-16 02:23 UTC (permalink / raw)
Remove the TBMIterateResult from the TBMIterator and TBMSharedIterator
and have tbm_[shared_]iterate() take a TBMIterateResult as a parameter.
This will allow multiple TBMIterateResults to exist concurrently
allowing asynchronous use of the TIDBitmap for prefetching, for example.
tbm_[shared]_iterate() now sets blockno to InvalidBlockNumber when the
bitmap is exhausted instead of returning NULL.
BitmapHeapScan callers of tbm_iterate make a TBMIterateResult locally
and pass it in.
Because GIN only needs a single TBMIterateResult, inline the matchResult
in the GinScanEntry to avoid having to separately manage memory for the
TBMIterateResult.
---
src/backend/access/gin/ginget.c | 48 +++++++++------
src/backend/access/gin/ginscan.c | 2 +-
src/backend/access/heap/heapam_handler.c | 32 +++++-----
src/backend/executor/nodeBitmapHeapscan.c | 38 ++++++------
src/backend/nodes/tidbitmap.c | 73 ++++++++++++-----------
src/include/access/gin_private.h | 2 +-
src/include/nodes/tidbitmap.h | 4 +-
7 files changed, 108 insertions(+), 91 deletions(-)
diff --git a/src/backend/access/gin/ginget.c b/src/backend/access/gin/ginget.c
index 0b4f2ebadb..3aa457a29e 100644
--- a/src/backend/access/gin/ginget.c
+++ b/src/backend/access/gin/ginget.c
@@ -332,10 +332,22 @@ restartScanEntry:
entry->list = NULL;
entry->nlist = 0;
entry->matchBitmap = NULL;
- entry->matchResult = NULL;
entry->reduceResult = false;
entry->predictNumberResult = 0;
+ /*
+ * MTODO: is it enough to set blockno to InvalidBlockNumber? In all the
+ * places were we previously set matchResult to NULL, I just set blockno
+ * to InvalidBlockNumber. It seems like this should be okay because that
+ * is usually what we check before using the matchResult members. But it
+ * might be safer to zero out the offsets array. But that is expensive.
+ */
+ entry->matchResult.blockno = InvalidBlockNumber;
+ entry->matchResult.ntuples = 0;
+ entry->matchResult.recheck = true;
+ memset(entry->matchResult.offsets, 0,
+ sizeof(OffsetNumber) * MaxHeapTuplesPerPage);
+
/*
* we should find entry, and begin scan of posting tree or just store
* posting list in memory
@@ -374,6 +386,7 @@ restartScanEntry:
{
if (entry->matchIterator)
tbm_end_iterate(entry->matchIterator);
+ entry->matchResult.blockno = InvalidBlockNumber;
entry->matchIterator = NULL;
tbm_free(entry->matchBitmap);
entry->matchBitmap = NULL;
@@ -823,18 +836,19 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
{
/*
* If we've exhausted all items on this block, move to next block
- * in the bitmap.
+ * in the bitmap. tbm_iterate() sets matchResult->blockno to
+ * InvalidBlockNumber when the bitmap is exhausted.
*/
- while (entry->matchResult == NULL ||
- (entry->matchResult->ntuples >= 0 &&
- entry->offset >= entry->matchResult->ntuples) ||
- entry->matchResult->blockno < advancePastBlk ||
+ while ((!BlockNumberIsValid(entry->matchResult.blockno)) ||
+ (entry->matchResult.ntuples >= 0 &&
+ entry->offset >= entry->matchResult.ntuples) ||
+ entry->matchResult.blockno < advancePastBlk ||
(ItemPointerIsLossyPage(&advancePast) &&
- entry->matchResult->blockno == advancePastBlk))
+ entry->matchResult.blockno == advancePastBlk))
{
- entry->matchResult = tbm_iterate(entry->matchIterator);
+ tbm_iterate(entry->matchIterator, &entry->matchResult);
- if (entry->matchResult == NULL)
+ if (!BlockNumberIsValid(entry->matchResult.blockno))
{
ItemPointerSetInvalid(&entry->curItem);
tbm_end_iterate(entry->matchIterator);
@@ -858,10 +872,10 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* We're now on the first page after advancePast which has any
* items on it. If it's a lossy result, return that.
*/
- if (entry->matchResult->ntuples < 0)
+ if (entry->matchResult.ntuples < 0)
{
ItemPointerSetLossyPage(&entry->curItem,
- entry->matchResult->blockno);
+ entry->matchResult.blockno);
/*
* We might as well fall out of the loop; we could not
@@ -875,27 +889,27 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* Not a lossy page. Skip over any offsets <= advancePast, and
* return that.
*/
- if (entry->matchResult->blockno == advancePastBlk)
+ if (entry->matchResult.blockno == advancePastBlk)
{
/*
* First, do a quick check against the last offset on the
* page. If that's > advancePast, so are all the other
* offsets, so just go back to the top to get the next page.
*/
- if (entry->matchResult->offsets[entry->matchResult->ntuples - 1] <= advancePastOff)
+ if (entry->matchResult.offsets[entry->matchResult.ntuples - 1] <= advancePastOff)
{
- entry->offset = entry->matchResult->ntuples;
+ entry->offset = entry->matchResult.ntuples;
continue;
}
/* Otherwise scan to find the first item > advancePast */
- while (entry->matchResult->offsets[entry->offset] <= advancePastOff)
+ while (entry->matchResult.offsets[entry->offset] <= advancePastOff)
entry->offset++;
}
ItemPointerSet(&entry->curItem,
- entry->matchResult->blockno,
- entry->matchResult->offsets[entry->offset]);
+ entry->matchResult.blockno,
+ entry->matchResult.offsets[entry->offset]);
entry->offset++;
/* Done unless we need to reduce the result */
diff --git a/src/backend/access/gin/ginscan.c b/src/backend/access/gin/ginscan.c
index af24d38544..033d525339 100644
--- a/src/backend/access/gin/ginscan.c
+++ b/src/backend/access/gin/ginscan.c
@@ -106,7 +106,7 @@ ginFillScanEntry(GinScanOpaque so, OffsetNumber attnum,
ItemPointerSetMin(&scanEntry->curItem);
scanEntry->matchBitmap = NULL;
scanEntry->matchIterator = NULL;
- scanEntry->matchResult = NULL;
+ scanEntry->matchResult.blockno = InvalidBlockNumber;
scanEntry->list = NULL;
scanEntry->nlist = 0;
scanEntry->offset = InvalidOffsetNumber;
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 266b34fe6b..adfc77684a 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2122,7 +2122,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Buffer buffer;
Snapshot snapshot;
int ntup;
- TBMIterateResult *tbmres;
+ TBMIterateResult tbmres;
hscan->rs_cindex = 0;
hscan->rs_ntuples = 0;
@@ -2135,11 +2135,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
CHECK_FOR_INTERRUPTS();
if (scan->shared_tbmiterator)
- tbmres = tbm_shared_iterate(scan->shared_tbmiterator);
+ tbm_shared_iterate(scan->shared_tbmiterator, &tbmres);
else
- tbmres = tbm_iterate(scan->tbmiterator);
+ tbm_iterate(scan->tbmiterator, &tbmres);
- if (tbmres == NULL)
+ if (!BlockNumberIsValid(tbmres.blockno))
{
/* no more entries in the bitmap */
Assert(hscan->rs_empty_tuples_pending == 0);
@@ -2154,11 +2154,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* isolation though, as we need to examine all invisible tuples
* reachable by the index.
*/
- } while (!IsolationIsSerializable() && tbmres->blockno >= hscan->rs_nblocks);
+ } while (!IsolationIsSerializable() && tbmres.blockno >= hscan->rs_nblocks);
/* Got a valid block */
- *blockno = tbmres->blockno;
- *recheck = tbmres->recheck;
+ *blockno = tbmres.blockno;
+ *recheck = tbmres.recheck;
/*
* We can skip fetching the heap page if we don't need any fields from the
@@ -2166,19 +2166,19 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* page are visible to our transaction.
*/
if (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmres->recheck &&
- VM_ALL_VISIBLE(scan->rs_rd, tbmres->blockno, &hscan->rs_vmbuffer))
+ !tbmres.recheck &&
+ VM_ALL_VISIBLE(scan->rs_rd, tbmres.blockno, &hscan->rs_vmbuffer))
{
/* can't be lossy in the skip_fetch case */
- Assert(tbmres->ntuples >= 0);
+ Assert(tbmres.ntuples >= 0);
Assert(hscan->rs_empty_tuples_pending >= 0);
- hscan->rs_empty_tuples_pending += tbmres->ntuples;
+ hscan->rs_empty_tuples_pending += tbmres.ntuples;
return true;
}
- block = tbmres->blockno;
+ block = tbmres.blockno;
/*
* Acquire pin on the target heap page, trading in any pin we held before.
@@ -2207,7 +2207,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
/*
* We need two separate strategies for lossy and non-lossy cases.
*/
- if (tbmres->ntuples >= 0)
+ if (tbmres.ntuples >= 0)
{
/*
* Bitmap is non-lossy, so we just look through the offsets listed in
@@ -2216,9 +2216,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
*/
int curslot;
- for (curslot = 0; curslot < tbmres->ntuples; curslot++)
+ for (curslot = 0; curslot < tbmres.ntuples; curslot++)
{
- OffsetNumber offnum = tbmres->offsets[curslot];
+ OffsetNumber offnum = tbmres.offsets[curslot];
ItemPointerData tid;
HeapTupleData heapTuple;
@@ -2268,7 +2268,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
- if (tbmres->ntuples < 0)
+ if (tbmres.ntuples < 0)
(*lossy_pages)++;
else
(*exact_pages)++;
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 96b55507a3..51c4360205 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -316,7 +316,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
{
#ifdef USE_PREFETCH
ParallelBitmapHeapState *pstate = node->pstate;
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
if (pstate == NULL)
{
@@ -330,8 +330,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
else if (prefetch_iterator)
{
/* Do not let the prefetch iterator get behind the main one */
- tbmpre = tbm_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
return;
}
@@ -366,8 +366,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
*/
if (prefetch_iterator)
{
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
}
}
@@ -435,10 +435,12 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (node->prefetch_pages < node->prefetch_target)
{
- TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator);
+ TBMIterateResult tbmpre;
bool skip_fetch;
- if (tbmpre == NULL)
+ tbm_iterate(prefetch_iterator, &tbmpre);
+
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_iterate(prefetch_iterator);
@@ -446,7 +448,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
node->prefetch_pages++;
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/*
* If we expect not to have to actually read this heap page,
@@ -455,13 +457,13 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
* prefetch_pages?)
*/
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
@@ -476,7 +478,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (1)
{
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
bool do_prefetch = false;
bool skip_fetch;
@@ -495,8 +497,8 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
if (!do_prefetch)
return;
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- if (tbmpre == NULL)
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_shared_iterate(prefetch_iterator);
@@ -504,17 +506,17 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/* As above, skip prefetch if we expect not to need page */
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
}
diff --git a/src/backend/nodes/tidbitmap.c b/src/backend/nodes/tidbitmap.c
index 1dc4c99bf9..309a44bdb8 100644
--- a/src/backend/nodes/tidbitmap.c
+++ b/src/backend/nodes/tidbitmap.c
@@ -172,7 +172,6 @@ struct TBMIterator
int spageptr; /* next spages index */
int schunkptr; /* next schunks index */
int schunkbit; /* next bit to check in current schunk */
- TBMIterateResult output;
};
/*
@@ -213,7 +212,6 @@ struct TBMSharedIterator
PTEntryArray *ptbase; /* pagetable element array */
PTIterationArray *ptpages; /* sorted exact page index list */
PTIterationArray *ptchunks; /* sorted lossy page index list */
- TBMIterateResult output;
};
/* Local function prototypes */
@@ -944,20 +942,21 @@ tbm_advance_schunkbit(PagetableEntry *chunk, int *schunkbitp)
/*
* tbm_iterate - scan through next page of a TIDBitmap
*
- * Returns a TBMIterateResult representing one page, or NULL if there are
- * no more pages to scan. Pages are guaranteed to be delivered in numerical
- * order. If result->ntuples < 0, then the bitmap is "lossy" and failed to
- * remember the exact tuples to look at on this page --- the caller must
- * examine all tuples on the page and check if they meet the intended
- * condition. If result->recheck is true, only the indicated tuples need
- * be examined, but the condition must be rechecked anyway. (For ease of
- * testing, recheck is always set true when ntuples < 0.)
+ * Caller must pass in a TBMIterateResult to be filled.
+ *
+ * Pages are guaranteed to be delivered in numerical order. tbmres->blockno is
+ * set to InvalidBlockNumber when there are no more pages to scan. If
+ * tbmres->ntuples < 0, then the bitmap is "lossy" and failed to remember the
+ * exact tuples to look at on this page --- the caller must examine all tuples
+ * on the page and check if they meet the intended condition. If
+ * tbmres->recheck is true, only the indicated tuples need be examined, but the
+ * condition must be rechecked anyway. (For ease of testing, recheck is always
+ * set true when ntuples < 0.)
*/
-TBMIterateResult *
-tbm_iterate(TBMIterator *iterator)
+void
+tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres)
{
TIDBitmap *tbm = iterator->tbm;
- TBMIterateResult *output = &(iterator->output);
Assert(tbm->iterating == TBM_ITERATING_PRIVATE);
@@ -985,6 +984,7 @@ tbm_iterate(TBMIterator *iterator)
* If both chunk and per-page data remain, must output the numerically
* earlier page.
*/
+ Assert(tbmres);
if (iterator->schunkptr < tbm->nchunks)
{
PagetableEntry *chunk = tbm->schunks[iterator->schunkptr];
@@ -995,11 +995,11 @@ tbm_iterate(TBMIterator *iterator)
chunk_blockno < tbm->spages[iterator->spageptr]->blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
iterator->schunkbit++;
- return output;
+ return;
}
}
@@ -1015,16 +1015,17 @@ tbm_iterate(TBMIterator *iterator)
page = tbm->spages[iterator->spageptr];
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
iterator->spageptr++;
- return output;
+ return;
}
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
@@ -1034,10 +1035,9 @@ tbm_iterate(TBMIterator *iterator)
* across multiple processes. We need to acquire the iterator LWLock,
* before accessing the shared members.
*/
-TBMIterateResult *
-tbm_shared_iterate(TBMSharedIterator *iterator)
+void
+tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres)
{
- TBMIterateResult *output = &iterator->output;
TBMSharedIteratorState *istate = iterator->state;
PagetableEntry *ptbase = NULL;
int *idxpages = NULL;
@@ -1088,13 +1088,13 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
chunk_blockno < ptbase[idxpages[istate->spageptr]].blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
istate->schunkbit++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
}
@@ -1104,21 +1104,22 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
int ntuples;
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
istate->spageptr++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
LWLockRelease(&istate->lock);
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 3013a44bae..3b432263bb 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -353,7 +353,7 @@ typedef struct GinScanEntryData
/* for a partial-match or full-scan query, we accumulate all TIDs here */
TIDBitmap *matchBitmap;
TBMIterator *matchIterator;
- TBMIterateResult *matchResult;
+ TBMIterateResult matchResult;
/* used for Posting list and one page in Posting tree */
ItemPointerData *list;
diff --git a/src/include/nodes/tidbitmap.h b/src/include/nodes/tidbitmap.h
index 432fae5296..f000c1af28 100644
--- a/src/include/nodes/tidbitmap.h
+++ b/src/include/nodes/tidbitmap.h
@@ -72,8 +72,8 @@ extern bool tbm_is_empty(const TIDBitmap *tbm);
extern TBMIterator *tbm_begin_iterate(TIDBitmap *tbm);
extern dsa_pointer tbm_prepare_shared_iterate(TIDBitmap *tbm);
-extern TBMIterateResult *tbm_iterate(TBMIterator *iterator);
-extern TBMIterateResult *tbm_shared_iterate(TBMSharedIterator *iterator);
+extern void tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres);
+extern void tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres);
extern void tbm_end_iterate(TBMIterator *iterator);
extern void tbm_end_shared_iterate(TBMSharedIterator *iterator);
extern TBMSharedIterator *tbm_attach_shared_iterate(dsa_area *dsa,
--
2.40.1
--7mdtsjmrzitrgzgx
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v9-0013-Push-BitmapHeapScan-prefetch-code-into-heapam.c.patch"
^ permalink raw reply [nested|flat] 59+ messages in thread
* [PATCH v10 13/17] Separate TBM[Shared]Iterator and TBMIterateResult
@ 2024-02-16 02:23 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Melanie Plageman @ 2024-02-16 02:23 UTC (permalink / raw)
Remove the TBMIterateResult from the TBMIterator and TBMSharedIterator
and have tbm_[shared_]iterate() take a TBMIterateResult as a parameter.
This will allow multiple TBMIterateResults to exist concurrently
allowing asynchronous use of the TIDBitmap for prefetching, for example.
tbm_[shared]_iterate() now sets blockno to InvalidBlockNumber when the
bitmap is exhausted instead of returning NULL.
BitmapHeapScan callers of tbm_iterate make a TBMIterateResult locally
and pass it in.
Because GIN only needs a single TBMIterateResult, inline the matchResult
in the GinScanEntry to avoid having to separately manage memory for the
TBMIterateResult.
---
src/backend/access/gin/ginget.c | 48 +++++++++------
src/backend/access/gin/ginscan.c | 2 +-
src/backend/access/heap/heapam_handler.c | 30 +++++-----
src/backend/executor/nodeBitmapHeapscan.c | 47 ++++++++-------
src/backend/nodes/tidbitmap.c | 73 ++++++++++++-----------
src/include/access/gin_private.h | 2 +-
src/include/executor/nodeBitmapHeapscan.h | 2 +-
src/include/nodes/tidbitmap.h | 4 +-
8 files changed, 113 insertions(+), 95 deletions(-)
diff --git a/src/backend/access/gin/ginget.c b/src/backend/access/gin/ginget.c
index 0b4f2ebadb..3aa457a29e 100644
--- a/src/backend/access/gin/ginget.c
+++ b/src/backend/access/gin/ginget.c
@@ -332,10 +332,22 @@ restartScanEntry:
entry->list = NULL;
entry->nlist = 0;
entry->matchBitmap = NULL;
- entry->matchResult = NULL;
entry->reduceResult = false;
entry->predictNumberResult = 0;
+ /*
+ * MTODO: is it enough to set blockno to InvalidBlockNumber? In all the
+ * places were we previously set matchResult to NULL, I just set blockno
+ * to InvalidBlockNumber. It seems like this should be okay because that
+ * is usually what we check before using the matchResult members. But it
+ * might be safer to zero out the offsets array. But that is expensive.
+ */
+ entry->matchResult.blockno = InvalidBlockNumber;
+ entry->matchResult.ntuples = 0;
+ entry->matchResult.recheck = true;
+ memset(entry->matchResult.offsets, 0,
+ sizeof(OffsetNumber) * MaxHeapTuplesPerPage);
+
/*
* we should find entry, and begin scan of posting tree or just store
* posting list in memory
@@ -374,6 +386,7 @@ restartScanEntry:
{
if (entry->matchIterator)
tbm_end_iterate(entry->matchIterator);
+ entry->matchResult.blockno = InvalidBlockNumber;
entry->matchIterator = NULL;
tbm_free(entry->matchBitmap);
entry->matchBitmap = NULL;
@@ -823,18 +836,19 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
{
/*
* If we've exhausted all items on this block, move to next block
- * in the bitmap.
+ * in the bitmap. tbm_iterate() sets matchResult->blockno to
+ * InvalidBlockNumber when the bitmap is exhausted.
*/
- while (entry->matchResult == NULL ||
- (entry->matchResult->ntuples >= 0 &&
- entry->offset >= entry->matchResult->ntuples) ||
- entry->matchResult->blockno < advancePastBlk ||
+ while ((!BlockNumberIsValid(entry->matchResult.blockno)) ||
+ (entry->matchResult.ntuples >= 0 &&
+ entry->offset >= entry->matchResult.ntuples) ||
+ entry->matchResult.blockno < advancePastBlk ||
(ItemPointerIsLossyPage(&advancePast) &&
- entry->matchResult->blockno == advancePastBlk))
+ entry->matchResult.blockno == advancePastBlk))
{
- entry->matchResult = tbm_iterate(entry->matchIterator);
+ tbm_iterate(entry->matchIterator, &entry->matchResult);
- if (entry->matchResult == NULL)
+ if (!BlockNumberIsValid(entry->matchResult.blockno))
{
ItemPointerSetInvalid(&entry->curItem);
tbm_end_iterate(entry->matchIterator);
@@ -858,10 +872,10 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* We're now on the first page after advancePast which has any
* items on it. If it's a lossy result, return that.
*/
- if (entry->matchResult->ntuples < 0)
+ if (entry->matchResult.ntuples < 0)
{
ItemPointerSetLossyPage(&entry->curItem,
- entry->matchResult->blockno);
+ entry->matchResult.blockno);
/*
* We might as well fall out of the loop; we could not
@@ -875,27 +889,27 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* Not a lossy page. Skip over any offsets <= advancePast, and
* return that.
*/
- if (entry->matchResult->blockno == advancePastBlk)
+ if (entry->matchResult.blockno == advancePastBlk)
{
/*
* First, do a quick check against the last offset on the
* page. If that's > advancePast, so are all the other
* offsets, so just go back to the top to get the next page.
*/
- if (entry->matchResult->offsets[entry->matchResult->ntuples - 1] <= advancePastOff)
+ if (entry->matchResult.offsets[entry->matchResult.ntuples - 1] <= advancePastOff)
{
- entry->offset = entry->matchResult->ntuples;
+ entry->offset = entry->matchResult.ntuples;
continue;
}
/* Otherwise scan to find the first item > advancePast */
- while (entry->matchResult->offsets[entry->offset] <= advancePastOff)
+ while (entry->matchResult.offsets[entry->offset] <= advancePastOff)
entry->offset++;
}
ItemPointerSet(&entry->curItem,
- entry->matchResult->blockno,
- entry->matchResult->offsets[entry->offset]);
+ entry->matchResult.blockno,
+ entry->matchResult.offsets[entry->offset]);
entry->offset++;
/* Done unless we need to reduce the result */
diff --git a/src/backend/access/gin/ginscan.c b/src/backend/access/gin/ginscan.c
index af24d38544..033d525339 100644
--- a/src/backend/access/gin/ginscan.c
+++ b/src/backend/access/gin/ginscan.c
@@ -106,7 +106,7 @@ ginFillScanEntry(GinScanOpaque so, OffsetNumber attnum,
ItemPointerSetMin(&scanEntry->curItem);
scanEntry->matchBitmap = NULL;
scanEntry->matchIterator = NULL;
- scanEntry->matchResult = NULL;
+ scanEntry->matchResult.blockno = InvalidBlockNumber;
scanEntry->list = NULL;
scanEntry->nlist = 0;
scanEntry->offset = InvalidOffsetNumber;
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index d85fee1e50..105137396b 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2122,7 +2122,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Buffer buffer;
Snapshot snapshot;
int ntup;
- TBMIterateResult *tbmres;
+ TBMIterateResult tbmres;
hscan->rs_cindex = 0;
hscan->rs_ntuples = 0;
@@ -2134,9 +2134,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
{
CHECK_FOR_INTERRUPTS();
- tbmres = bhs_iterate(scan->rs_bhs_iterator);
+ bhs_iterate(scan->rs_bhs_iterator, &tbmres);
- if (tbmres == NULL)
+ if (!BlockNumberIsValid(tbmres.blockno))
{
/* no more entries in the bitmap */
Assert(hscan->rs_empty_tuples_pending == 0);
@@ -2151,11 +2151,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* isolation though, as we need to examine all invisible tuples
* reachable by the index.
*/
- } while (!IsolationIsSerializable() && tbmres->blockno >= hscan->rs_nblocks);
+ } while (!IsolationIsSerializable() && tbmres.blockno >= hscan->rs_nblocks);
/* Got a valid block */
- *blockno = tbmres->blockno;
- *recheck = tbmres->recheck;
+ *blockno = tbmres.blockno;
+ *recheck = tbmres.recheck;
/*
* We can skip fetching the heap page if we don't need any fields from the
@@ -2163,19 +2163,19 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* page are visible to our transaction.
*/
if (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmres->recheck &&
- VM_ALL_VISIBLE(scan->rs_rd, tbmres->blockno, &hscan->rs_vmbuffer))
+ !tbmres.recheck &&
+ VM_ALL_VISIBLE(scan->rs_rd, tbmres.blockno, &hscan->rs_vmbuffer))
{
/* can't be lossy in the skip_fetch case */
- Assert(tbmres->ntuples >= 0);
+ Assert(tbmres.ntuples >= 0);
Assert(hscan->rs_empty_tuples_pending >= 0);
- hscan->rs_empty_tuples_pending += tbmres->ntuples;
+ hscan->rs_empty_tuples_pending += tbmres.ntuples;
return true;
}
- block = tbmres->blockno;
+ block = tbmres.blockno;
/*
* Acquire pin on the target heap page, trading in any pin we held before.
@@ -2204,7 +2204,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
/*
* We need two separate strategies for lossy and non-lossy cases.
*/
- if (tbmres->ntuples >= 0)
+ if (tbmres.ntuples >= 0)
{
/*
* Bitmap is non-lossy, so we just look through the offsets listed in
@@ -2213,9 +2213,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
*/
int curslot;
- for (curslot = 0; curslot < tbmres->ntuples; curslot++)
+ for (curslot = 0; curslot < tbmres.ntuples; curslot++)
{
- OffsetNumber offnum = tbmres->offsets[curslot];
+ OffsetNumber offnum = tbmres.offsets[curslot];
ItemPointerData tid;
HeapTupleData heapTuple;
@@ -2265,7 +2265,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
- if (tbmres->ntuples < 0)
+ if (tbmres.ntuples < 0)
(*lossy_pages)++;
else
(*exact_pages)++;
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 076e1ff674..78f79aafff 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -77,15 +77,16 @@ bhs_begin_iterate(TIDBitmap *tbm, dsa_pointer shared_area, dsa_area *personal_ar
return result;
}
-TBMIterateResult *
-bhs_iterate(BitmapHeapIterator *iterator)
+void
+bhs_iterate(BitmapHeapIterator *iterator, TBMIterateResult *result)
{
Assert(iterator);
+ Assert(result);
if (iterator->serial)
- return tbm_iterate(iterator->serial);
+ tbm_iterate(iterator->serial, result);
else
- return tbm_shared_iterate(iterator->parallel);
+ tbm_shared_iterate(iterator->parallel, result);
}
void
@@ -348,7 +349,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
#ifdef USE_PREFETCH
ParallelBitmapHeapState *pstate = node->pstate;
BitmapHeapIterator *prefetch_iterator = node->pf_iterator;
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
if (pstate == NULL)
{
@@ -360,8 +361,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
else if (prefetch_iterator)
{
/* Do not let the prefetch iterator get behind the main one */
- tbmpre = bhs_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ bhs_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
return;
}
@@ -394,8 +395,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
*/
if (prefetch_iterator)
{
- tbmpre = bhs_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ bhs_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
}
}
@@ -462,10 +463,12 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (node->prefetch_pages < node->prefetch_target)
{
- TBMIterateResult *tbmpre = bhs_iterate(prefetch_iterator);
+ TBMIterateResult tbmpre;
bool skip_fetch;
- if (tbmpre == NULL)
+ bhs_iterate(prefetch_iterator, &tbmpre);
+
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
bhs_end_iterate(prefetch_iterator);
@@ -473,7 +476,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
node->prefetch_pages++;
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/*
* If we expect not to have to actually read this heap page,
@@ -482,13 +485,13 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
* prefetch_pages?)
*/
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
@@ -501,7 +504,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (1)
{
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
bool do_prefetch = false;
bool skip_fetch;
@@ -520,8 +523,8 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
if (!do_prefetch)
return;
- tbmpre = bhs_iterate(prefetch_iterator);
- if (tbmpre == NULL)
+ bhs_iterate(prefetch_iterator, &tbmpre);
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
bhs_end_iterate(prefetch_iterator);
@@ -529,17 +532,17 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/* As above, skip prefetch if we expect not to need page */
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
}
diff --git a/src/backend/nodes/tidbitmap.c b/src/backend/nodes/tidbitmap.c
index 1dc4c99bf9..309a44bdb8 100644
--- a/src/backend/nodes/tidbitmap.c
+++ b/src/backend/nodes/tidbitmap.c
@@ -172,7 +172,6 @@ struct TBMIterator
int spageptr; /* next spages index */
int schunkptr; /* next schunks index */
int schunkbit; /* next bit to check in current schunk */
- TBMIterateResult output;
};
/*
@@ -213,7 +212,6 @@ struct TBMSharedIterator
PTEntryArray *ptbase; /* pagetable element array */
PTIterationArray *ptpages; /* sorted exact page index list */
PTIterationArray *ptchunks; /* sorted lossy page index list */
- TBMIterateResult output;
};
/* Local function prototypes */
@@ -944,20 +942,21 @@ tbm_advance_schunkbit(PagetableEntry *chunk, int *schunkbitp)
/*
* tbm_iterate - scan through next page of a TIDBitmap
*
- * Returns a TBMIterateResult representing one page, or NULL if there are
- * no more pages to scan. Pages are guaranteed to be delivered in numerical
- * order. If result->ntuples < 0, then the bitmap is "lossy" and failed to
- * remember the exact tuples to look at on this page --- the caller must
- * examine all tuples on the page and check if they meet the intended
- * condition. If result->recheck is true, only the indicated tuples need
- * be examined, but the condition must be rechecked anyway. (For ease of
- * testing, recheck is always set true when ntuples < 0.)
+ * Caller must pass in a TBMIterateResult to be filled.
+ *
+ * Pages are guaranteed to be delivered in numerical order. tbmres->blockno is
+ * set to InvalidBlockNumber when there are no more pages to scan. If
+ * tbmres->ntuples < 0, then the bitmap is "lossy" and failed to remember the
+ * exact tuples to look at on this page --- the caller must examine all tuples
+ * on the page and check if they meet the intended condition. If
+ * tbmres->recheck is true, only the indicated tuples need be examined, but the
+ * condition must be rechecked anyway. (For ease of testing, recheck is always
+ * set true when ntuples < 0.)
*/
-TBMIterateResult *
-tbm_iterate(TBMIterator *iterator)
+void
+tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres)
{
TIDBitmap *tbm = iterator->tbm;
- TBMIterateResult *output = &(iterator->output);
Assert(tbm->iterating == TBM_ITERATING_PRIVATE);
@@ -985,6 +984,7 @@ tbm_iterate(TBMIterator *iterator)
* If both chunk and per-page data remain, must output the numerically
* earlier page.
*/
+ Assert(tbmres);
if (iterator->schunkptr < tbm->nchunks)
{
PagetableEntry *chunk = tbm->schunks[iterator->schunkptr];
@@ -995,11 +995,11 @@ tbm_iterate(TBMIterator *iterator)
chunk_blockno < tbm->spages[iterator->spageptr]->blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
iterator->schunkbit++;
- return output;
+ return;
}
}
@@ -1015,16 +1015,17 @@ tbm_iterate(TBMIterator *iterator)
page = tbm->spages[iterator->spageptr];
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
iterator->spageptr++;
- return output;
+ return;
}
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
@@ -1034,10 +1035,9 @@ tbm_iterate(TBMIterator *iterator)
* across multiple processes. We need to acquire the iterator LWLock,
* before accessing the shared members.
*/
-TBMIterateResult *
-tbm_shared_iterate(TBMSharedIterator *iterator)
+void
+tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres)
{
- TBMIterateResult *output = &iterator->output;
TBMSharedIteratorState *istate = iterator->state;
PagetableEntry *ptbase = NULL;
int *idxpages = NULL;
@@ -1088,13 +1088,13 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
chunk_blockno < ptbase[idxpages[istate->spageptr]].blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
istate->schunkbit++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
}
@@ -1104,21 +1104,22 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
int ntuples;
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
istate->spageptr++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
LWLockRelease(&istate->lock);
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 3013a44bae..3b432263bb 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -353,7 +353,7 @@ typedef struct GinScanEntryData
/* for a partial-match or full-scan query, we accumulate all TIDs here */
TIDBitmap *matchBitmap;
TBMIterator *matchIterator;
- TBMIterateResult *matchResult;
+ TBMIterateResult matchResult;
/* used for Posting list and one page in Posting tree */
ItemPointerData *list;
diff --git a/src/include/executor/nodeBitmapHeapscan.h b/src/include/executor/nodeBitmapHeapscan.h
index cb56d20dc6..3c330f86e6 100644
--- a/src/include/executor/nodeBitmapHeapscan.h
+++ b/src/include/executor/nodeBitmapHeapscan.h
@@ -34,7 +34,7 @@ typedef struct BitmapHeapIterator
struct TBMSharedIterator *parallel;
} BitmapHeapIterator;
-extern TBMIterateResult *bhs_iterate(BitmapHeapIterator *iterator);
+extern void bhs_iterate(BitmapHeapIterator *iterator, TBMIterateResult *result);
extern void bhs_end_iterate(BitmapHeapIterator *iterator);
diff --git a/src/include/nodes/tidbitmap.h b/src/include/nodes/tidbitmap.h
index 432fae5296..f000c1af28 100644
--- a/src/include/nodes/tidbitmap.h
+++ b/src/include/nodes/tidbitmap.h
@@ -72,8 +72,8 @@ extern bool tbm_is_empty(const TIDBitmap *tbm);
extern TBMIterator *tbm_begin_iterate(TIDBitmap *tbm);
extern dsa_pointer tbm_prepare_shared_iterate(TIDBitmap *tbm);
-extern TBMIterateResult *tbm_iterate(TBMIterator *iterator);
-extern TBMIterateResult *tbm_shared_iterate(TBMSharedIterator *iterator);
+extern void tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres);
+extern void tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres);
extern void tbm_end_iterate(TBMIterator *iterator);
extern void tbm_end_shared_iterate(TBMSharedIterator *iterator);
extern TBMSharedIterator *tbm_attach_shared_iterate(dsa_area *dsa,
--
2.40.1
--3o7pc6dfau5a5hry
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v10-0014-Push-BitmapHeapScan-prefetch-code-into-heapam.c.patch"
^ permalink raw reply [nested|flat] 59+ messages in thread
* [PATCH v9 12/17] Separate TBM[Shared]Iterator and TBMIterateResult
@ 2024-02-16 02:23 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Melanie Plageman @ 2024-02-16 02:23 UTC (permalink / raw)
Remove the TBMIterateResult from the TBMIterator and TBMSharedIterator
and have tbm_[shared_]iterate() take a TBMIterateResult as a parameter.
This will allow multiple TBMIterateResults to exist concurrently
allowing asynchronous use of the TIDBitmap for prefetching, for example.
tbm_[shared]_iterate() now sets blockno to InvalidBlockNumber when the
bitmap is exhausted instead of returning NULL.
BitmapHeapScan callers of tbm_iterate make a TBMIterateResult locally
and pass it in.
Because GIN only needs a single TBMIterateResult, inline the matchResult
in the GinScanEntry to avoid having to separately manage memory for the
TBMIterateResult.
---
src/backend/access/gin/ginget.c | 48 +++++++++------
src/backend/access/gin/ginscan.c | 2 +-
src/backend/access/heap/heapam_handler.c | 32 +++++-----
src/backend/executor/nodeBitmapHeapscan.c | 38 ++++++------
src/backend/nodes/tidbitmap.c | 73 ++++++++++++-----------
src/include/access/gin_private.h | 2 +-
src/include/nodes/tidbitmap.h | 4 +-
7 files changed, 108 insertions(+), 91 deletions(-)
diff --git a/src/backend/access/gin/ginget.c b/src/backend/access/gin/ginget.c
index 0b4f2ebadb..3aa457a29e 100644
--- a/src/backend/access/gin/ginget.c
+++ b/src/backend/access/gin/ginget.c
@@ -332,10 +332,22 @@ restartScanEntry:
entry->list = NULL;
entry->nlist = 0;
entry->matchBitmap = NULL;
- entry->matchResult = NULL;
entry->reduceResult = false;
entry->predictNumberResult = 0;
+ /*
+ * MTODO: is it enough to set blockno to InvalidBlockNumber? In all the
+ * places were we previously set matchResult to NULL, I just set blockno
+ * to InvalidBlockNumber. It seems like this should be okay because that
+ * is usually what we check before using the matchResult members. But it
+ * might be safer to zero out the offsets array. But that is expensive.
+ */
+ entry->matchResult.blockno = InvalidBlockNumber;
+ entry->matchResult.ntuples = 0;
+ entry->matchResult.recheck = true;
+ memset(entry->matchResult.offsets, 0,
+ sizeof(OffsetNumber) * MaxHeapTuplesPerPage);
+
/*
* we should find entry, and begin scan of posting tree or just store
* posting list in memory
@@ -374,6 +386,7 @@ restartScanEntry:
{
if (entry->matchIterator)
tbm_end_iterate(entry->matchIterator);
+ entry->matchResult.blockno = InvalidBlockNumber;
entry->matchIterator = NULL;
tbm_free(entry->matchBitmap);
entry->matchBitmap = NULL;
@@ -823,18 +836,19 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
{
/*
* If we've exhausted all items on this block, move to next block
- * in the bitmap.
+ * in the bitmap. tbm_iterate() sets matchResult->blockno to
+ * InvalidBlockNumber when the bitmap is exhausted.
*/
- while (entry->matchResult == NULL ||
- (entry->matchResult->ntuples >= 0 &&
- entry->offset >= entry->matchResult->ntuples) ||
- entry->matchResult->blockno < advancePastBlk ||
+ while ((!BlockNumberIsValid(entry->matchResult.blockno)) ||
+ (entry->matchResult.ntuples >= 0 &&
+ entry->offset >= entry->matchResult.ntuples) ||
+ entry->matchResult.blockno < advancePastBlk ||
(ItemPointerIsLossyPage(&advancePast) &&
- entry->matchResult->blockno == advancePastBlk))
+ entry->matchResult.blockno == advancePastBlk))
{
- entry->matchResult = tbm_iterate(entry->matchIterator);
+ tbm_iterate(entry->matchIterator, &entry->matchResult);
- if (entry->matchResult == NULL)
+ if (!BlockNumberIsValid(entry->matchResult.blockno))
{
ItemPointerSetInvalid(&entry->curItem);
tbm_end_iterate(entry->matchIterator);
@@ -858,10 +872,10 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* We're now on the first page after advancePast which has any
* items on it. If it's a lossy result, return that.
*/
- if (entry->matchResult->ntuples < 0)
+ if (entry->matchResult.ntuples < 0)
{
ItemPointerSetLossyPage(&entry->curItem,
- entry->matchResult->blockno);
+ entry->matchResult.blockno);
/*
* We might as well fall out of the loop; we could not
@@ -875,27 +889,27 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* Not a lossy page. Skip over any offsets <= advancePast, and
* return that.
*/
- if (entry->matchResult->blockno == advancePastBlk)
+ if (entry->matchResult.blockno == advancePastBlk)
{
/*
* First, do a quick check against the last offset on the
* page. If that's > advancePast, so are all the other
* offsets, so just go back to the top to get the next page.
*/
- if (entry->matchResult->offsets[entry->matchResult->ntuples - 1] <= advancePastOff)
+ if (entry->matchResult.offsets[entry->matchResult.ntuples - 1] <= advancePastOff)
{
- entry->offset = entry->matchResult->ntuples;
+ entry->offset = entry->matchResult.ntuples;
continue;
}
/* Otherwise scan to find the first item > advancePast */
- while (entry->matchResult->offsets[entry->offset] <= advancePastOff)
+ while (entry->matchResult.offsets[entry->offset] <= advancePastOff)
entry->offset++;
}
ItemPointerSet(&entry->curItem,
- entry->matchResult->blockno,
- entry->matchResult->offsets[entry->offset]);
+ entry->matchResult.blockno,
+ entry->matchResult.offsets[entry->offset]);
entry->offset++;
/* Done unless we need to reduce the result */
diff --git a/src/backend/access/gin/ginscan.c b/src/backend/access/gin/ginscan.c
index af24d38544..033d525339 100644
--- a/src/backend/access/gin/ginscan.c
+++ b/src/backend/access/gin/ginscan.c
@@ -106,7 +106,7 @@ ginFillScanEntry(GinScanOpaque so, OffsetNumber attnum,
ItemPointerSetMin(&scanEntry->curItem);
scanEntry->matchBitmap = NULL;
scanEntry->matchIterator = NULL;
- scanEntry->matchResult = NULL;
+ scanEntry->matchResult.blockno = InvalidBlockNumber;
scanEntry->list = NULL;
scanEntry->nlist = 0;
scanEntry->offset = InvalidOffsetNumber;
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 266b34fe6b..adfc77684a 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2122,7 +2122,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Buffer buffer;
Snapshot snapshot;
int ntup;
- TBMIterateResult *tbmres;
+ TBMIterateResult tbmres;
hscan->rs_cindex = 0;
hscan->rs_ntuples = 0;
@@ -2135,11 +2135,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
CHECK_FOR_INTERRUPTS();
if (scan->shared_tbmiterator)
- tbmres = tbm_shared_iterate(scan->shared_tbmiterator);
+ tbm_shared_iterate(scan->shared_tbmiterator, &tbmres);
else
- tbmres = tbm_iterate(scan->tbmiterator);
+ tbm_iterate(scan->tbmiterator, &tbmres);
- if (tbmres == NULL)
+ if (!BlockNumberIsValid(tbmres.blockno))
{
/* no more entries in the bitmap */
Assert(hscan->rs_empty_tuples_pending == 0);
@@ -2154,11 +2154,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* isolation though, as we need to examine all invisible tuples
* reachable by the index.
*/
- } while (!IsolationIsSerializable() && tbmres->blockno >= hscan->rs_nblocks);
+ } while (!IsolationIsSerializable() && tbmres.blockno >= hscan->rs_nblocks);
/* Got a valid block */
- *blockno = tbmres->blockno;
- *recheck = tbmres->recheck;
+ *blockno = tbmres.blockno;
+ *recheck = tbmres.recheck;
/*
* We can skip fetching the heap page if we don't need any fields from the
@@ -2166,19 +2166,19 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* page are visible to our transaction.
*/
if (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmres->recheck &&
- VM_ALL_VISIBLE(scan->rs_rd, tbmres->blockno, &hscan->rs_vmbuffer))
+ !tbmres.recheck &&
+ VM_ALL_VISIBLE(scan->rs_rd, tbmres.blockno, &hscan->rs_vmbuffer))
{
/* can't be lossy in the skip_fetch case */
- Assert(tbmres->ntuples >= 0);
+ Assert(tbmres.ntuples >= 0);
Assert(hscan->rs_empty_tuples_pending >= 0);
- hscan->rs_empty_tuples_pending += tbmres->ntuples;
+ hscan->rs_empty_tuples_pending += tbmres.ntuples;
return true;
}
- block = tbmres->blockno;
+ block = tbmres.blockno;
/*
* Acquire pin on the target heap page, trading in any pin we held before.
@@ -2207,7 +2207,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
/*
* We need two separate strategies for lossy and non-lossy cases.
*/
- if (tbmres->ntuples >= 0)
+ if (tbmres.ntuples >= 0)
{
/*
* Bitmap is non-lossy, so we just look through the offsets listed in
@@ -2216,9 +2216,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
*/
int curslot;
- for (curslot = 0; curslot < tbmres->ntuples; curslot++)
+ for (curslot = 0; curslot < tbmres.ntuples; curslot++)
{
- OffsetNumber offnum = tbmres->offsets[curslot];
+ OffsetNumber offnum = tbmres.offsets[curslot];
ItemPointerData tid;
HeapTupleData heapTuple;
@@ -2268,7 +2268,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
- if (tbmres->ntuples < 0)
+ if (tbmres.ntuples < 0)
(*lossy_pages)++;
else
(*exact_pages)++;
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 96b55507a3..51c4360205 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -316,7 +316,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
{
#ifdef USE_PREFETCH
ParallelBitmapHeapState *pstate = node->pstate;
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
if (pstate == NULL)
{
@@ -330,8 +330,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
else if (prefetch_iterator)
{
/* Do not let the prefetch iterator get behind the main one */
- tbmpre = tbm_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
return;
}
@@ -366,8 +366,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
*/
if (prefetch_iterator)
{
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
}
}
@@ -435,10 +435,12 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (node->prefetch_pages < node->prefetch_target)
{
- TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator);
+ TBMIterateResult tbmpre;
bool skip_fetch;
- if (tbmpre == NULL)
+ tbm_iterate(prefetch_iterator, &tbmpre);
+
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_iterate(prefetch_iterator);
@@ -446,7 +448,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
node->prefetch_pages++;
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/*
* If we expect not to have to actually read this heap page,
@@ -455,13 +457,13 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
* prefetch_pages?)
*/
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
@@ -476,7 +478,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (1)
{
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
bool do_prefetch = false;
bool skip_fetch;
@@ -495,8 +497,8 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
if (!do_prefetch)
return;
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- if (tbmpre == NULL)
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_shared_iterate(prefetch_iterator);
@@ -504,17 +506,17 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/* As above, skip prefetch if we expect not to need page */
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
}
diff --git a/src/backend/nodes/tidbitmap.c b/src/backend/nodes/tidbitmap.c
index 1dc4c99bf9..309a44bdb8 100644
--- a/src/backend/nodes/tidbitmap.c
+++ b/src/backend/nodes/tidbitmap.c
@@ -172,7 +172,6 @@ struct TBMIterator
int spageptr; /* next spages index */
int schunkptr; /* next schunks index */
int schunkbit; /* next bit to check in current schunk */
- TBMIterateResult output;
};
/*
@@ -213,7 +212,6 @@ struct TBMSharedIterator
PTEntryArray *ptbase; /* pagetable element array */
PTIterationArray *ptpages; /* sorted exact page index list */
PTIterationArray *ptchunks; /* sorted lossy page index list */
- TBMIterateResult output;
};
/* Local function prototypes */
@@ -944,20 +942,21 @@ tbm_advance_schunkbit(PagetableEntry *chunk, int *schunkbitp)
/*
* tbm_iterate - scan through next page of a TIDBitmap
*
- * Returns a TBMIterateResult representing one page, or NULL if there are
- * no more pages to scan. Pages are guaranteed to be delivered in numerical
- * order. If result->ntuples < 0, then the bitmap is "lossy" and failed to
- * remember the exact tuples to look at on this page --- the caller must
- * examine all tuples on the page and check if they meet the intended
- * condition. If result->recheck is true, only the indicated tuples need
- * be examined, but the condition must be rechecked anyway. (For ease of
- * testing, recheck is always set true when ntuples < 0.)
+ * Caller must pass in a TBMIterateResult to be filled.
+ *
+ * Pages are guaranteed to be delivered in numerical order. tbmres->blockno is
+ * set to InvalidBlockNumber when there are no more pages to scan. If
+ * tbmres->ntuples < 0, then the bitmap is "lossy" and failed to remember the
+ * exact tuples to look at on this page --- the caller must examine all tuples
+ * on the page and check if they meet the intended condition. If
+ * tbmres->recheck is true, only the indicated tuples need be examined, but the
+ * condition must be rechecked anyway. (For ease of testing, recheck is always
+ * set true when ntuples < 0.)
*/
-TBMIterateResult *
-tbm_iterate(TBMIterator *iterator)
+void
+tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres)
{
TIDBitmap *tbm = iterator->tbm;
- TBMIterateResult *output = &(iterator->output);
Assert(tbm->iterating == TBM_ITERATING_PRIVATE);
@@ -985,6 +984,7 @@ tbm_iterate(TBMIterator *iterator)
* If both chunk and per-page data remain, must output the numerically
* earlier page.
*/
+ Assert(tbmres);
if (iterator->schunkptr < tbm->nchunks)
{
PagetableEntry *chunk = tbm->schunks[iterator->schunkptr];
@@ -995,11 +995,11 @@ tbm_iterate(TBMIterator *iterator)
chunk_blockno < tbm->spages[iterator->spageptr]->blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
iterator->schunkbit++;
- return output;
+ return;
}
}
@@ -1015,16 +1015,17 @@ tbm_iterate(TBMIterator *iterator)
page = tbm->spages[iterator->spageptr];
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
iterator->spageptr++;
- return output;
+ return;
}
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
@@ -1034,10 +1035,9 @@ tbm_iterate(TBMIterator *iterator)
* across multiple processes. We need to acquire the iterator LWLock,
* before accessing the shared members.
*/
-TBMIterateResult *
-tbm_shared_iterate(TBMSharedIterator *iterator)
+void
+tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres)
{
- TBMIterateResult *output = &iterator->output;
TBMSharedIteratorState *istate = iterator->state;
PagetableEntry *ptbase = NULL;
int *idxpages = NULL;
@@ -1088,13 +1088,13 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
chunk_blockno < ptbase[idxpages[istate->spageptr]].blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
istate->schunkbit++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
}
@@ -1104,21 +1104,22 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
int ntuples;
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
istate->spageptr++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
LWLockRelease(&istate->lock);
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 3013a44bae..3b432263bb 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -353,7 +353,7 @@ typedef struct GinScanEntryData
/* for a partial-match or full-scan query, we accumulate all TIDs here */
TIDBitmap *matchBitmap;
TBMIterator *matchIterator;
- TBMIterateResult *matchResult;
+ TBMIterateResult matchResult;
/* used for Posting list and one page in Posting tree */
ItemPointerData *list;
diff --git a/src/include/nodes/tidbitmap.h b/src/include/nodes/tidbitmap.h
index 432fae5296..f000c1af28 100644
--- a/src/include/nodes/tidbitmap.h
+++ b/src/include/nodes/tidbitmap.h
@@ -72,8 +72,8 @@ extern bool tbm_is_empty(const TIDBitmap *tbm);
extern TBMIterator *tbm_begin_iterate(TIDBitmap *tbm);
extern dsa_pointer tbm_prepare_shared_iterate(TIDBitmap *tbm);
-extern TBMIterateResult *tbm_iterate(TBMIterator *iterator);
-extern TBMIterateResult *tbm_shared_iterate(TBMSharedIterator *iterator);
+extern void tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres);
+extern void tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres);
extern void tbm_end_iterate(TBMIterator *iterator);
extern void tbm_end_shared_iterate(TBMSharedIterator *iterator);
extern TBMSharedIterator *tbm_attach_shared_iterate(dsa_area *dsa,
--
2.40.1
--7mdtsjmrzitrgzgx
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v9-0013-Push-BitmapHeapScan-prefetch-code-into-heapam.c.patch"
^ permalink raw reply [nested|flat] 59+ messages in thread
* [PATCH v9 12/17] Separate TBM[Shared]Iterator and TBMIterateResult
@ 2024-02-16 02:23 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Melanie Plageman @ 2024-02-16 02:23 UTC (permalink / raw)
Remove the TBMIterateResult from the TBMIterator and TBMSharedIterator
and have tbm_[shared_]iterate() take a TBMIterateResult as a parameter.
This will allow multiple TBMIterateResults to exist concurrently
allowing asynchronous use of the TIDBitmap for prefetching, for example.
tbm_[shared]_iterate() now sets blockno to InvalidBlockNumber when the
bitmap is exhausted instead of returning NULL.
BitmapHeapScan callers of tbm_iterate make a TBMIterateResult locally
and pass it in.
Because GIN only needs a single TBMIterateResult, inline the matchResult
in the GinScanEntry to avoid having to separately manage memory for the
TBMIterateResult.
---
src/backend/access/gin/ginget.c | 48 +++++++++------
src/backend/access/gin/ginscan.c | 2 +-
src/backend/access/heap/heapam_handler.c | 32 +++++-----
src/backend/executor/nodeBitmapHeapscan.c | 38 ++++++------
src/backend/nodes/tidbitmap.c | 73 ++++++++++++-----------
src/include/access/gin_private.h | 2 +-
src/include/nodes/tidbitmap.h | 4 +-
7 files changed, 108 insertions(+), 91 deletions(-)
diff --git a/src/backend/access/gin/ginget.c b/src/backend/access/gin/ginget.c
index 0b4f2ebadb..3aa457a29e 100644
--- a/src/backend/access/gin/ginget.c
+++ b/src/backend/access/gin/ginget.c
@@ -332,10 +332,22 @@ restartScanEntry:
entry->list = NULL;
entry->nlist = 0;
entry->matchBitmap = NULL;
- entry->matchResult = NULL;
entry->reduceResult = false;
entry->predictNumberResult = 0;
+ /*
+ * MTODO: is it enough to set blockno to InvalidBlockNumber? In all the
+ * places were we previously set matchResult to NULL, I just set blockno
+ * to InvalidBlockNumber. It seems like this should be okay because that
+ * is usually what we check before using the matchResult members. But it
+ * might be safer to zero out the offsets array. But that is expensive.
+ */
+ entry->matchResult.blockno = InvalidBlockNumber;
+ entry->matchResult.ntuples = 0;
+ entry->matchResult.recheck = true;
+ memset(entry->matchResult.offsets, 0,
+ sizeof(OffsetNumber) * MaxHeapTuplesPerPage);
+
/*
* we should find entry, and begin scan of posting tree or just store
* posting list in memory
@@ -374,6 +386,7 @@ restartScanEntry:
{
if (entry->matchIterator)
tbm_end_iterate(entry->matchIterator);
+ entry->matchResult.blockno = InvalidBlockNumber;
entry->matchIterator = NULL;
tbm_free(entry->matchBitmap);
entry->matchBitmap = NULL;
@@ -823,18 +836,19 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
{
/*
* If we've exhausted all items on this block, move to next block
- * in the bitmap.
+ * in the bitmap. tbm_iterate() sets matchResult->blockno to
+ * InvalidBlockNumber when the bitmap is exhausted.
*/
- while (entry->matchResult == NULL ||
- (entry->matchResult->ntuples >= 0 &&
- entry->offset >= entry->matchResult->ntuples) ||
- entry->matchResult->blockno < advancePastBlk ||
+ while ((!BlockNumberIsValid(entry->matchResult.blockno)) ||
+ (entry->matchResult.ntuples >= 0 &&
+ entry->offset >= entry->matchResult.ntuples) ||
+ entry->matchResult.blockno < advancePastBlk ||
(ItemPointerIsLossyPage(&advancePast) &&
- entry->matchResult->blockno == advancePastBlk))
+ entry->matchResult.blockno == advancePastBlk))
{
- entry->matchResult = tbm_iterate(entry->matchIterator);
+ tbm_iterate(entry->matchIterator, &entry->matchResult);
- if (entry->matchResult == NULL)
+ if (!BlockNumberIsValid(entry->matchResult.blockno))
{
ItemPointerSetInvalid(&entry->curItem);
tbm_end_iterate(entry->matchIterator);
@@ -858,10 +872,10 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* We're now on the first page after advancePast which has any
* items on it. If it's a lossy result, return that.
*/
- if (entry->matchResult->ntuples < 0)
+ if (entry->matchResult.ntuples < 0)
{
ItemPointerSetLossyPage(&entry->curItem,
- entry->matchResult->blockno);
+ entry->matchResult.blockno);
/*
* We might as well fall out of the loop; we could not
@@ -875,27 +889,27 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* Not a lossy page. Skip over any offsets <= advancePast, and
* return that.
*/
- if (entry->matchResult->blockno == advancePastBlk)
+ if (entry->matchResult.blockno == advancePastBlk)
{
/*
* First, do a quick check against the last offset on the
* page. If that's > advancePast, so are all the other
* offsets, so just go back to the top to get the next page.
*/
- if (entry->matchResult->offsets[entry->matchResult->ntuples - 1] <= advancePastOff)
+ if (entry->matchResult.offsets[entry->matchResult.ntuples - 1] <= advancePastOff)
{
- entry->offset = entry->matchResult->ntuples;
+ entry->offset = entry->matchResult.ntuples;
continue;
}
/* Otherwise scan to find the first item > advancePast */
- while (entry->matchResult->offsets[entry->offset] <= advancePastOff)
+ while (entry->matchResult.offsets[entry->offset] <= advancePastOff)
entry->offset++;
}
ItemPointerSet(&entry->curItem,
- entry->matchResult->blockno,
- entry->matchResult->offsets[entry->offset]);
+ entry->matchResult.blockno,
+ entry->matchResult.offsets[entry->offset]);
entry->offset++;
/* Done unless we need to reduce the result */
diff --git a/src/backend/access/gin/ginscan.c b/src/backend/access/gin/ginscan.c
index af24d38544..033d525339 100644
--- a/src/backend/access/gin/ginscan.c
+++ b/src/backend/access/gin/ginscan.c
@@ -106,7 +106,7 @@ ginFillScanEntry(GinScanOpaque so, OffsetNumber attnum,
ItemPointerSetMin(&scanEntry->curItem);
scanEntry->matchBitmap = NULL;
scanEntry->matchIterator = NULL;
- scanEntry->matchResult = NULL;
+ scanEntry->matchResult.blockno = InvalidBlockNumber;
scanEntry->list = NULL;
scanEntry->nlist = 0;
scanEntry->offset = InvalidOffsetNumber;
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 266b34fe6b..adfc77684a 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2122,7 +2122,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Buffer buffer;
Snapshot snapshot;
int ntup;
- TBMIterateResult *tbmres;
+ TBMIterateResult tbmres;
hscan->rs_cindex = 0;
hscan->rs_ntuples = 0;
@@ -2135,11 +2135,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
CHECK_FOR_INTERRUPTS();
if (scan->shared_tbmiterator)
- tbmres = tbm_shared_iterate(scan->shared_tbmiterator);
+ tbm_shared_iterate(scan->shared_tbmiterator, &tbmres);
else
- tbmres = tbm_iterate(scan->tbmiterator);
+ tbm_iterate(scan->tbmiterator, &tbmres);
- if (tbmres == NULL)
+ if (!BlockNumberIsValid(tbmres.blockno))
{
/* no more entries in the bitmap */
Assert(hscan->rs_empty_tuples_pending == 0);
@@ -2154,11 +2154,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* isolation though, as we need to examine all invisible tuples
* reachable by the index.
*/
- } while (!IsolationIsSerializable() && tbmres->blockno >= hscan->rs_nblocks);
+ } while (!IsolationIsSerializable() && tbmres.blockno >= hscan->rs_nblocks);
/* Got a valid block */
- *blockno = tbmres->blockno;
- *recheck = tbmres->recheck;
+ *blockno = tbmres.blockno;
+ *recheck = tbmres.recheck;
/*
* We can skip fetching the heap page if we don't need any fields from the
@@ -2166,19 +2166,19 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* page are visible to our transaction.
*/
if (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmres->recheck &&
- VM_ALL_VISIBLE(scan->rs_rd, tbmres->blockno, &hscan->rs_vmbuffer))
+ !tbmres.recheck &&
+ VM_ALL_VISIBLE(scan->rs_rd, tbmres.blockno, &hscan->rs_vmbuffer))
{
/* can't be lossy in the skip_fetch case */
- Assert(tbmres->ntuples >= 0);
+ Assert(tbmres.ntuples >= 0);
Assert(hscan->rs_empty_tuples_pending >= 0);
- hscan->rs_empty_tuples_pending += tbmres->ntuples;
+ hscan->rs_empty_tuples_pending += tbmres.ntuples;
return true;
}
- block = tbmres->blockno;
+ block = tbmres.blockno;
/*
* Acquire pin on the target heap page, trading in any pin we held before.
@@ -2207,7 +2207,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
/*
* We need two separate strategies for lossy and non-lossy cases.
*/
- if (tbmres->ntuples >= 0)
+ if (tbmres.ntuples >= 0)
{
/*
* Bitmap is non-lossy, so we just look through the offsets listed in
@@ -2216,9 +2216,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
*/
int curslot;
- for (curslot = 0; curslot < tbmres->ntuples; curslot++)
+ for (curslot = 0; curslot < tbmres.ntuples; curslot++)
{
- OffsetNumber offnum = tbmres->offsets[curslot];
+ OffsetNumber offnum = tbmres.offsets[curslot];
ItemPointerData tid;
HeapTupleData heapTuple;
@@ -2268,7 +2268,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
- if (tbmres->ntuples < 0)
+ if (tbmres.ntuples < 0)
(*lossy_pages)++;
else
(*exact_pages)++;
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 96b55507a3..51c4360205 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -316,7 +316,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
{
#ifdef USE_PREFETCH
ParallelBitmapHeapState *pstate = node->pstate;
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
if (pstate == NULL)
{
@@ -330,8 +330,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
else if (prefetch_iterator)
{
/* Do not let the prefetch iterator get behind the main one */
- tbmpre = tbm_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
return;
}
@@ -366,8 +366,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
*/
if (prefetch_iterator)
{
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
}
}
@@ -435,10 +435,12 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (node->prefetch_pages < node->prefetch_target)
{
- TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator);
+ TBMIterateResult tbmpre;
bool skip_fetch;
- if (tbmpre == NULL)
+ tbm_iterate(prefetch_iterator, &tbmpre);
+
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_iterate(prefetch_iterator);
@@ -446,7 +448,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
node->prefetch_pages++;
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/*
* If we expect not to have to actually read this heap page,
@@ -455,13 +457,13 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
* prefetch_pages?)
*/
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
@@ -476,7 +478,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (1)
{
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
bool do_prefetch = false;
bool skip_fetch;
@@ -495,8 +497,8 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
if (!do_prefetch)
return;
- tbmpre = tbm_shared_iterate(prefetch_iterator);
- if (tbmpre == NULL)
+ tbm_shared_iterate(prefetch_iterator, &tbmpre);
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
tbm_end_shared_iterate(prefetch_iterator);
@@ -504,17 +506,17 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/* As above, skip prefetch if we expect not to need page */
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
}
diff --git a/src/backend/nodes/tidbitmap.c b/src/backend/nodes/tidbitmap.c
index 1dc4c99bf9..309a44bdb8 100644
--- a/src/backend/nodes/tidbitmap.c
+++ b/src/backend/nodes/tidbitmap.c
@@ -172,7 +172,6 @@ struct TBMIterator
int spageptr; /* next spages index */
int schunkptr; /* next schunks index */
int schunkbit; /* next bit to check in current schunk */
- TBMIterateResult output;
};
/*
@@ -213,7 +212,6 @@ struct TBMSharedIterator
PTEntryArray *ptbase; /* pagetable element array */
PTIterationArray *ptpages; /* sorted exact page index list */
PTIterationArray *ptchunks; /* sorted lossy page index list */
- TBMIterateResult output;
};
/* Local function prototypes */
@@ -944,20 +942,21 @@ tbm_advance_schunkbit(PagetableEntry *chunk, int *schunkbitp)
/*
* tbm_iterate - scan through next page of a TIDBitmap
*
- * Returns a TBMIterateResult representing one page, or NULL if there are
- * no more pages to scan. Pages are guaranteed to be delivered in numerical
- * order. If result->ntuples < 0, then the bitmap is "lossy" and failed to
- * remember the exact tuples to look at on this page --- the caller must
- * examine all tuples on the page and check if they meet the intended
- * condition. If result->recheck is true, only the indicated tuples need
- * be examined, but the condition must be rechecked anyway. (For ease of
- * testing, recheck is always set true when ntuples < 0.)
+ * Caller must pass in a TBMIterateResult to be filled.
+ *
+ * Pages are guaranteed to be delivered in numerical order. tbmres->blockno is
+ * set to InvalidBlockNumber when there are no more pages to scan. If
+ * tbmres->ntuples < 0, then the bitmap is "lossy" and failed to remember the
+ * exact tuples to look at on this page --- the caller must examine all tuples
+ * on the page and check if they meet the intended condition. If
+ * tbmres->recheck is true, only the indicated tuples need be examined, but the
+ * condition must be rechecked anyway. (For ease of testing, recheck is always
+ * set true when ntuples < 0.)
*/
-TBMIterateResult *
-tbm_iterate(TBMIterator *iterator)
+void
+tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres)
{
TIDBitmap *tbm = iterator->tbm;
- TBMIterateResult *output = &(iterator->output);
Assert(tbm->iterating == TBM_ITERATING_PRIVATE);
@@ -985,6 +984,7 @@ tbm_iterate(TBMIterator *iterator)
* If both chunk and per-page data remain, must output the numerically
* earlier page.
*/
+ Assert(tbmres);
if (iterator->schunkptr < tbm->nchunks)
{
PagetableEntry *chunk = tbm->schunks[iterator->schunkptr];
@@ -995,11 +995,11 @@ tbm_iterate(TBMIterator *iterator)
chunk_blockno < tbm->spages[iterator->spageptr]->blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
iterator->schunkbit++;
- return output;
+ return;
}
}
@@ -1015,16 +1015,17 @@ tbm_iterate(TBMIterator *iterator)
page = tbm->spages[iterator->spageptr];
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
iterator->spageptr++;
- return output;
+ return;
}
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
@@ -1034,10 +1035,9 @@ tbm_iterate(TBMIterator *iterator)
* across multiple processes. We need to acquire the iterator LWLock,
* before accessing the shared members.
*/
-TBMIterateResult *
-tbm_shared_iterate(TBMSharedIterator *iterator)
+void
+tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres)
{
- TBMIterateResult *output = &iterator->output;
TBMSharedIteratorState *istate = iterator->state;
PagetableEntry *ptbase = NULL;
int *idxpages = NULL;
@@ -1088,13 +1088,13 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
chunk_blockno < ptbase[idxpages[istate->spageptr]].blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
istate->schunkbit++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
}
@@ -1104,21 +1104,22 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
int ntuples;
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
istate->spageptr++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
LWLockRelease(&istate->lock);
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 3013a44bae..3b432263bb 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -353,7 +353,7 @@ typedef struct GinScanEntryData
/* for a partial-match or full-scan query, we accumulate all TIDs here */
TIDBitmap *matchBitmap;
TBMIterator *matchIterator;
- TBMIterateResult *matchResult;
+ TBMIterateResult matchResult;
/* used for Posting list and one page in Posting tree */
ItemPointerData *list;
diff --git a/src/include/nodes/tidbitmap.h b/src/include/nodes/tidbitmap.h
index 432fae5296..f000c1af28 100644
--- a/src/include/nodes/tidbitmap.h
+++ b/src/include/nodes/tidbitmap.h
@@ -72,8 +72,8 @@ extern bool tbm_is_empty(const TIDBitmap *tbm);
extern TBMIterator *tbm_begin_iterate(TIDBitmap *tbm);
extern dsa_pointer tbm_prepare_shared_iterate(TIDBitmap *tbm);
-extern TBMIterateResult *tbm_iterate(TBMIterator *iterator);
-extern TBMIterateResult *tbm_shared_iterate(TBMSharedIterator *iterator);
+extern void tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres);
+extern void tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres);
extern void tbm_end_iterate(TBMIterator *iterator);
extern void tbm_end_shared_iterate(TBMSharedIterator *iterator);
extern TBMSharedIterator *tbm_attach_shared_iterate(dsa_area *dsa,
--
2.40.1
--7mdtsjmrzitrgzgx
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v9-0013-Push-BitmapHeapScan-prefetch-code-into-heapam.c.patch"
^ permalink raw reply [nested|flat] 59+ messages in thread
* [PATCH v11 13/17] Separate TBM[Shared]Iterator and TBMIterateResult
@ 2024-02-16 02:23 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Melanie Plageman @ 2024-02-16 02:23 UTC (permalink / raw)
Remove the TBMIterateResult from the TBMIterator and TBMSharedIterator
and have tbm_[shared_]iterate() take a TBMIterateResult as a parameter.
This will allow multiple TBMIterateResults to exist concurrently
allowing asynchronous use of the TIDBitmap for prefetching, for example.
tbm_[shared]_iterate() now sets blockno to InvalidBlockNumber when the
bitmap is exhausted instead of returning NULL.
BitmapHeapScan callers of tbm_iterate make a TBMIterateResult locally
and pass it in.
Because GIN only needs a single TBMIterateResult, inline the matchResult
in the GinScanEntry to avoid having to separately manage memory for the
TBMIterateResult.
---
src/backend/access/gin/ginget.c | 48 +++++++++------
src/backend/access/gin/ginscan.c | 2 +-
src/backend/access/heap/heapam_handler.c | 30 +++++-----
src/backend/executor/nodeBitmapHeapscan.c | 47 ++++++++-------
src/backend/nodes/tidbitmap.c | 73 ++++++++++++-----------
src/include/access/gin_private.h | 2 +-
src/include/executor/nodeBitmapHeapscan.h | 2 +-
src/include/nodes/tidbitmap.h | 4 +-
8 files changed, 113 insertions(+), 95 deletions(-)
diff --git a/src/backend/access/gin/ginget.c b/src/backend/access/gin/ginget.c
index 0b4f2ebadb..3aa457a29e 100644
--- a/src/backend/access/gin/ginget.c
+++ b/src/backend/access/gin/ginget.c
@@ -332,10 +332,22 @@ restartScanEntry:
entry->list = NULL;
entry->nlist = 0;
entry->matchBitmap = NULL;
- entry->matchResult = NULL;
entry->reduceResult = false;
entry->predictNumberResult = 0;
+ /*
+ * MTODO: is it enough to set blockno to InvalidBlockNumber? In all the
+ * places were we previously set matchResult to NULL, I just set blockno
+ * to InvalidBlockNumber. It seems like this should be okay because that
+ * is usually what we check before using the matchResult members. But it
+ * might be safer to zero out the offsets array. But that is expensive.
+ */
+ entry->matchResult.blockno = InvalidBlockNumber;
+ entry->matchResult.ntuples = 0;
+ entry->matchResult.recheck = true;
+ memset(entry->matchResult.offsets, 0,
+ sizeof(OffsetNumber) * MaxHeapTuplesPerPage);
+
/*
* we should find entry, and begin scan of posting tree or just store
* posting list in memory
@@ -374,6 +386,7 @@ restartScanEntry:
{
if (entry->matchIterator)
tbm_end_iterate(entry->matchIterator);
+ entry->matchResult.blockno = InvalidBlockNumber;
entry->matchIterator = NULL;
tbm_free(entry->matchBitmap);
entry->matchBitmap = NULL;
@@ -823,18 +836,19 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
{
/*
* If we've exhausted all items on this block, move to next block
- * in the bitmap.
+ * in the bitmap. tbm_iterate() sets matchResult->blockno to
+ * InvalidBlockNumber when the bitmap is exhausted.
*/
- while (entry->matchResult == NULL ||
- (entry->matchResult->ntuples >= 0 &&
- entry->offset >= entry->matchResult->ntuples) ||
- entry->matchResult->blockno < advancePastBlk ||
+ while ((!BlockNumberIsValid(entry->matchResult.blockno)) ||
+ (entry->matchResult.ntuples >= 0 &&
+ entry->offset >= entry->matchResult.ntuples) ||
+ entry->matchResult.blockno < advancePastBlk ||
(ItemPointerIsLossyPage(&advancePast) &&
- entry->matchResult->blockno == advancePastBlk))
+ entry->matchResult.blockno == advancePastBlk))
{
- entry->matchResult = tbm_iterate(entry->matchIterator);
+ tbm_iterate(entry->matchIterator, &entry->matchResult);
- if (entry->matchResult == NULL)
+ if (!BlockNumberIsValid(entry->matchResult.blockno))
{
ItemPointerSetInvalid(&entry->curItem);
tbm_end_iterate(entry->matchIterator);
@@ -858,10 +872,10 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* We're now on the first page after advancePast which has any
* items on it. If it's a lossy result, return that.
*/
- if (entry->matchResult->ntuples < 0)
+ if (entry->matchResult.ntuples < 0)
{
ItemPointerSetLossyPage(&entry->curItem,
- entry->matchResult->blockno);
+ entry->matchResult.blockno);
/*
* We might as well fall out of the loop; we could not
@@ -875,27 +889,27 @@ entryGetItem(GinState *ginstate, GinScanEntry entry,
* Not a lossy page. Skip over any offsets <= advancePast, and
* return that.
*/
- if (entry->matchResult->blockno == advancePastBlk)
+ if (entry->matchResult.blockno == advancePastBlk)
{
/*
* First, do a quick check against the last offset on the
* page. If that's > advancePast, so are all the other
* offsets, so just go back to the top to get the next page.
*/
- if (entry->matchResult->offsets[entry->matchResult->ntuples - 1] <= advancePastOff)
+ if (entry->matchResult.offsets[entry->matchResult.ntuples - 1] <= advancePastOff)
{
- entry->offset = entry->matchResult->ntuples;
+ entry->offset = entry->matchResult.ntuples;
continue;
}
/* Otherwise scan to find the first item > advancePast */
- while (entry->matchResult->offsets[entry->offset] <= advancePastOff)
+ while (entry->matchResult.offsets[entry->offset] <= advancePastOff)
entry->offset++;
}
ItemPointerSet(&entry->curItem,
- entry->matchResult->blockno,
- entry->matchResult->offsets[entry->offset]);
+ entry->matchResult.blockno,
+ entry->matchResult.offsets[entry->offset]);
entry->offset++;
/* Done unless we need to reduce the result */
diff --git a/src/backend/access/gin/ginscan.c b/src/backend/access/gin/ginscan.c
index af24d38544..033d525339 100644
--- a/src/backend/access/gin/ginscan.c
+++ b/src/backend/access/gin/ginscan.c
@@ -106,7 +106,7 @@ ginFillScanEntry(GinScanOpaque so, OffsetNumber attnum,
ItemPointerSetMin(&scanEntry->curItem);
scanEntry->matchBitmap = NULL;
scanEntry->matchIterator = NULL;
- scanEntry->matchResult = NULL;
+ scanEntry->matchResult.blockno = InvalidBlockNumber;
scanEntry->list = NULL;
scanEntry->nlist = 0;
scanEntry->offset = InvalidOffsetNumber;
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 81a7488007..f7e4d1094d 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2180,7 +2180,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Buffer buffer;
Snapshot snapshot;
int ntup;
- TBMIterateResult *tbmres;
+ TBMIterateResult tbmres;
hscan->rs_cindex = 0;
hscan->rs_ntuples = 0;
@@ -2192,9 +2192,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
{
CHECK_FOR_INTERRUPTS();
- tbmres = bhs_iterate(scan->rs_bhs_iterator);
+ bhs_iterate(scan->rs_bhs_iterator, &tbmres);
- if (tbmres == NULL)
+ if (!BlockNumberIsValid(tbmres.blockno))
{
/* no more entries in the bitmap */
Assert(hscan->rs_empty_tuples_pending == 0);
@@ -2209,11 +2209,11 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* isolation though, as we need to examine all invisible tuples
* reachable by the index.
*/
- } while (!IsolationIsSerializable() && tbmres->blockno >= hscan->rs_nblocks);
+ } while (!IsolationIsSerializable() && tbmres.blockno >= hscan->rs_nblocks);
/* Got a valid block */
- *blockno = tbmres->blockno;
- *recheck = tbmres->recheck;
+ *blockno = tbmres.blockno;
+ *recheck = tbmres.recheck;
/*
* We can skip fetching the heap page if we don't need any fields from the
@@ -2221,19 +2221,19 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
* page are visible to our transaction.
*/
if (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmres->recheck &&
- VM_ALL_VISIBLE(scan->rs_rd, tbmres->blockno, &hscan->rs_vmbuffer))
+ !tbmres.recheck &&
+ VM_ALL_VISIBLE(scan->rs_rd, tbmres.blockno, &hscan->rs_vmbuffer))
{
/* can't be lossy in the skip_fetch case */
- Assert(tbmres->ntuples >= 0);
+ Assert(tbmres.ntuples >= 0);
Assert(hscan->rs_empty_tuples_pending >= 0);
- hscan->rs_empty_tuples_pending += tbmres->ntuples;
+ hscan->rs_empty_tuples_pending += tbmres.ntuples;
return true;
}
- block = tbmres->blockno;
+ block = tbmres.blockno;
/*
* Acquire pin on the target heap page, trading in any pin we held before.
@@ -2262,7 +2262,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
/*
* We need two separate strategies for lossy and non-lossy cases.
*/
- if (tbmres->ntuples >= 0)
+ if (tbmres.ntuples >= 0)
{
/*
* Bitmap is non-lossy, so we just look through the offsets listed in
@@ -2271,9 +2271,9 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
*/
int curslot;
- for (curslot = 0; curslot < tbmres->ntuples; curslot++)
+ for (curslot = 0; curslot < tbmres.ntuples; curslot++)
{
- OffsetNumber offnum = tbmres->offsets[curslot];
+ OffsetNumber offnum = tbmres.offsets[curslot];
ItemPointerData tid;
HeapTupleData heapTuple;
@@ -2323,7 +2323,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
- if (tbmres->ntuples < 0)
+ if (tbmres.ntuples < 0)
(*lossy_pages)++;
else
(*exact_pages)++;
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 076e1ff674..78f79aafff 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -77,15 +77,16 @@ bhs_begin_iterate(TIDBitmap *tbm, dsa_pointer shared_area, dsa_area *personal_ar
return result;
}
-TBMIterateResult *
-bhs_iterate(BitmapHeapIterator *iterator)
+void
+bhs_iterate(BitmapHeapIterator *iterator, TBMIterateResult *result)
{
Assert(iterator);
+ Assert(result);
if (iterator->serial)
- return tbm_iterate(iterator->serial);
+ tbm_iterate(iterator->serial, result);
else
- return tbm_shared_iterate(iterator->parallel);
+ tbm_shared_iterate(iterator->parallel, result);
}
void
@@ -348,7 +349,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
#ifdef USE_PREFETCH
ParallelBitmapHeapState *pstate = node->pstate;
BitmapHeapIterator *prefetch_iterator = node->pf_iterator;
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
if (pstate == NULL)
{
@@ -360,8 +361,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
else if (prefetch_iterator)
{
/* Do not let the prefetch iterator get behind the main one */
- tbmpre = bhs_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ bhs_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
return;
}
@@ -394,8 +395,8 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node)
*/
if (prefetch_iterator)
{
- tbmpre = bhs_iterate(prefetch_iterator);
- node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber;
+ bhs_iterate(prefetch_iterator, &tbmpre);
+ node->pfblockno = tbmpre.blockno;
}
}
}
@@ -462,10 +463,12 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (node->prefetch_pages < node->prefetch_target)
{
- TBMIterateResult *tbmpre = bhs_iterate(prefetch_iterator);
+ TBMIterateResult tbmpre;
bool skip_fetch;
- if (tbmpre == NULL)
+ bhs_iterate(prefetch_iterator, &tbmpre);
+
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
bhs_end_iterate(prefetch_iterator);
@@ -473,7 +476,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
node->prefetch_pages++;
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/*
* If we expect not to have to actually read this heap page,
@@ -482,13 +485,13 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
* prefetch_pages?)
*/
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
@@ -501,7 +504,7 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
{
while (1)
{
- TBMIterateResult *tbmpre;
+ TBMIterateResult tbmpre;
bool do_prefetch = false;
bool skip_fetch;
@@ -520,8 +523,8 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
if (!do_prefetch)
return;
- tbmpre = bhs_iterate(prefetch_iterator);
- if (tbmpre == NULL)
+ bhs_iterate(prefetch_iterator, &tbmpre);
+ if (!BlockNumberIsValid(tbmpre.blockno))
{
/* No more pages to prefetch */
bhs_end_iterate(prefetch_iterator);
@@ -529,17 +532,17 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan)
break;
}
- node->pfblockno = tbmpre->blockno;
+ node->pfblockno = tbmpre.blockno;
/* As above, skip prefetch if we expect not to need page */
skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) &&
- !tbmpre->recheck &&
+ !tbmpre.recheck &&
VM_ALL_VISIBLE(node->ss.ss_currentRelation,
- tbmpre->blockno,
+ tbmpre.blockno,
&node->pvmbuffer));
if (!skip_fetch)
- PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno);
+ PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno);
}
}
}
diff --git a/src/backend/nodes/tidbitmap.c b/src/backend/nodes/tidbitmap.c
index 1dc4c99bf9..309a44bdb8 100644
--- a/src/backend/nodes/tidbitmap.c
+++ b/src/backend/nodes/tidbitmap.c
@@ -172,7 +172,6 @@ struct TBMIterator
int spageptr; /* next spages index */
int schunkptr; /* next schunks index */
int schunkbit; /* next bit to check in current schunk */
- TBMIterateResult output;
};
/*
@@ -213,7 +212,6 @@ struct TBMSharedIterator
PTEntryArray *ptbase; /* pagetable element array */
PTIterationArray *ptpages; /* sorted exact page index list */
PTIterationArray *ptchunks; /* sorted lossy page index list */
- TBMIterateResult output;
};
/* Local function prototypes */
@@ -944,20 +942,21 @@ tbm_advance_schunkbit(PagetableEntry *chunk, int *schunkbitp)
/*
* tbm_iterate - scan through next page of a TIDBitmap
*
- * Returns a TBMIterateResult representing one page, or NULL if there are
- * no more pages to scan. Pages are guaranteed to be delivered in numerical
- * order. If result->ntuples < 0, then the bitmap is "lossy" and failed to
- * remember the exact tuples to look at on this page --- the caller must
- * examine all tuples on the page and check if they meet the intended
- * condition. If result->recheck is true, only the indicated tuples need
- * be examined, but the condition must be rechecked anyway. (For ease of
- * testing, recheck is always set true when ntuples < 0.)
+ * Caller must pass in a TBMIterateResult to be filled.
+ *
+ * Pages are guaranteed to be delivered in numerical order. tbmres->blockno is
+ * set to InvalidBlockNumber when there are no more pages to scan. If
+ * tbmres->ntuples < 0, then the bitmap is "lossy" and failed to remember the
+ * exact tuples to look at on this page --- the caller must examine all tuples
+ * on the page and check if they meet the intended condition. If
+ * tbmres->recheck is true, only the indicated tuples need be examined, but the
+ * condition must be rechecked anyway. (For ease of testing, recheck is always
+ * set true when ntuples < 0.)
*/
-TBMIterateResult *
-tbm_iterate(TBMIterator *iterator)
+void
+tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres)
{
TIDBitmap *tbm = iterator->tbm;
- TBMIterateResult *output = &(iterator->output);
Assert(tbm->iterating == TBM_ITERATING_PRIVATE);
@@ -985,6 +984,7 @@ tbm_iterate(TBMIterator *iterator)
* If both chunk and per-page data remain, must output the numerically
* earlier page.
*/
+ Assert(tbmres);
if (iterator->schunkptr < tbm->nchunks)
{
PagetableEntry *chunk = tbm->schunks[iterator->schunkptr];
@@ -995,11 +995,11 @@ tbm_iterate(TBMIterator *iterator)
chunk_blockno < tbm->spages[iterator->spageptr]->blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
iterator->schunkbit++;
- return output;
+ return;
}
}
@@ -1015,16 +1015,17 @@ tbm_iterate(TBMIterator *iterator)
page = tbm->spages[iterator->spageptr];
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
iterator->spageptr++;
- return output;
+ return;
}
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
@@ -1034,10 +1035,9 @@ tbm_iterate(TBMIterator *iterator)
* across multiple processes. We need to acquire the iterator LWLock,
* before accessing the shared members.
*/
-TBMIterateResult *
-tbm_shared_iterate(TBMSharedIterator *iterator)
+void
+tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres)
{
- TBMIterateResult *output = &iterator->output;
TBMSharedIteratorState *istate = iterator->state;
PagetableEntry *ptbase = NULL;
int *idxpages = NULL;
@@ -1088,13 +1088,13 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
chunk_blockno < ptbase[idxpages[istate->spageptr]].blockno)
{
/* Return a lossy page indicator from the chunk */
- output->blockno = chunk_blockno;
- output->ntuples = -1;
- output->recheck = true;
+ tbmres->blockno = chunk_blockno;
+ tbmres->ntuples = -1;
+ tbmres->recheck = true;
istate->schunkbit++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
}
@@ -1104,21 +1104,22 @@ tbm_shared_iterate(TBMSharedIterator *iterator)
int ntuples;
/* scan bitmap to extract individual offset numbers */
- ntuples = tbm_extract_page_tuple(page, output);
- output->blockno = page->blockno;
- output->ntuples = ntuples;
- output->recheck = page->recheck;
+ ntuples = tbm_extract_page_tuple(page, tbmres);
+ tbmres->blockno = page->blockno;
+ tbmres->ntuples = ntuples;
+ tbmres->recheck = page->recheck;
istate->spageptr++;
LWLockRelease(&istate->lock);
- return output;
+ return;
}
LWLockRelease(&istate->lock);
/* Nothing more in the bitmap */
- return NULL;
+ tbmres->blockno = InvalidBlockNumber;
+ return;
}
/*
diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.h
index 3013a44bae..3b432263bb 100644
--- a/src/include/access/gin_private.h
+++ b/src/include/access/gin_private.h
@@ -353,7 +353,7 @@ typedef struct GinScanEntryData
/* for a partial-match or full-scan query, we accumulate all TIDs here */
TIDBitmap *matchBitmap;
TBMIterator *matchIterator;
- TBMIterateResult *matchResult;
+ TBMIterateResult matchResult;
/* used for Posting list and one page in Posting tree */
ItemPointerData *list;
diff --git a/src/include/executor/nodeBitmapHeapscan.h b/src/include/executor/nodeBitmapHeapscan.h
index cb56d20dc6..3c330f86e6 100644
--- a/src/include/executor/nodeBitmapHeapscan.h
+++ b/src/include/executor/nodeBitmapHeapscan.h
@@ -34,7 +34,7 @@ typedef struct BitmapHeapIterator
struct TBMSharedIterator *parallel;
} BitmapHeapIterator;
-extern TBMIterateResult *bhs_iterate(BitmapHeapIterator *iterator);
+extern void bhs_iterate(BitmapHeapIterator *iterator, TBMIterateResult *result);
extern void bhs_end_iterate(BitmapHeapIterator *iterator);
diff --git a/src/include/nodes/tidbitmap.h b/src/include/nodes/tidbitmap.h
index 432fae5296..f000c1af28 100644
--- a/src/include/nodes/tidbitmap.h
+++ b/src/include/nodes/tidbitmap.h
@@ -72,8 +72,8 @@ extern bool tbm_is_empty(const TIDBitmap *tbm);
extern TBMIterator *tbm_begin_iterate(TIDBitmap *tbm);
extern dsa_pointer tbm_prepare_shared_iterate(TIDBitmap *tbm);
-extern TBMIterateResult *tbm_iterate(TBMIterator *iterator);
-extern TBMIterateResult *tbm_shared_iterate(TBMSharedIterator *iterator);
+extern void tbm_iterate(TBMIterator *iterator, TBMIterateResult *tbmres);
+extern void tbm_shared_iterate(TBMSharedIterator *iterator, TBMIterateResult *tbmres);
extern void tbm_end_iterate(TBMIterator *iterator);
extern void tbm_end_shared_iterate(TBMSharedIterator *iterator);
extern TBMSharedIterator *tbm_attach_shared_iterate(dsa_area *dsa,
--
2.40.1
--owzzsiozz6hgpp7e
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v11-0014-Push-BitmapHeapScan-prefetch-code-into-heapam.c.patch"
^ permalink raw reply [nested|flat] 59+ messages in thread
end of thread, other threads:[~2024-02-16 02:23 UTC | newest]
Thread overview: 59+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-02-07 08:08 Re: Synchronizing slots from primary to standby Ashutosh Sharma <[email protected]>
2022-02-07 20:32 ` Andres Freund <[email protected]>
2022-02-08 14:57 ` Ashutosh Sharma <[email protected]>
2022-02-10 21:47 ` Bruce Momjian <[email protected]>
2022-02-11 14:26 ` Peter Eisentraut <[email protected]>
2022-02-18 15:25 ` James Coleman <[email protected]>
2022-11-15 09:02 ` Drouvot, Bertrand <[email protected]>
2023-04-14 13:22 ` Drouvot, Bertrand <[email protected]>
2023-04-17 14:05 ` Drouvot, Bertrand <[email protected]>
2023-06-16 09:56 ` Amit Kapila <[email protected]>
2023-06-19 06:02 ` Drouvot, Bertrand <[email protected]>
2023-06-19 10:03 ` Amit Kapila <[email protected]>
2023-06-19 16:26 ` Drouvot, Bertrand <[email protected]>
2023-06-20 10:22 ` Amit Kapila <[email protected]>
2023-06-26 05:45 ` Drouvot, Bertrand <[email protected]>
2023-06-26 10:34 ` Amit Kapila <[email protected]>
2023-06-28 06:49 ` Drouvot, Bertrand <[email protected]>
2023-06-29 10:36 ` Amit Kapila <[email protected]>
2023-07-09 07:30 ` Bharath Rupireddy <[email protected]>
2023-07-10 03:36 ` Amit Kapila <[email protected]>
2023-07-20 11:34 ` shveta malik <[email protected]>
2023-07-21 06:05 ` Bharath Rupireddy <[email protected]>
2023-07-21 11:46 ` shveta malik <[email protected]>
2023-07-24 02:32 ` Bharath Rupireddy <[email protected]>
2023-07-24 03:30 ` Amit Kapila <[email protected]>
2023-07-26 05:01 ` shveta malik <[email protected]>
2023-07-27 05:24 ` Amit Kapila <[email protected]>
2023-07-28 15:24 ` Bharath Rupireddy <[email protected]>
2023-08-01 11:31 ` shveta malik <[email protected]>
2023-08-02 18:57 ` Bharath Rupireddy <[email protected]>
2023-08-03 10:49 ` shveta malik <[email protected]>
2023-07-28 14:39 ` Bharath Rupireddy <[email protected]>
2023-08-04 09:14 ` Drouvot, Bertrand <[email protected]>
2023-08-04 11:32 ` shveta malik <[email protected]>
2023-08-07 09:47 ` Drouvot, Bertrand <[email protected]>
2023-07-26 12:25 ` shveta malik <[email protected]>
2023-07-27 06:54 ` Amit Kapila <[email protected]>
2023-08-04 07:40 ` Drouvot, Bertrand <[email protected]>
2023-06-29 10:22 ` Hayato Kuroda (Fujitsu) <[email protected]>
2023-06-29 10:45 ` Amit Kapila <[email protected]>
2023-06-30 06:09 ` Drouvot, Bertrand <[email protected]>
2023-07-01 01:42 ` Amit Kapila <[email protected]>
2024-02-16 02:23 [PATCH v7 11/13] Separate TBM[Shared]Iterator and TBMIterateResult Melanie Plageman <[email protected]>
2024-02-16 02:23 [PATCH v8 12/17] Separate TBM[Shared]Iterator and TBMIterateResult Melanie Plageman <[email protected]>
2024-02-16 02:23 [PATCH v7 11/13] Separate TBM[Shared]Iterator and TBMIterateResult Melanie Plageman <[email protected]>
2024-02-16 02:23 [PATCH v8 12/17] Separate TBM[Shared]Iterator and TBMIterateResult Melanie Plageman <[email protected]>
2024-02-16 02:23 [PATCH v7 11/13] Separate TBM[Shared]Iterator and TBMIterateResult Melanie Plageman <[email protected]>
2024-02-16 02:23 [PATCH v12 13/17] Separate TBM[Shared]Iterator and TBMIterateResult Melanie Plageman <[email protected]>
2024-02-16 02:23 [PATCH v6 12/14] Separate TBM[Shared]Iterator and TBMIterateResult Melanie Plageman <[email protected]>
2024-02-16 02:23 [PATCH v6 12/14] Separate TBM[Shared]Iterator and TBMIterateResult Melanie Plageman <[email protected]>
2024-02-16 02:23 [PATCH v10 13/17] Separate TBM[Shared]Iterator and TBMIterateResult Melanie Plageman <[email protected]>
2024-02-16 02:23 [PATCH v11 13/17] Separate TBM[Shared]Iterator and TBMIterateResult Melanie Plageman <[email protected]>
2024-02-16 02:23 [PATCH v6 12/14] Separate TBM[Shared]Iterator and TBMIterateResult Melanie Plageman <[email protected]>
2024-02-16 02:23 [PATCH v8 12/17] Separate TBM[Shared]Iterator and TBMIterateResult Melanie Plageman <[email protected]>
2024-02-16 02:23 [PATCH v9 12/17] Separate TBM[Shared]Iterator and TBMIterateResult Melanie Plageman <[email protected]>
2024-02-16 02:23 [PATCH v10 13/17] Separate TBM[Shared]Iterator and TBMIterateResult Melanie Plageman <[email protected]>
2024-02-16 02:23 [PATCH v9 12/17] Separate TBM[Shared]Iterator and TBMIterateResult Melanie Plageman <[email protected]>
2024-02-16 02:23 [PATCH v9 12/17] Separate TBM[Shared]Iterator and TBMIterateResult Melanie Plageman <[email protected]>
2024-02-16 02:23 [PATCH v11 13/17] Separate TBM[Shared]Iterator and TBMIterateResult Melanie Plageman <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox