agora inbox for [email protected]
help / color / mirror / Atom feedComment fix and question about dshash.c
50+ messages / 13 participants
[nested] [flat]
* Comment fix and question about dshash.c
@ 2018-10-26 11:32 Antonin Houska <[email protected]>
2018-10-26 21:03 ` Re: Comment fix and question about dshash.c Thomas Munro <[email protected]>
0 siblings, 1 reply; 50+ messages in thread
From: Antonin Houska @ 2018-10-26 11:32 UTC (permalink / raw)
To: pgsql-hackers
1. The return type of resize() function is void, so I propose part of the
header comment to be removed:
diff --git a/src/backend/lib/dshash.c b/src/backend/lib/dshash.c
index b46f7c4cfd..b2b8fe60e1 100644
--- a/src/backend/lib/dshash.c
+++ b/src/backend/lib/dshash.c
@@ -672,9 +672,7 @@ delete_item(dshash_table *hash_table, dshash_table_item *item)
/*
* Grow the hash table if necessary to the requested number of buckets. The
- * requested size must be double some previously observed size. Returns true
- * if the table was successfully expanded or found to be big enough already
- * (because another backend expanded it).
+ * requested size must be double some previously observed size.
*
* Must be called without any partition lock held.
*/
2. Can anyone please explain this macro?
/* Max entries before we need to grow. Half + quarter = 75% load factor. */
#define MAX_COUNT_PER_PARTITION(hash_table) \
(BUCKETS_PER_PARTITION(hash_table->size_log2) / 2 + \
BUCKETS_PER_PARTITION(hash_table->size_log2) / 4)
I'm failing to understand why the maximum number of hash table entries in a
partition should be smaller than the number of buckets in that partition.
The fact that MAX_COUNT_PER_PARTITION refers to entries and not buckets is
obvious from this condition in dshash_find_or_insert()
/* Check if we are getting too full. */
if (partition->count > MAX_COUNT_PER_PARTITION(hash_table))
--
Antonin Houska
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26, A-2700 Wiener Neustadt
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Comment fix and question about dshash.c
2018-10-26 11:32 Comment fix and question about dshash.c Antonin Houska <[email protected]>
@ 2018-10-26 21:03 ` Thomas Munro <[email protected]>
2018-10-26 21:38 ` Re: Comment fix and question about dshash.c Thomas Munro <[email protected]>
2018-10-27 11:22 ` Re: Comment fix and question about dshash.c Antonin Houska <[email protected]>
0 siblings, 2 replies; 50+ messages in thread
From: Thomas Munro @ 2018-10-26 21:03 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: pgsql-hackers
On Sat, Oct 27, 2018 at 12:30 AM Antonin Houska <[email protected]> wrote:
> 1. The return type of resize() function is void, so I propose part of the
> header comment to be removed:
>
> diff --git a/src/backend/lib/dshash.c b/src/backend/lib/dshash.c
> index b46f7c4cfd..b2b8fe60e1 100644
> --- a/src/backend/lib/dshash.c
> +++ b/src/backend/lib/dshash.c
> @@ -672,9 +672,7 @@ delete_item(dshash_table *hash_table, dshash_table_item *item)
>
> /*
> * Grow the hash table if necessary to the requested number of buckets. The
> - * requested size must be double some previously observed size. Returns true
> - * if the table was successfully expanded or found to be big enough already
> - * (because another backend expanded it).
> + * requested size must be double some previously observed size.
> *
> * Must be called without any partition lock held.
> */
Thanks, will fix.
> 2. Can anyone please explain this macro?
>
> /* Max entries before we need to grow. Half + quarter = 75% load factor. */
> #define MAX_COUNT_PER_PARTITION(hash_table) \
> (BUCKETS_PER_PARTITION(hash_table->size_log2) / 2 + \
> BUCKETS_PER_PARTITION(hash_table->size_log2) / 4)
>
> I'm failing to understand why the maximum number of hash table entries in a
> partition should be smaller than the number of buckets in that partition.
>
> The fact that MAX_COUNT_PER_PARTITION refers to entries and not buckets is
> obvious from this condition in dshash_find_or_insert()
>
> /* Check if we are getting too full. */
> if (partition->count > MAX_COUNT_PER_PARTITION(hash_table))
Are you saying there is a bug in this logic (which is nbuckets * 0.75
expressed as integer maths), or saying that 0.75 is not a good maximum
load factor? I looked around at a couple of general purpose hash
tables and saw that some used 0.75 and some used 1.0, as a wasted
space-vs-collision trade-off. If I have my maths right[1], with 0.75
you expect to have 75 entries in ~53 buckets, but with 1.0 you expect
to have 100 entries in ~64 buckets. It'd be a fair criticism that
that's arbitrarily different to the choice made for hash joins, and
dynahash's default is 1 (though it's a run-time parameter).
[1] https://math.stackexchange.com/questions/470662/average-number-of-bins-occupied-when-throwing-n-ball...
--
Thomas Munro
http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Comment fix and question about dshash.c
2018-10-26 11:32 Comment fix and question about dshash.c Antonin Houska <[email protected]>
2018-10-26 21:03 ` Re: Comment fix and question about dshash.c Thomas Munro <[email protected]>
@ 2018-10-26 21:38 ` Thomas Munro <[email protected]>
1 sibling, 0 replies; 50+ messages in thread
From: Thomas Munro @ 2018-10-26 21:38 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: pgsql-hackers
On Sat, Oct 27, 2018 at 10:03 AM Thomas Munro
<[email protected]> wrote:
> space-vs-collision trade-off. If I have my maths right[1], with 0.75
> you expect to have 75 entries in ~53 buckets, but with 1.0 you expect
> to have 100 entries in ~64 buckets. It'd be a fair criticism that
Forgot to add: assuming 100 buckets, for illustration (the real number
is a power of 2).
--
Thomas Munro
http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Comment fix and question about dshash.c
2018-10-26 11:32 Comment fix and question about dshash.c Antonin Houska <[email protected]>
2018-10-26 21:03 ` Re: Comment fix and question about dshash.c Thomas Munro <[email protected]>
@ 2018-10-27 11:22 ` Antonin Houska <[email protected]>
2018-10-27 11:51 ` Re: Comment fix and question about dshash.c Antonin Houska <[email protected]>
1 sibling, 1 reply; 50+ messages in thread
From: Antonin Houska @ 2018-10-27 11:22 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: pgsql-hackers
Thomas Munro <[email protected]> wrote:
> On Sat, Oct 27, 2018 at 12:30 AM Antonin Houska <[email protected]> wrote:
>
> Thanks, will fix.
>
> > 2. Can anyone please explain this macro?
> >
> > /* Max entries before we need to grow. Half + quarter = 75% load factor. */
> > #define MAX_COUNT_PER_PARTITION(hash_table) \
> > (BUCKETS_PER_PARTITION(hash_table->size_log2) / 2 + \
> > BUCKETS_PER_PARTITION(hash_table->size_log2) / 4)
> >
> > I'm failing to understand why the maximum number of hash table entries in a
> > partition should be smaller than the number of buckets in that partition.
> >
> > The fact that MAX_COUNT_PER_PARTITION refers to entries and not buckets is
> > obvious from this condition in dshash_find_or_insert()
> >
> > /* Check if we are getting too full. */
> > if (partition->count > MAX_COUNT_PER_PARTITION(hash_table))
>
> Are you saying there is a bug in this logic (which is nbuckets * 0.75
> expressed as integer maths), or saying that 0.75 is not a good maximum
> load factor? I looked around at a couple of general purpose hash
> tables and saw that some used 0.75 and some used 1.0, as a wasted
> space-vs-collision trade-off. If I have my maths right[1], with 0.75
> you expect to have 75 entries in ~53 buckets, but with 1.0 you expect
> to have 100 entries in ~64 buckets.
I don't know how exactly you apply the [1] formula (what is "n" and what is
"N" in your case?), but my consideration was much simpler: For example, if
BUCKETS_PER_PARTITION returns 8 (power of 2 is expected here and also more
convenient), then MAX_COUNT_PER_PARTITION returns 8 / 2 + 8 / 4 = 6. Thus the
hashtable gets resized if we're going to add the 7th entry to the partition,
i.e. we the number of entries in the partition is lower than the number of
buckets. Is that o.k.?
> [1] https://math.stackexchange.com/questions/470662/average-number-of-bins-occupied-when-throwing-n-ball...
--
Antonin Houska
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26, A-2700 Wiener Neustadt
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Comment fix and question about dshash.c
2018-10-26 11:32 Comment fix and question about dshash.c Antonin Houska <[email protected]>
2018-10-26 21:03 ` Re: Comment fix and question about dshash.c Thomas Munro <[email protected]>
2018-10-27 11:22 ` Re: Comment fix and question about dshash.c Antonin Houska <[email protected]>
@ 2018-10-27 11:51 ` Antonin Houska <[email protected]>
2018-10-27 12:22 ` Re: Comment fix and question about dshash.c Tomas Vondra <[email protected]>
0 siblings, 1 reply; 50+ messages in thread
From: Antonin Houska @ 2018-10-27 11:51 UTC (permalink / raw)
To: ; +Cc: Thomas Munro <[email protected]>; pgsql-hackers
Antonin Houska <[email protected]> wrote:
> Thomas Munro <[email protected]> wrote:
>
> > On Sat, Oct 27, 2018 at 12:30 AM Antonin Houska <[email protected]> wrote:
> >
> > Are you saying there is a bug in this logic (which is nbuckets * 0.75
> > expressed as integer maths), or saying that 0.75 is not a good maximum
> > load factor? I looked around at a couple of general purpose hash
> > tables and saw that some used 0.75 and some used 1.0, as a wasted
> > space-vs-collision trade-off. If I have my maths right[1], with 0.75
> > you expect to have 75 entries in ~53 buckets, but with 1.0 you expect
> > to have 100 entries in ~64 buckets.
>
> I don't know how exactly you apply the [1] formula (what is "n" and what is
> "N" in your case?), but my consideration was much simpler: For example, if
> BUCKETS_PER_PARTITION returns 8 (power of 2 is expected here and also more
> convenient), then MAX_COUNT_PER_PARTITION returns 8 / 2 + 8 / 4 = 6. Thus the
> hashtable gets resized if we're going to add the 7th entry to the partition,
> i.e. we the number of entries in the partition is lower than the number of
> buckets. Is that o.k.?
Well, it may be o.k. I've just checked what the fill factor means in hash
index and it's also the number of entries divided by the number of buckets.
--
Antonin Houska
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26, A-2700 Wiener Neustadt
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Comment fix and question about dshash.c
2018-10-26 11:32 Comment fix and question about dshash.c Antonin Houska <[email protected]>
2018-10-26 21:03 ` Re: Comment fix and question about dshash.c Thomas Munro <[email protected]>
2018-10-27 11:22 ` Re: Comment fix and question about dshash.c Antonin Houska <[email protected]>
2018-10-27 11:51 ` Re: Comment fix and question about dshash.c Antonin Houska <[email protected]>
@ 2018-10-27 12:22 ` Tomas Vondra <[email protected]>
2018-10-28 02:27 ` Re: Comment fix and question about dshash.c Thomas Munro <[email protected]>
0 siblings, 1 reply; 50+ messages in thread
From: Tomas Vondra @ 2018-10-27 12:22 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Thomas Munro <[email protected]>; pgsql-hackers
On 10/27/2018 01:51 PM, Antonin Houska wrote:
> Antonin Houska <[email protected]> wrote:
>
>> Thomas Munro <[email protected]> wrote:
>>
>>> On Sat, Oct 27, 2018 at 12:30 AM Antonin Houska <[email protected]> wrote:
>>>
>>> Are you saying there is a bug in this logic (which is nbuckets * 0.75
>>> expressed as integer maths), or saying that 0.75 is not a good maximum
>>> load factor? I looked around at a couple of general purpose hash
>>> tables and saw that some used 0.75 and some used 1.0, as a wasted
>>> space-vs-collision trade-off. If I have my maths right[1], with 0.75
>>> you expect to have 75 entries in ~53 buckets, but with 1.0 you expect
>>> to have 100 entries in ~64 buckets.
>>
>> I don't know how exactly you apply the [1] formula (what is "n" and what is
>> "N" in your case?), but my consideration was much simpler: For example, if
>> BUCKETS_PER_PARTITION returns 8 (power of 2 is expected here and also more
>> convenient), then MAX_COUNT_PER_PARTITION returns 8 / 2 + 8 / 4 = 6. Thus the
>> hashtable gets resized if we're going to add the 7th entry to the partition,
>> i.e. we the number of entries in the partition is lower than the number of
>> buckets. Is that o.k.?
>
> Well, it may be o.k. I've just checked what the fill factor means in hash
> index and it's also the number of entries divided by the number of buckets.
>
Using load factor ~0.75 is definitely the right thing to do. One way to
interpret it is "average chain length" (or whatever is the approach in
that particular hash table implementations) and one of the main points
of hash tables is to eliminate linear scans. We could pick a value
closer to 1.0, but experience says it's not worth it - it's easy to get
a annoyingly long chains due to hash collisions, in exchange for fairly
minimal space savings.
That being said, I wonder if we should tweak NTUP_PER_BUCKET=1 in hash
joins to a lower value. IIRC we ended up with 1.0 because originally it
was set to 10.0, and we reduced it to 1.0 in 9.5 which gave us nice
speedup. But I don't recall if we tried using even lower value (probably
not). Maybe we don't need to do that because we only build the hash
table at the very end, when we exactly know how many entries will it
contain, so we don't need to do lookups and inserts at the same time,
and we don't need to grow the hash table (at least in the non-parallel
case). And we end up with 0.75 load factor on average, due to the
doubling (the sizes are essentially uniformly distributed between
0.5+epsilon and 1.0-epsilon).
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Comment fix and question about dshash.c
2018-10-26 11:32 Comment fix and question about dshash.c Antonin Houska <[email protected]>
2018-10-26 21:03 ` Re: Comment fix and question about dshash.c Thomas Munro <[email protected]>
2018-10-27 11:22 ` Re: Comment fix and question about dshash.c Antonin Houska <[email protected]>
2018-10-27 11:51 ` Re: Comment fix and question about dshash.c Antonin Houska <[email protected]>
2018-10-27 12:22 ` Re: Comment fix and question about dshash.c Tomas Vondra <[email protected]>
@ 2018-10-28 02:27 ` Thomas Munro <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Thomas Munro @ 2018-10-28 02:27 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Antonin Houska <[email protected]>; pgsql-hackers
On Sun, Oct 28, 2018 at 1:22 AM Tomas Vondra
<[email protected]> wrote:
> On 10/27/2018 01:51 PM, Antonin Houska wrote:
> > Antonin Houska <[email protected]> wrote:
> >> Thomas Munro <[email protected]> wrote:
> >>> On Sat, Oct 27, 2018 at 12:30 AM Antonin Houska <[email protected]> wrote:
> >>> Are you saying there is a bug in this logic (which is nbuckets * 0.75
> >>> expressed as integer maths), or saying that 0.75 is not a good maximum
> >>> load factor? I looked around at a couple of general purpose hash
> >>> tables and saw that some used 0.75 and some used 1.0, as a wasted
> >>> space-vs-collision trade-off. If I have my maths right[1], with 0.75
> >>> you expect to have 75 entries in ~53 buckets, but with 1.0 you expect
> >>> to have 100 entries in ~64 buckets.
> >>
> >> I don't know how exactly you apply the [1] formula (what is "n" and what is
> >> "N" in your case?), but my consideration was much simpler: For example, if
> >> BUCKETS_PER_PARTITION returns 8 (power of 2 is expected here and also more
> >> convenient), then MAX_COUNT_PER_PARTITION returns 8 / 2 + 8 / 4 = 6. Thus the
> >> hashtable gets resized if we're going to add the 7th entry to the partition,
> >> i.e. we the number of entries in the partition is lower than the number of
> >> buckets. Is that o.k.?
n balls = the keys being hashed and insert
N bins = the hash table buckets
Unless we know of some special properties of the hash function and
input data, I believe we have to treat the hashes like uniformly
distributed random numbers when making predictions, hence the
balls-into-bins probability stuff that can tell you about the expected
distribution.
The expected number of occupied bins for 1 million balls into 1 million bins:
select 1000000.0 * (1.0 - pow(1.0 - (1.0 / 1000000), 1000000.0));
-> 632120.7427683549057142050000000
The same thing by counting distinct positive random numbers modulo 1 million:
select count(distinct (random() * 200000000)::int % 1000000) from
generate_series(1, 1000000) ss;
-> 632373, 632246, 631954, ...
Distinct hashes of the first 1 million integers (arbitrary keys)
modulo 1 million (buckets):
select count(distinct abs(hashoid(n)) % 1000000) from
generate_series(1, 1000000) ss(n);
-> 632115
(For a moment I was confused about getting a higher number until I
realised I need abs() to avoid doubling the effective number of
buckets...)
> > Well, it may be o.k. I've just checked what the fill factor means in hash
> > index and it's also the number of entries divided by the number of buckets.
> >
>
> Using load factor ~0.75 is definitely the right thing to do. One way to
> interpret it is "average chain length" (or whatever is the approach in
> that particular hash table implementations) and one of the main points
> of hash tables is to eliminate linear scans. We could pick a value
> closer to 1.0, but experience says it's not worth it - it's easy to get
> a annoyingly long chains due to hash collisions, in exchange for fairly
> minimal space savings.
Using the hashes of the first 1 million integers, let's try putting
them into different numbers of buckets, and see how many buckets we
get with each chain length:
WITH entries AS (SELECT abs(hashoid(generate_series(1, 1000000))) %
NBUCKETS AS bucket),
lengths AS (SELECT bucket, COUNT(*) AS chain_length FROM entries
GROUP BY bucket)
SELECT chain_length, COUNT(*) AS count
FROM lengths GROUP BY chain_length ORDER BY 2 DESC;
NBUCKETS = 1000000 (load factor 1.0)
chain_length | count
--------------+--------
1 | 367537
2 | 184580
3 | 61109
4 | 15197
5 | 3079
6 | 509
7 | 95
8 | 7
9 | 2
NBUCKETS = 1333333 (load factor 0.75)
chain_length | count
--------------+--------
1 | 472012
2 | 177174
3 | 44348
4 | 8361
5 | 1243
6 | 143
7 | 9
8 | 2
NBUCKETS = 2000000 (load factor 0.5)
chain_length | count
--------------+--------
1 | 606451
2 | 151741
3 | 25226
4 | 3148
5 | 323
6 | 28
7 | 2
> That being said, I wonder if we should tweak NTUP_PER_BUCKET=1 in hash
> joins to a lower value. IIRC we ended up with 1.0 because originally it
> was set to 10.0, and we reduced it to 1.0 in 9.5 which gave us nice
> speedup. But I don't recall if we tried using even lower value (probably
> not). Maybe we don't need to do that because we only build the hash
> table at the very end, when we exactly know how many entries will it
> contain, so we don't need to do lookups and inserts at the same time,
> and we don't need to grow the hash table (at least in the non-parallel
> case). And we end up with 0.75 load factor on average, due to the
> doubling (the sizes are essentially uniformly distributed between
> 0.5+epsilon and 1.0-epsilon).
Yeah, it would indeed be interesting to experiment with load factors
lower than 1.0. Every link in the chain is a cache miss. In future
we should work on mitigating that by prefetching during both building
and probing (see nearby thread), but I suspect you can only do that
effectively for the bucket header and perhaps the first tuple in the
chain; if I'm right then longer chains will become even more expensive
relative to short ones.
By the way, aside from wasting memory, when you add extra buckets you
make full/right outer hash joins slower because they scan all the
buckets.
--
Thomas Munro
http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Add 64-bit XIDs into PostgreSQL 15
@ 2022-12-09 12:49 Aleksander Alekseev <[email protected]>
2022-12-09 13:54 ` Re: Add 64-bit XIDs into PostgreSQL 15 adherent postgres <[email protected]>
0 siblings, 1 reply; 50+ messages in thread
From: Aleksander Alekseev @ 2022-12-09 12:49 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: adherent postgres <[email protected]>; Chris Travers <[email protected]>; Chris Travers <[email protected]>; Bruce Momjian <[email protected]>
Hi adherent,
> Robertmhaas said that the project Zheap is dead（https://twitter.com/andy_pavlo/status/1590703943176589312）, which means that we cannot use Zheap to deal with the issue of xid wraparound and dead tuples in tables. The dead tuple issue is not a big deal because I can still use pg_repack to handle, although pg_repack will cause wal log to increase dramatically and may take one or two days to handle a large table. During this time the database can be accessed by external users, but the xid wraparound will cause PostgreSQL to be down, which is a disaster for DBAs. Maybe you are not a DBA, or your are from a small country, Database system tps is very low, so xid32 is enough for your database system , Oracle's scn was also 32bits, however, Oracle realized the issue and changed scn to 64 bits. The transaction id in mysql is 48 bits. MySQL didn't fix the transaction id wraparound problem because they think that 48 bits is enough for the transaction id. This project has been running for almost 1 year and now it is coming to an end. I strongly disagree with your idea of stopping this patch, and I suspect you are a saboteur. I strongly disagree with your viewpoint, as it is not a fundamental way to solve the xid wraparound problem. The PostgreSQL community urgently needs developers who solve problems like this, not bury one' head in the sand
This is not uncommon for people on the mailing list to have
disagreements. This is part of the process, we all are looking for
consensus. It's true that different people have different use cases in
mind and different backgrounds as well. It doesn't mean these use
cases are wrong and/or the experience is irrelevant and/or the
received feedback should be just discarded.
Although I also expressed my disagreement with Chris before, let's not
assume any bad intent and especially sabotage as you put it. (Unless
you have a strong proof of this of course which I doubt you have.) We
want all kinds of feedback to be welcomed here. I'm sure our goal here
is mutual, to make PostgreSQL even better than it is now. The only
problem is that the definition of "better" varies sometimes.
I see you believe that 64-bit XIDs are going to be useful. That's
great! Tell us more about your case and how the patch is going to help
with it. Also, maybe you could test your load with the applied
patchset and tell us whether it makes things better or worse?
Personally I would love hearing this from you.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Add 64-bit XIDs into PostgreSQL 15
2022-12-09 12:49 Re: Add 64-bit XIDs into PostgreSQL 15 Aleksander Alekseev <[email protected]>
@ 2022-12-09 13:54 ` adherent postgres <[email protected]>
2022-12-09 14:13 ` Re: Add 64-bit XIDs into PostgreSQL 15 Pavel Borisov <[email protected]>
2022-12-09 14:29 ` Re: Add 64-bit XIDs into PostgreSQL 15 Maxim Orlov <[email protected]>
0 siblings, 2 replies; 50+ messages in thread
From: adherent postgres @ 2022-12-09 13:54 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; [email protected] <[email protected]>; +Cc: Chris Travers <[email protected]>; Chris Travers <[email protected]>; Bruce Momjian <[email protected]>
Hi Aleksander Alekseev
I think the xids 32bit transformation project has been dragged on for too long. Huawei's openGauss referenced this patch to implement xids 64bit, and Postgrespro also implemented xids 64bit, which is enough to prove that their worries are redundant.I think postgresql has no reason not to implement xid 64 bit. What about your opinion?
Best whish
________________________________
发件人: Aleksander Alekseev <[email protected]>
发送时间: 2022年12月9日 20:49
收件人: [email protected] <[email protected]>
抄送: adherent postgres <[email protected]>; Chris Travers <[email protected]>; Chris Travers <[email protected]>; Bruce Momjian <[email protected]>
主题: Re: Add 64-bit XIDs into PostgreSQL 15
Hi adherent,
> Robertmhaas said that the project Zheap is dead（https://twitter.com/andy_pavlo/status/1590703943176589312）, which means that we cannot use Zheap to deal with the issue of xid wraparound and dead tuples in tables. The dead tuple issue is not a big deal because I can still use pg_repack to handle, although pg_repack will cause wal log to increase dramatically and may take one or two days to handle a large table. During this time the database can be accessed by external users, but the xid wraparound will cause PostgreSQL to be down, which is a disaster for DBAs. Maybe you are not a DBA, or your are from a small country, Database system tps is very low, so xid32 is enough for your database system , Oracle's scn was also 32bits, however, Oracle realized the issue and changed scn to 64 bits. The transaction id in mysql is 48 bits. MySQL didn't fix the transaction id wraparound problem because they think that 48 bits is enough for the transaction id. This project has been running for almost 1 year and now it is coming to an end. I strongly disagree with your idea of stopping this patch, and I suspect you are a saboteur. I strongly disagree with your viewpoint, as it is not a fundamental way to solve the xid wraparound problem. The PostgreSQL community urgently needs developers who solve problems like this, not bury one' head in the sand
This is not uncommon for people on the mailing list to have
disagreements. This is part of the process, we all are looking for
consensus. It's true that different people have different use cases in
mind and different backgrounds as well. It doesn't mean these use
cases are wrong and/or the experience is irrelevant and/or the
received feedback should be just discarded.
Although I also expressed my disagreement with Chris before, let's not
assume any bad intent and especially sabotage as you put it. (Unless
you have a strong proof of this of course which I doubt you have.) We
want all kinds of feedback to be welcomed here. I'm sure our goal here
is mutual, to make PostgreSQL even better than it is now. The only
problem is that the definition of "better" varies sometimes.
I see you believe that 64-bit XIDs are going to be useful. That's
great! Tell us more about your case and how the patch is going to help
with it. Also, maybe you could test your load with the applied
patchset and tell us whether it makes things better or worse?
Personally I would love hearing this from you.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Add 64-bit XIDs into PostgreSQL 15
2022-12-09 12:49 Re: Add 64-bit XIDs into PostgreSQL 15 Aleksander Alekseev <[email protected]>
2022-12-09 13:54 ` Re: Add 64-bit XIDs into PostgreSQL 15 adherent postgres <[email protected]>
@ 2022-12-09 14:13 ` Pavel Borisov <[email protected]>
1 sibling, 0 replies; 50+ messages in thread
From: Pavel Borisov @ 2022-12-09 14:13 UTC (permalink / raw)
To: adherent postgres <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; [email protected] <[email protected]>; Chris Travers <[email protected]>; Chris Travers <[email protected]>; Bruce Momjian <[email protected]>
Hi, Adherent!
On Fri, 9 Dec 2022 at 17:54, adherent postgres
<[email protected]> wrote:
>
> Hi Aleksander Alekseev
> I think the xids 32bit transformation project has been dragged on for too long. Huawei's openGauss referenced this patch to implement xids 64bit, and Postgrespro also implemented xids 64bit, which is enough to prove that their worries are redundant.I think postgresql has no reason not to implement xid 64 bit. What about your opinion?
I agree it's high time to commit 64xids into PostgreSQL.
If you can do your review of the whole proposed patchset or only the
part that is likely to be committed earlier [1] it would help a lot!
I'd recommend beginning with the last version of the patch in thread
[1]. First, it is easier. Also, this review is going to be useful
sooner and will help a committer on January commitfest a lot.
[1]: https://www.postgresql.org/message-id/CAFiTN-uudj2PY8GsUzFtLYFpBoq_rKegW3On_8ZHdxB1mVv3-A%40mail.gma...
Regards,
Pavel Borisov,
Supabase
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Add 64-bit XIDs into PostgreSQL 15
2022-12-09 12:49 Re: Add 64-bit XIDs into PostgreSQL 15 Aleksander Alekseev <[email protected]>
2022-12-09 13:54 ` Re: Add 64-bit XIDs into PostgreSQL 15 adherent postgres <[email protected]>
@ 2022-12-09 14:29 ` Maxim Orlov <[email protected]>
1 sibling, 0 replies; 50+ messages in thread
From: Maxim Orlov @ 2022-12-09 14:29 UTC (permalink / raw)
To: adherent postgres <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; [email protected] <[email protected]>; Chris Travers <[email protected]>; Chris Travers <[email protected]>; Bruce Momjian <[email protected]>
On Fri, 9 Dec 2022 at 16:54, adherent postgres <
[email protected]> wrote:
> Hi Aleksander Alekseev
> I think the xids 32bit transformation project has been dragged on for too
> long. Huawei's openGauss referenced this patch to implement xids 64bit, and
> Postgrespro also implemented xids 64bit, which is enough to prove that
> their worries are redundant.I think postgresql has no reason not to
> implement xid 64 bit. What about your opinion?
>
Yeah, I totally agree, the time has come. With a high transaction load,
Postgres become more and more difficult to maintain.
The problem is in the overall complexity of the patch set. We need more
reviewers.
Since committing such a big patch is not viable once at a time, from the
start of the work we did split it into several logical parts.
The evolution concept is more preferable in this case. As far as I can see,
there is overall consensus to commit SLRU related
changes first.
--
Best regards,
Maxim Orlov.
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH v11 06/17] table_scan_bitmap_next_block() returns lossy or exact
@ 2024-02-27 01:34 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Melanie Plageman @ 2024-02-27 01:34 UTC (permalink / raw)
Future commits will remove the TBMIterateResult from BitmapHeapNext() --
pushing it into the table AM-specific code. So, the table AM must inform
BitmapHeapNext() whether or not the current block is lossy or exact for
the purposes of the counters used in EXPLAIN.
---
src/backend/access/heap/heapam_handler.c | 5 ++++-
src/backend/executor/nodeBitmapHeapscan.c | 10 +++++-----
src/include/access/tableam.h | 14 ++++++++++----
3 files changed, 19 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 5ba8cb3657..ce47a158ae 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2172,7 +2172,8 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths,
static bool
heapam_scan_bitmap_next_block(TableScanDesc scan,
- TBMIterateResult *tbmres)
+ TBMIterateResult *tbmres,
+ bool *lossy)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
BlockNumber block = tbmres->blockno;
@@ -2300,6 +2301,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
+ *lossy = tbmres->ntuples < 0;
+
return ntup > 0;
}
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 404de0595e..c95e3412da 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -211,7 +211,7 @@ BitmapHeapNext(BitmapHeapScanState *node)
for (;;)
{
- bool valid;
+ bool valid, lossy;
CHECK_FOR_INTERRUPTS();
@@ -232,12 +232,12 @@ BitmapHeapNext(BitmapHeapScanState *node)
BitmapAdjustPrefetchIterator(node, tbmres->blockno);
- valid = table_scan_bitmap_next_block(scan, tbmres);
+ valid = table_scan_bitmap_next_block(scan, tbmres, &lossy);
- if (tbmres->ntuples >= 0)
- node->exact_pages++;
- else
+ if (lossy)
node->lossy_pages++;
+ else
+ node->exact_pages++;
if (!valid)
{
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index ae758857bd..5c2a7b7422 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -815,6 +815,9 @@ typedef struct TableAmRoutine
* on the page have to be returned, otherwise the tuples at offsets in
* `tbmres->offsets` need to be returned.
*
+ * lossy indicates whether or not the block's representation in the bitmap
+ * is lossy or exact.
+ *
* XXX: Currently this may only be implemented if the AM uses md.c as its
* storage manager, and uses ItemPointer->ip_blkid in a manner that maps
* blockids directly to the underlying storage. nodeBitmapHeapscan.c
@@ -830,7 +833,8 @@ typedef struct TableAmRoutine
* scan_bitmap_next_tuple need to exist, or neither.
*/
bool (*scan_bitmap_next_block) (TableScanDesc scan,
- struct TBMIterateResult *tbmres);
+ struct TBMIterateResult *tbmres,
+ bool *lossy);
/*
* Fetch the next tuple of a bitmap table scan into `slot` and return true
@@ -2023,14 +2027,16 @@ table_relation_estimate_size(Relation rel, int32 *attr_widths,
* Prepare to fetch / check / return tuples from `tbmres->blockno` as part of
* a bitmap table scan. `scan` needs to have been started via
* table_beginscan_bm(). Returns false if there are no tuples to be found on
- * the page, true otherwise.
+ * the page, true otherwise. lossy is set to true if bitmap is lossy for the
+ * selected block and false otherwise.
*
* Note, this is an optionally implemented function, therefore should only be
* used after verifying the presence (at plan time or such).
*/
static inline bool
table_scan_bitmap_next_block(TableScanDesc scan,
- struct TBMIterateResult *tbmres)
+ struct TBMIterateResult *tbmres,
+ bool *lossy)
{
/*
* We don't expect direct calls to table_scan_bitmap_next_block with valid
@@ -2041,7 +2047,7 @@ table_scan_bitmap_next_block(TableScanDesc scan,
elog(ERROR, "unexpected table_scan_bitmap_next_block call during logical decoding");
return scan->rs_rd->rd_tableam->scan_bitmap_next_block(scan,
- tbmres);
+ tbmres, lossy);
}
/*
--
2.40.1
--owzzsiozz6hgpp7e
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v11-0007-Reduce-scope-of-BitmapHeapScan-tbmiterator-local.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH v7 06/13] table_scan_bitmap_next_block() returns lossy or exact
@ 2024-02-27 01:34 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Melanie Plageman @ 2024-02-27 01:34 UTC (permalink / raw)
Future commits will remove the TBMIterateResult from BitmapHeapNext() --
pushing it into the table AM-specific code. So, the table AM must inform
BitmapHeapNext() whether or not the current block is lossy or exact for
the purposes of the counters used in EXPLAIN.
---
src/backend/access/heap/heapam_handler.c | 5 ++++-
src/backend/executor/nodeBitmapHeapscan.c | 10 +++++-----
src/include/access/tableam.h | 14 ++++++++++----
3 files changed, 19 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index c9b9b4c00f1..10c1c3b616b 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2112,7 +2112,8 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths,
static bool
heapam_scan_bitmap_next_block(TableScanDesc scan,
- TBMIterateResult *tbmres)
+ TBMIterateResult *tbmres,
+ bool *lossy)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
BlockNumber block = tbmres->blockno;
@@ -2240,6 +2241,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
+ *lossy = tbmres->ntuples < 0;
+
return ntup > 0;
}
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index a6b98fa12a1..b2397fe2054 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -211,7 +211,7 @@ BitmapHeapNext(BitmapHeapScanState *node)
for (;;)
{
- bool valid;
+ bool valid, lossy;
CHECK_FOR_INTERRUPTS();
@@ -232,12 +232,12 @@ BitmapHeapNext(BitmapHeapScanState *node)
BitmapAdjustPrefetchIterator(node, tbmres->blockno);
- valid = table_scan_bitmap_next_block(scan, tbmres);
+ valid = table_scan_bitmap_next_block(scan, tbmres, &lossy);
- if (tbmres->ntuples >= 0)
- node->exact_pages++;
- else
+ if (lossy)
node->lossy_pages++;
+ else
+ node->exact_pages++;
if (!valid)
{
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index c43a8b3dea5..f1d0d4b78e3 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -796,6 +796,9 @@ typedef struct TableAmRoutine
* on the page have to be returned, otherwise the tuples at offsets in
* `tbmres->offsets` need to be returned.
*
+ * lossy indicates whether or not the block's representation in the bitmap
+ * is lossy or exact.
+ *
* XXX: Currently this may only be implemented if the AM uses md.c as its
* storage manager, and uses ItemPointer->ip_blkid in a manner that maps
* blockids directly to the underlying storage. nodeBitmapHeapscan.c
@@ -811,7 +814,8 @@ typedef struct TableAmRoutine
* scan_bitmap_next_tuple need to exist, or neither.
*/
bool (*scan_bitmap_next_block) (TableScanDesc scan,
- struct TBMIterateResult *tbmres);
+ struct TBMIterateResult *tbmres,
+ bool *lossy);
/*
* Fetch the next tuple of a bitmap table scan into `slot` and return true
@@ -1951,14 +1955,16 @@ table_relation_estimate_size(Relation rel, int32 *attr_widths,
* Prepare to fetch / check / return tuples from `tbmres->blockno` as part of
* a bitmap table scan. `scan` needs to have been started via
* table_beginscan_bm(). Returns false if there are no tuples to be found on
- * the page, true otherwise.
+ * the page, true otherwise. lossy is set to true if bitmap is lossy for the
+ * selected block and false otherwise.
*
* Note, this is an optionally implemented function, therefore should only be
* used after verifying the presence (at plan time or such).
*/
static inline bool
table_scan_bitmap_next_block(TableScanDesc scan,
- struct TBMIterateResult *tbmres)
+ struct TBMIterateResult *tbmres,
+ bool *lossy)
{
/*
* We don't expect direct calls to table_scan_bitmap_next_block with valid
@@ -1969,7 +1975,7 @@ table_scan_bitmap_next_block(TableScanDesc scan,
elog(ERROR, "unexpected table_scan_bitmap_next_block call during logical decoding");
return scan->rs_rd->rd_tableam->scan_bitmap_next_block(scan,
- tbmres);
+ tbmres, lossy);
}
/*
--
2.40.1
--kqqpqghcwbcc3dt5
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0007-Reduce-scope-of-BitmapHeapScan-tbmiterator-local-.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH v6 07/14] table_scan_bitmap_next_block() returns lossy or exact
@ 2024-02-27 01:34 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Melanie Plageman @ 2024-02-27 01:34 UTC (permalink / raw)
Future commits will remove the TBMIterateResult from BitmapHeapNext() --
pushing it into the table AM-specific code. So, the table AM must inform
BitmapHeapNext() whether or not the current block is lossy or exact for
the purposes of the counters used in EXPLAIN.
---
src/backend/access/heap/heapam_handler.c | 5 ++++-
src/backend/executor/nodeBitmapHeapscan.c | 10 +++++-----
src/include/access/tableam.h | 14 ++++++++++----
3 files changed, 19 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index c9b9b4c00f1..10c1c3b616b 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2112,7 +2112,8 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths,
static bool
heapam_scan_bitmap_next_block(TableScanDesc scan,
- TBMIterateResult *tbmres)
+ TBMIterateResult *tbmres,
+ bool *lossy)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
BlockNumber block = tbmres->blockno;
@@ -2240,6 +2241,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
+ *lossy = tbmres->ntuples < 0;
+
return ntup > 0;
}
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 38b4596a775..08e346ae7b3 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -211,7 +211,7 @@ BitmapHeapNext(BitmapHeapScanState *node)
for (;;)
{
- bool valid;
+ bool valid, lossy;
CHECK_FOR_INTERRUPTS();
@@ -232,12 +232,12 @@ BitmapHeapNext(BitmapHeapScanState *node)
BitmapAdjustPrefetchIterator(node, tbmres->blockno);
- valid = table_scan_bitmap_next_block(scan, tbmres);
+ valid = table_scan_bitmap_next_block(scan, tbmres, &lossy);
- if (tbmres->ntuples >= 0)
- node->exact_pages++;
- else
+ if (lossy)
node->lossy_pages++;
+ else
+ node->exact_pages++;
if (!valid)
{
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index c43a8b3dea5..f1d0d4b78e3 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -796,6 +796,9 @@ typedef struct TableAmRoutine
* on the page have to be returned, otherwise the tuples at offsets in
* `tbmres->offsets` need to be returned.
*
+ * lossy indicates whether or not the block's representation in the bitmap
+ * is lossy or exact.
+ *
* XXX: Currently this may only be implemented if the AM uses md.c as its
* storage manager, and uses ItemPointer->ip_blkid in a manner that maps
* blockids directly to the underlying storage. nodeBitmapHeapscan.c
@@ -811,7 +814,8 @@ typedef struct TableAmRoutine
* scan_bitmap_next_tuple need to exist, or neither.
*/
bool (*scan_bitmap_next_block) (TableScanDesc scan,
- struct TBMIterateResult *tbmres);
+ struct TBMIterateResult *tbmres,
+ bool *lossy);
/*
* Fetch the next tuple of a bitmap table scan into `slot` and return true
@@ -1951,14 +1955,16 @@ table_relation_estimate_size(Relation rel, int32 *attr_widths,
* Prepare to fetch / check / return tuples from `tbmres->blockno` as part of
* a bitmap table scan. `scan` needs to have been started via
* table_beginscan_bm(). Returns false if there are no tuples to be found on
- * the page, true otherwise.
+ * the page, true otherwise. lossy is set to true if bitmap is lossy for the
+ * selected block and false otherwise.
*
* Note, this is an optionally implemented function, therefore should only be
* used after verifying the presence (at plan time or such).
*/
static inline bool
table_scan_bitmap_next_block(TableScanDesc scan,
- struct TBMIterateResult *tbmres)
+ struct TBMIterateResult *tbmres,
+ bool *lossy)
{
/*
* We don't expect direct calls to table_scan_bitmap_next_block with valid
@@ -1969,7 +1975,7 @@ table_scan_bitmap_next_block(TableScanDesc scan,
elog(ERROR, "unexpected table_scan_bitmap_next_block call during logical decoding");
return scan->rs_rd->rd_tableam->scan_bitmap_next_block(scan,
- tbmres);
+ tbmres, lossy);
}
/*
--
2.40.1
--w4wcjcocxsm37usi
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0008-Reduce-scope-of-BitmapHeapScan-tbmiterator-local-.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH v10 06/17] table_scan_bitmap_next_block() returns lossy or exact
@ 2024-02-27 01:34 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Melanie Plageman @ 2024-02-27 01:34 UTC (permalink / raw)
Future commits will remove the TBMIterateResult from BitmapHeapNext() --
pushing it into the table AM-specific code. So, the table AM must inform
BitmapHeapNext() whether or not the current block is lossy or exact for
the purposes of the counters used in EXPLAIN.
---
src/backend/access/heap/heapam_handler.c | 5 ++++-
src/backend/executor/nodeBitmapHeapscan.c | 10 +++++-----
src/include/access/tableam.h | 14 ++++++++++----
3 files changed, 19 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 7fdccaf613..849cac3947 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2114,7 +2114,8 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths,
static bool
heapam_scan_bitmap_next_block(TableScanDesc scan,
- TBMIterateResult *tbmres)
+ TBMIterateResult *tbmres,
+ bool *lossy)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
BlockNumber block = tbmres->blockno;
@@ -2242,6 +2243,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
+ *lossy = tbmres->ntuples < 0;
+
return ntup > 0;
}
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 404de0595e..c95e3412da 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -211,7 +211,7 @@ BitmapHeapNext(BitmapHeapScanState *node)
for (;;)
{
- bool valid;
+ bool valid, lossy;
CHECK_FOR_INTERRUPTS();
@@ -232,12 +232,12 @@ BitmapHeapNext(BitmapHeapScanState *node)
BitmapAdjustPrefetchIterator(node, tbmres->blockno);
- valid = table_scan_bitmap_next_block(scan, tbmres);
+ valid = table_scan_bitmap_next_block(scan, tbmres, &lossy);
- if (tbmres->ntuples >= 0)
- node->exact_pages++;
- else
+ if (lossy)
node->lossy_pages++;
+ else
+ node->exact_pages++;
if (!valid)
{
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index 1bc5f7c057..b9ba4f9fb3 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -804,6 +804,9 @@ typedef struct TableAmRoutine
* on the page have to be returned, otherwise the tuples at offsets in
* `tbmres->offsets` need to be returned.
*
+ * lossy indicates whether or not the block's representation in the bitmap
+ * is lossy or exact.
+ *
* XXX: Currently this may only be implemented if the AM uses md.c as its
* storage manager, and uses ItemPointer->ip_blkid in a manner that maps
* blockids directly to the underlying storage. nodeBitmapHeapscan.c
@@ -819,7 +822,8 @@ typedef struct TableAmRoutine
* scan_bitmap_next_tuple need to exist, or neither.
*/
bool (*scan_bitmap_next_block) (TableScanDesc scan,
- struct TBMIterateResult *tbmres);
+ struct TBMIterateResult *tbmres,
+ bool *lossy);
/*
* Fetch the next tuple of a bitmap table scan into `slot` and return true
@@ -1988,14 +1992,16 @@ table_relation_estimate_size(Relation rel, int32 *attr_widths,
* Prepare to fetch / check / return tuples from `tbmres->blockno` as part of
* a bitmap table scan. `scan` needs to have been started via
* table_beginscan_bm(). Returns false if there are no tuples to be found on
- * the page, true otherwise.
+ * the page, true otherwise. lossy is set to true if bitmap is lossy for the
+ * selected block and false otherwise.
*
* Note, this is an optionally implemented function, therefore should only be
* used after verifying the presence (at plan time or such).
*/
static inline bool
table_scan_bitmap_next_block(TableScanDesc scan,
- struct TBMIterateResult *tbmres)
+ struct TBMIterateResult *tbmres,
+ bool *lossy)
{
/*
* We don't expect direct calls to table_scan_bitmap_next_block with valid
@@ -2006,7 +2012,7 @@ table_scan_bitmap_next_block(TableScanDesc scan,
elog(ERROR, "unexpected table_scan_bitmap_next_block call during logical decoding");
return scan->rs_rd->rd_tableam->scan_bitmap_next_block(scan,
- tbmres);
+ tbmres, lossy);
}
/*
--
2.40.1
--3o7pc6dfau5a5hry
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v10-0007-Reduce-scope-of-BitmapHeapScan-tbmiterator-local.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH v9 06/17] table_scan_bitmap_next_block() returns lossy or exact
@ 2024-02-27 01:34 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Melanie Plageman @ 2024-02-27 01:34 UTC (permalink / raw)
Future commits will remove the TBMIterateResult from BitmapHeapNext() --
pushing it into the table AM-specific code. So, the table AM must inform
BitmapHeapNext() whether or not the current block is lossy or exact for
the purposes of the counters used in EXPLAIN.
---
src/backend/access/heap/heapam_handler.c | 5 ++++-
src/backend/executor/nodeBitmapHeapscan.c | 10 +++++-----
src/include/access/tableam.h | 14 ++++++++++----
3 files changed, 19 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 7fdccaf613..849cac3947 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2114,7 +2114,8 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths,
static bool
heapam_scan_bitmap_next_block(TableScanDesc scan,
- TBMIterateResult *tbmres)
+ TBMIterateResult *tbmres,
+ bool *lossy)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
BlockNumber block = tbmres->blockno;
@@ -2242,6 +2243,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
+ *lossy = tbmres->ntuples < 0;
+
return ntup > 0;
}
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 404de0595e..c95e3412da 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -211,7 +211,7 @@ BitmapHeapNext(BitmapHeapScanState *node)
for (;;)
{
- bool valid;
+ bool valid, lossy;
CHECK_FOR_INTERRUPTS();
@@ -232,12 +232,12 @@ BitmapHeapNext(BitmapHeapScanState *node)
BitmapAdjustPrefetchIterator(node, tbmres->blockno);
- valid = table_scan_bitmap_next_block(scan, tbmres);
+ valid = table_scan_bitmap_next_block(scan, tbmres, &lossy);
- if (tbmres->ntuples >= 0)
- node->exact_pages++;
- else
+ if (lossy)
node->lossy_pages++;
+ else
+ node->exact_pages++;
if (!valid)
{
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index 1bc5f7c057..b9ba4f9fb3 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -804,6 +804,9 @@ typedef struct TableAmRoutine
* on the page have to be returned, otherwise the tuples at offsets in
* `tbmres->offsets` need to be returned.
*
+ * lossy indicates whether or not the block's representation in the bitmap
+ * is lossy or exact.
+ *
* XXX: Currently this may only be implemented if the AM uses md.c as its
* storage manager, and uses ItemPointer->ip_blkid in a manner that maps
* blockids directly to the underlying storage. nodeBitmapHeapscan.c
@@ -819,7 +822,8 @@ typedef struct TableAmRoutine
* scan_bitmap_next_tuple need to exist, or neither.
*/
bool (*scan_bitmap_next_block) (TableScanDesc scan,
- struct TBMIterateResult *tbmres);
+ struct TBMIterateResult *tbmres,
+ bool *lossy);
/*
* Fetch the next tuple of a bitmap table scan into `slot` and return true
@@ -1988,14 +1992,16 @@ table_relation_estimate_size(Relation rel, int32 *attr_widths,
* Prepare to fetch / check / return tuples from `tbmres->blockno` as part of
* a bitmap table scan. `scan` needs to have been started via
* table_beginscan_bm(). Returns false if there are no tuples to be found on
- * the page, true otherwise.
+ * the page, true otherwise. lossy is set to true if bitmap is lossy for the
+ * selected block and false otherwise.
*
* Note, this is an optionally implemented function, therefore should only be
* used after verifying the presence (at plan time or such).
*/
static inline bool
table_scan_bitmap_next_block(TableScanDesc scan,
- struct TBMIterateResult *tbmres)
+ struct TBMIterateResult *tbmres,
+ bool *lossy)
{
/*
* We don't expect direct calls to table_scan_bitmap_next_block with valid
@@ -2006,7 +2012,7 @@ table_scan_bitmap_next_block(TableScanDesc scan,
elog(ERROR, "unexpected table_scan_bitmap_next_block call during logical decoding");
return scan->rs_rd->rd_tableam->scan_bitmap_next_block(scan,
- tbmres);
+ tbmres, lossy);
}
/*
--
2.40.1
--7mdtsjmrzitrgzgx
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v9-0007-Reduce-scope-of-BitmapHeapScan-tbmiterator-local-.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH v9 06/17] table_scan_bitmap_next_block() returns lossy or exact
@ 2024-02-27 01:34 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Melanie Plageman @ 2024-02-27 01:34 UTC (permalink / raw)
Future commits will remove the TBMIterateResult from BitmapHeapNext() --
pushing it into the table AM-specific code. So, the table AM must inform
BitmapHeapNext() whether or not the current block is lossy or exact for
the purposes of the counters used in EXPLAIN.
---
src/backend/access/heap/heapam_handler.c | 5 ++++-
src/backend/executor/nodeBitmapHeapscan.c | 10 +++++-----
src/include/access/tableam.h | 14 ++++++++++----
3 files changed, 19 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 7fdccaf613..849cac3947 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2114,7 +2114,8 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths,
static bool
heapam_scan_bitmap_next_block(TableScanDesc scan,
- TBMIterateResult *tbmres)
+ TBMIterateResult *tbmres,
+ bool *lossy)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
BlockNumber block = tbmres->blockno;
@@ -2242,6 +2243,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
+ *lossy = tbmres->ntuples < 0;
+
return ntup > 0;
}
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 404de0595e..c95e3412da 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -211,7 +211,7 @@ BitmapHeapNext(BitmapHeapScanState *node)
for (;;)
{
- bool valid;
+ bool valid, lossy;
CHECK_FOR_INTERRUPTS();
@@ -232,12 +232,12 @@ BitmapHeapNext(BitmapHeapScanState *node)
BitmapAdjustPrefetchIterator(node, tbmres->blockno);
- valid = table_scan_bitmap_next_block(scan, tbmres);
+ valid = table_scan_bitmap_next_block(scan, tbmres, &lossy);
- if (tbmres->ntuples >= 0)
- node->exact_pages++;
- else
+ if (lossy)
node->lossy_pages++;
+ else
+ node->exact_pages++;
if (!valid)
{
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index 1bc5f7c057..b9ba4f9fb3 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -804,6 +804,9 @@ typedef struct TableAmRoutine
* on the page have to be returned, otherwise the tuples at offsets in
* `tbmres->offsets` need to be returned.
*
+ * lossy indicates whether or not the block's representation in the bitmap
+ * is lossy or exact.
+ *
* XXX: Currently this may only be implemented if the AM uses md.c as its
* storage manager, and uses ItemPointer->ip_blkid in a manner that maps
* blockids directly to the underlying storage. nodeBitmapHeapscan.c
@@ -819,7 +822,8 @@ typedef struct TableAmRoutine
* scan_bitmap_next_tuple need to exist, or neither.
*/
bool (*scan_bitmap_next_block) (TableScanDesc scan,
- struct TBMIterateResult *tbmres);
+ struct TBMIterateResult *tbmres,
+ bool *lossy);
/*
* Fetch the next tuple of a bitmap table scan into `slot` and return true
@@ -1988,14 +1992,16 @@ table_relation_estimate_size(Relation rel, int32 *attr_widths,
* Prepare to fetch / check / return tuples from `tbmres->blockno` as part of
* a bitmap table scan. `scan` needs to have been started via
* table_beginscan_bm(). Returns false if there are no tuples to be found on
- * the page, true otherwise.
+ * the page, true otherwise. lossy is set to true if bitmap is lossy for the
+ * selected block and false otherwise.
*
* Note, this is an optionally implemented function, therefore should only be
* used after verifying the presence (at plan time or such).
*/
static inline bool
table_scan_bitmap_next_block(TableScanDesc scan,
- struct TBMIterateResult *tbmres)
+ struct TBMIterateResult *tbmres,
+ bool *lossy)
{
/*
* We don't expect direct calls to table_scan_bitmap_next_block with valid
@@ -2006,7 +2012,7 @@ table_scan_bitmap_next_block(TableScanDesc scan,
elog(ERROR, "unexpected table_scan_bitmap_next_block call during logical decoding");
return scan->rs_rd->rd_tableam->scan_bitmap_next_block(scan,
- tbmres);
+ tbmres, lossy);
}
/*
--
2.40.1
--7mdtsjmrzitrgzgx
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v9-0007-Reduce-scope-of-BitmapHeapScan-tbmiterator-local-.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH v10 06/17] table_scan_bitmap_next_block() returns lossy or exact
@ 2024-02-27 01:34 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Melanie Plageman @ 2024-02-27 01:34 UTC (permalink / raw)
Future commits will remove the TBMIterateResult from BitmapHeapNext() --
pushing it into the table AM-specific code. So, the table AM must inform
BitmapHeapNext() whether or not the current block is lossy or exact for
the purposes of the counters used in EXPLAIN.
---
src/backend/access/heap/heapam_handler.c | 5 ++++-
src/backend/executor/nodeBitmapHeapscan.c | 10 +++++-----
src/include/access/tableam.h | 14 ++++++++++----
3 files changed, 19 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 7fdccaf613..849cac3947 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2114,7 +2114,8 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths,
static bool
heapam_scan_bitmap_next_block(TableScanDesc scan,
- TBMIterateResult *tbmres)
+ TBMIterateResult *tbmres,
+ bool *lossy)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
BlockNumber block = tbmres->blockno;
@@ -2242,6 +2243,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
+ *lossy = tbmres->ntuples < 0;
+
return ntup > 0;
}
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 404de0595e..c95e3412da 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -211,7 +211,7 @@ BitmapHeapNext(BitmapHeapScanState *node)
for (;;)
{
- bool valid;
+ bool valid, lossy;
CHECK_FOR_INTERRUPTS();
@@ -232,12 +232,12 @@ BitmapHeapNext(BitmapHeapScanState *node)
BitmapAdjustPrefetchIterator(node, tbmres->blockno);
- valid = table_scan_bitmap_next_block(scan, tbmres);
+ valid = table_scan_bitmap_next_block(scan, tbmres, &lossy);
- if (tbmres->ntuples >= 0)
- node->exact_pages++;
- else
+ if (lossy)
node->lossy_pages++;
+ else
+ node->exact_pages++;
if (!valid)
{
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index 1bc5f7c057..b9ba4f9fb3 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -804,6 +804,9 @@ typedef struct TableAmRoutine
* on the page have to be returned, otherwise the tuples at offsets in
* `tbmres->offsets` need to be returned.
*
+ * lossy indicates whether or not the block's representation in the bitmap
+ * is lossy or exact.
+ *
* XXX: Currently this may only be implemented if the AM uses md.c as its
* storage manager, and uses ItemPointer->ip_blkid in a manner that maps
* blockids directly to the underlying storage. nodeBitmapHeapscan.c
@@ -819,7 +822,8 @@ typedef struct TableAmRoutine
* scan_bitmap_next_tuple need to exist, or neither.
*/
bool (*scan_bitmap_next_block) (TableScanDesc scan,
- struct TBMIterateResult *tbmres);
+ struct TBMIterateResult *tbmres,
+ bool *lossy);
/*
* Fetch the next tuple of a bitmap table scan into `slot` and return true
@@ -1988,14 +1992,16 @@ table_relation_estimate_size(Relation rel, int32 *attr_widths,
* Prepare to fetch / check / return tuples from `tbmres->blockno` as part of
* a bitmap table scan. `scan` needs to have been started via
* table_beginscan_bm(). Returns false if there are no tuples to be found on
- * the page, true otherwise.
+ * the page, true otherwise. lossy is set to true if bitmap is lossy for the
+ * selected block and false otherwise.
*
* Note, this is an optionally implemented function, therefore should only be
* used after verifying the presence (at plan time or such).
*/
static inline bool
table_scan_bitmap_next_block(TableScanDesc scan,
- struct TBMIterateResult *tbmres)
+ struct TBMIterateResult *tbmres,
+ bool *lossy)
{
/*
* We don't expect direct calls to table_scan_bitmap_next_block with valid
@@ -2006,7 +2012,7 @@ table_scan_bitmap_next_block(TableScanDesc scan,
elog(ERROR, "unexpected table_scan_bitmap_next_block call during logical decoding");
return scan->rs_rd->rd_tableam->scan_bitmap_next_block(scan,
- tbmres);
+ tbmres, lossy);
}
/*
--
2.40.1
--3o7pc6dfau5a5hry
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v10-0007-Reduce-scope-of-BitmapHeapScan-tbmiterator-local.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH v12 06/17] table_scan_bitmap_next_block() returns lossy or exact
@ 2024-02-27 01:34 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Melanie Plageman @ 2024-02-27 01:34 UTC (permalink / raw)
Future commits will remove the TBMIterateResult from BitmapHeapNext() --
pushing it into the table AM-specific code. So, the table AM must inform
BitmapHeapNext() whether or not the current block is lossy or exact for
the purposes of the counters used in EXPLAIN.
---
src/backend/access/heap/heapam_handler.c | 5 ++++-
src/backend/executor/nodeBitmapHeapscan.c | 10 +++++-----
src/include/access/tableam.h | 14 ++++++++++----
3 files changed, 19 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 4c5fe8f090d..973734e9ffa 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2199,7 +2199,8 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths,
static bool
heapam_scan_bitmap_next_block(TableScanDesc scan,
- TBMIterateResult *tbmres)
+ TBMIterateResult *tbmres,
+ bool *lossy)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
BlockNumber block = tbmres->blockno;
@@ -2327,6 +2328,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
+ *lossy = tbmres->ntuples < 0;
+
return ntup > 0;
}
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 404de0595ec..c95e3412da8 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -211,7 +211,7 @@ BitmapHeapNext(BitmapHeapScanState *node)
for (;;)
{
- bool valid;
+ bool valid, lossy;
CHECK_FOR_INTERRUPTS();
@@ -232,12 +232,12 @@ BitmapHeapNext(BitmapHeapScanState *node)
BitmapAdjustPrefetchIterator(node, tbmres->blockno);
- valid = table_scan_bitmap_next_block(scan, tbmres);
+ valid = table_scan_bitmap_next_block(scan, tbmres, &lossy);
- if (tbmres->ntuples >= 0)
- node->exact_pages++;
- else
+ if (lossy)
node->lossy_pages++;
+ else
+ node->exact_pages++;
if (!valid)
{
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index e8d04b4dec6..19e99c81092 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -811,6 +811,9 @@ typedef struct TableAmRoutine
* on the page have to be returned, otherwise the tuples at offsets in
* `tbmres->offsets` need to be returned.
*
+ * lossy indicates whether or not the block's representation in the bitmap
+ * is lossy or exact.
+ *
* XXX: Currently this may only be implemented if the AM uses md.c as its
* storage manager, and uses ItemPointer->ip_blkid in a manner that maps
* blockids directly to the underlying storage. nodeBitmapHeapscan.c
@@ -826,7 +829,8 @@ typedef struct TableAmRoutine
* scan_bitmap_next_tuple need to exist, or neither.
*/
bool (*scan_bitmap_next_block) (TableScanDesc scan,
- struct TBMIterateResult *tbmres);
+ struct TBMIterateResult *tbmres,
+ bool *lossy);
/*
* Fetch the next tuple of a bitmap table scan into `slot` and return true
@@ -2013,14 +2017,16 @@ table_relation_estimate_size(Relation rel, int32 *attr_widths,
* Prepare to fetch / check / return tuples from `tbmres->blockno` as part of
* a bitmap table scan. `scan` needs to have been started via
* table_beginscan_bm(). Returns false if there are no tuples to be found on
- * the page, true otherwise.
+ * the page, true otherwise. lossy is set to true if bitmap is lossy for the
+ * selected block and false otherwise.
*
* Note, this is an optionally implemented function, therefore should only be
* used after verifying the presence (at plan time or such).
*/
static inline bool
table_scan_bitmap_next_block(TableScanDesc scan,
- struct TBMIterateResult *tbmres)
+ struct TBMIterateResult *tbmres,
+ bool *lossy)
{
/*
* We don't expect direct calls to table_scan_bitmap_next_block with valid
@@ -2031,7 +2037,7 @@ table_scan_bitmap_next_block(TableScanDesc scan,
elog(ERROR, "unexpected table_scan_bitmap_next_block call during logical decoding");
return scan->rs_rd->rd_tableam->scan_bitmap_next_block(scan,
- tbmres);
+ tbmres, lossy);
}
/*
--
2.40.1
--6jpz2j246qmht4bt
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v12-0007-Reduce-scope-of-BitmapHeapScan-tbmiterator-local.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH v10 06/17] table_scan_bitmap_next_block() returns lossy or exact
@ 2024-02-27 01:34 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Melanie Plageman @ 2024-02-27 01:34 UTC (permalink / raw)
Future commits will remove the TBMIterateResult from BitmapHeapNext() --
pushing it into the table AM-specific code. So, the table AM must inform
BitmapHeapNext() whether or not the current block is lossy or exact for
the purposes of the counters used in EXPLAIN.
---
src/backend/access/heap/heapam_handler.c | 5 ++++-
src/backend/executor/nodeBitmapHeapscan.c | 10 +++++-----
src/include/access/tableam.h | 14 ++++++++++----
3 files changed, 19 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 7fdccaf613..849cac3947 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2114,7 +2114,8 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths,
static bool
heapam_scan_bitmap_next_block(TableScanDesc scan,
- TBMIterateResult *tbmres)
+ TBMIterateResult *tbmres,
+ bool *lossy)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
BlockNumber block = tbmres->blockno;
@@ -2242,6 +2243,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
+ *lossy = tbmres->ntuples < 0;
+
return ntup > 0;
}
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 404de0595e..c95e3412da 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -211,7 +211,7 @@ BitmapHeapNext(BitmapHeapScanState *node)
for (;;)
{
- bool valid;
+ bool valid, lossy;
CHECK_FOR_INTERRUPTS();
@@ -232,12 +232,12 @@ BitmapHeapNext(BitmapHeapScanState *node)
BitmapAdjustPrefetchIterator(node, tbmres->blockno);
- valid = table_scan_bitmap_next_block(scan, tbmres);
+ valid = table_scan_bitmap_next_block(scan, tbmres, &lossy);
- if (tbmres->ntuples >= 0)
- node->exact_pages++;
- else
+ if (lossy)
node->lossy_pages++;
+ else
+ node->exact_pages++;
if (!valid)
{
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index 1bc5f7c057..b9ba4f9fb3 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -804,6 +804,9 @@ typedef struct TableAmRoutine
* on the page have to be returned, otherwise the tuples at offsets in
* `tbmres->offsets` need to be returned.
*
+ * lossy indicates whether or not the block's representation in the bitmap
+ * is lossy or exact.
+ *
* XXX: Currently this may only be implemented if the AM uses md.c as its
* storage manager, and uses ItemPointer->ip_blkid in a manner that maps
* blockids directly to the underlying storage. nodeBitmapHeapscan.c
@@ -819,7 +822,8 @@ typedef struct TableAmRoutine
* scan_bitmap_next_tuple need to exist, or neither.
*/
bool (*scan_bitmap_next_block) (TableScanDesc scan,
- struct TBMIterateResult *tbmres);
+ struct TBMIterateResult *tbmres,
+ bool *lossy);
/*
* Fetch the next tuple of a bitmap table scan into `slot` and return true
@@ -1988,14 +1992,16 @@ table_relation_estimate_size(Relation rel, int32 *attr_widths,
* Prepare to fetch / check / return tuples from `tbmres->blockno` as part of
* a bitmap table scan. `scan` needs to have been started via
* table_beginscan_bm(). Returns false if there are no tuples to be found on
- * the page, true otherwise.
+ * the page, true otherwise. lossy is set to true if bitmap is lossy for the
+ * selected block and false otherwise.
*
* Note, this is an optionally implemented function, therefore should only be
* used after verifying the presence (at plan time or such).
*/
static inline bool
table_scan_bitmap_next_block(TableScanDesc scan,
- struct TBMIterateResult *tbmres)
+ struct TBMIterateResult *tbmres,
+ bool *lossy)
{
/*
* We don't expect direct calls to table_scan_bitmap_next_block with valid
@@ -2006,7 +2012,7 @@ table_scan_bitmap_next_block(TableScanDesc scan,
elog(ERROR, "unexpected table_scan_bitmap_next_block call during logical decoding");
return scan->rs_rd->rd_tableam->scan_bitmap_next_block(scan,
- tbmres);
+ tbmres, lossy);
}
/*
--
2.40.1
--3o7pc6dfau5a5hry
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v10-0007-Reduce-scope-of-BitmapHeapScan-tbmiterator-local.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH v6 07/14] table_scan_bitmap_next_block() returns lossy or exact
@ 2024-02-27 01:34 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Melanie Plageman @ 2024-02-27 01:34 UTC (permalink / raw)
Future commits will remove the TBMIterateResult from BitmapHeapNext() --
pushing it into the table AM-specific code. So, the table AM must inform
BitmapHeapNext() whether or not the current block is lossy or exact for
the purposes of the counters used in EXPLAIN.
---
src/backend/access/heap/heapam_handler.c | 5 ++++-
src/backend/executor/nodeBitmapHeapscan.c | 10 +++++-----
src/include/access/tableam.h | 14 ++++++++++----
3 files changed, 19 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index c9b9b4c00f1..10c1c3b616b 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2112,7 +2112,8 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths,
static bool
heapam_scan_bitmap_next_block(TableScanDesc scan,
- TBMIterateResult *tbmres)
+ TBMIterateResult *tbmres,
+ bool *lossy)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
BlockNumber block = tbmres->blockno;
@@ -2240,6 +2241,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
+ *lossy = tbmres->ntuples < 0;
+
return ntup > 0;
}
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 38b4596a775..08e346ae7b3 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -211,7 +211,7 @@ BitmapHeapNext(BitmapHeapScanState *node)
for (;;)
{
- bool valid;
+ bool valid, lossy;
CHECK_FOR_INTERRUPTS();
@@ -232,12 +232,12 @@ BitmapHeapNext(BitmapHeapScanState *node)
BitmapAdjustPrefetchIterator(node, tbmres->blockno);
- valid = table_scan_bitmap_next_block(scan, tbmres);
+ valid = table_scan_bitmap_next_block(scan, tbmres, &lossy);
- if (tbmres->ntuples >= 0)
- node->exact_pages++;
- else
+ if (lossy)
node->lossy_pages++;
+ else
+ node->exact_pages++;
if (!valid)
{
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index c43a8b3dea5..f1d0d4b78e3 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -796,6 +796,9 @@ typedef struct TableAmRoutine
* on the page have to be returned, otherwise the tuples at offsets in
* `tbmres->offsets` need to be returned.
*
+ * lossy indicates whether or not the block's representation in the bitmap
+ * is lossy or exact.
+ *
* XXX: Currently this may only be implemented if the AM uses md.c as its
* storage manager, and uses ItemPointer->ip_blkid in a manner that maps
* blockids directly to the underlying storage. nodeBitmapHeapscan.c
@@ -811,7 +814,8 @@ typedef struct TableAmRoutine
* scan_bitmap_next_tuple need to exist, or neither.
*/
bool (*scan_bitmap_next_block) (TableScanDesc scan,
- struct TBMIterateResult *tbmres);
+ struct TBMIterateResult *tbmres,
+ bool *lossy);
/*
* Fetch the next tuple of a bitmap table scan into `slot` and return true
@@ -1951,14 +1955,16 @@ table_relation_estimate_size(Relation rel, int32 *attr_widths,
* Prepare to fetch / check / return tuples from `tbmres->blockno` as part of
* a bitmap table scan. `scan` needs to have been started via
* table_beginscan_bm(). Returns false if there are no tuples to be found on
- * the page, true otherwise.
+ * the page, true otherwise. lossy is set to true if bitmap is lossy for the
+ * selected block and false otherwise.
*
* Note, this is an optionally implemented function, therefore should only be
* used after verifying the presence (at plan time or such).
*/
static inline bool
table_scan_bitmap_next_block(TableScanDesc scan,
- struct TBMIterateResult *tbmres)
+ struct TBMIterateResult *tbmres,
+ bool *lossy)
{
/*
* We don't expect direct calls to table_scan_bitmap_next_block with valid
@@ -1969,7 +1975,7 @@ table_scan_bitmap_next_block(TableScanDesc scan,
elog(ERROR, "unexpected table_scan_bitmap_next_block call during logical decoding");
return scan->rs_rd->rd_tableam->scan_bitmap_next_block(scan,
- tbmres);
+ tbmres, lossy);
}
/*
--
2.40.1
--w4wcjcocxsm37usi
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0008-Reduce-scope-of-BitmapHeapScan-tbmiterator-local-.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH v6 07/14] table_scan_bitmap_next_block() returns lossy or exact
@ 2024-02-27 01:34 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Melanie Plageman @ 2024-02-27 01:34 UTC (permalink / raw)
Future commits will remove the TBMIterateResult from BitmapHeapNext() --
pushing it into the table AM-specific code. So, the table AM must inform
BitmapHeapNext() whether or not the current block is lossy or exact for
the purposes of the counters used in EXPLAIN.
---
src/backend/access/heap/heapam_handler.c | 5 ++++-
src/backend/executor/nodeBitmapHeapscan.c | 10 +++++-----
src/include/access/tableam.h | 14 ++++++++++----
3 files changed, 19 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index c9b9b4c00f1..10c1c3b616b 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2112,7 +2112,8 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths,
static bool
heapam_scan_bitmap_next_block(TableScanDesc scan,
- TBMIterateResult *tbmres)
+ TBMIterateResult *tbmres,
+ bool *lossy)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
BlockNumber block = tbmres->blockno;
@@ -2240,6 +2241,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
+ *lossy = tbmres->ntuples < 0;
+
return ntup > 0;
}
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 38b4596a775..08e346ae7b3 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -211,7 +211,7 @@ BitmapHeapNext(BitmapHeapScanState *node)
for (;;)
{
- bool valid;
+ bool valid, lossy;
CHECK_FOR_INTERRUPTS();
@@ -232,12 +232,12 @@ BitmapHeapNext(BitmapHeapScanState *node)
BitmapAdjustPrefetchIterator(node, tbmres->blockno);
- valid = table_scan_bitmap_next_block(scan, tbmres);
+ valid = table_scan_bitmap_next_block(scan, tbmres, &lossy);
- if (tbmres->ntuples >= 0)
- node->exact_pages++;
- else
+ if (lossy)
node->lossy_pages++;
+ else
+ node->exact_pages++;
if (!valid)
{
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index c43a8b3dea5..f1d0d4b78e3 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -796,6 +796,9 @@ typedef struct TableAmRoutine
* on the page have to be returned, otherwise the tuples at offsets in
* `tbmres->offsets` need to be returned.
*
+ * lossy indicates whether or not the block's representation in the bitmap
+ * is lossy or exact.
+ *
* XXX: Currently this may only be implemented if the AM uses md.c as its
* storage manager, and uses ItemPointer->ip_blkid in a manner that maps
* blockids directly to the underlying storage. nodeBitmapHeapscan.c
@@ -811,7 +814,8 @@ typedef struct TableAmRoutine
* scan_bitmap_next_tuple need to exist, or neither.
*/
bool (*scan_bitmap_next_block) (TableScanDesc scan,
- struct TBMIterateResult *tbmres);
+ struct TBMIterateResult *tbmres,
+ bool *lossy);
/*
* Fetch the next tuple of a bitmap table scan into `slot` and return true
@@ -1951,14 +1955,16 @@ table_relation_estimate_size(Relation rel, int32 *attr_widths,
* Prepare to fetch / check / return tuples from `tbmres->blockno` as part of
* a bitmap table scan. `scan` needs to have been started via
* table_beginscan_bm(). Returns false if there are no tuples to be found on
- * the page, true otherwise.
+ * the page, true otherwise. lossy is set to true if bitmap is lossy for the
+ * selected block and false otherwise.
*
* Note, this is an optionally implemented function, therefore should only be
* used after verifying the presence (at plan time or such).
*/
static inline bool
table_scan_bitmap_next_block(TableScanDesc scan,
- struct TBMIterateResult *tbmres)
+ struct TBMIterateResult *tbmres,
+ bool *lossy)
{
/*
* We don't expect direct calls to table_scan_bitmap_next_block with valid
@@ -1969,7 +1975,7 @@ table_scan_bitmap_next_block(TableScanDesc scan,
elog(ERROR, "unexpected table_scan_bitmap_next_block call during logical decoding");
return scan->rs_rd->rd_tableam->scan_bitmap_next_block(scan,
- tbmres);
+ tbmres, lossy);
}
/*
--
2.40.1
--w4wcjcocxsm37usi
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0008-Reduce-scope-of-BitmapHeapScan-tbmiterator-local-.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH v8 06/17] table_scan_bitmap_next_block() returns lossy or exact
@ 2024-02-27 01:34 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Melanie Plageman @ 2024-02-27 01:34 UTC (permalink / raw)
Future commits will remove the TBMIterateResult from BitmapHeapNext() --
pushing it into the table AM-specific code. So, the table AM must inform
BitmapHeapNext() whether or not the current block is lossy or exact for
the purposes of the counters used in EXPLAIN.
---
src/backend/access/heap/heapam_handler.c | 5 ++++-
src/backend/executor/nodeBitmapHeapscan.c | 10 +++++-----
src/include/access/tableam.h | 14 ++++++++++----
3 files changed, 19 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 7fdccaf613..849cac3947 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2114,7 +2114,8 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths,
static bool
heapam_scan_bitmap_next_block(TableScanDesc scan,
- TBMIterateResult *tbmres)
+ TBMIterateResult *tbmres,
+ bool *lossy)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
BlockNumber block = tbmres->blockno;
@@ -2242,6 +2243,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
+ *lossy = tbmres->ntuples < 0;
+
return ntup > 0;
}
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 404de0595e..c95e3412da 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -211,7 +211,7 @@ BitmapHeapNext(BitmapHeapScanState *node)
for (;;)
{
- bool valid;
+ bool valid, lossy;
CHECK_FOR_INTERRUPTS();
@@ -232,12 +232,12 @@ BitmapHeapNext(BitmapHeapScanState *node)
BitmapAdjustPrefetchIterator(node, tbmres->blockno);
- valid = table_scan_bitmap_next_block(scan, tbmres);
+ valid = table_scan_bitmap_next_block(scan, tbmres, &lossy);
- if (tbmres->ntuples >= 0)
- node->exact_pages++;
- else
+ if (lossy)
node->lossy_pages++;
+ else
+ node->exact_pages++;
if (!valid)
{
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index 1bc5f7c057..b9ba4f9fb3 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -804,6 +804,9 @@ typedef struct TableAmRoutine
* on the page have to be returned, otherwise the tuples at offsets in
* `tbmres->offsets` need to be returned.
*
+ * lossy indicates whether or not the block's representation in the bitmap
+ * is lossy or exact.
+ *
* XXX: Currently this may only be implemented if the AM uses md.c as its
* storage manager, and uses ItemPointer->ip_blkid in a manner that maps
* blockids directly to the underlying storage. nodeBitmapHeapscan.c
@@ -819,7 +822,8 @@ typedef struct TableAmRoutine
* scan_bitmap_next_tuple need to exist, or neither.
*/
bool (*scan_bitmap_next_block) (TableScanDesc scan,
- struct TBMIterateResult *tbmres);
+ struct TBMIterateResult *tbmres,
+ bool *lossy);
/*
* Fetch the next tuple of a bitmap table scan into `slot` and return true
@@ -1988,14 +1992,16 @@ table_relation_estimate_size(Relation rel, int32 *attr_widths,
* Prepare to fetch / check / return tuples from `tbmres->blockno` as part of
* a bitmap table scan. `scan` needs to have been started via
* table_beginscan_bm(). Returns false if there are no tuples to be found on
- * the page, true otherwise.
+ * the page, true otherwise. lossy is set to true if bitmap is lossy for the
+ * selected block and false otherwise.
*
* Note, this is an optionally implemented function, therefore should only be
* used after verifying the presence (at plan time or such).
*/
static inline bool
table_scan_bitmap_next_block(TableScanDesc scan,
- struct TBMIterateResult *tbmres)
+ struct TBMIterateResult *tbmres,
+ bool *lossy)
{
/*
* We don't expect direct calls to table_scan_bitmap_next_block with valid
@@ -2006,7 +2012,7 @@ table_scan_bitmap_next_block(TableScanDesc scan,
elog(ERROR, "unexpected table_scan_bitmap_next_block call during logical decoding");
return scan->rs_rd->rd_tableam->scan_bitmap_next_block(scan,
- tbmres);
+ tbmres, lossy);
}
/*
--
2.40.1
--xqq4defy3uncu6k6
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v8-0007-Reduce-scope-of-BitmapHeapScan-tbmiterator-local-.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH v7 06/13] table_scan_bitmap_next_block() returns lossy or exact
@ 2024-02-27 01:34 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Melanie Plageman @ 2024-02-27 01:34 UTC (permalink / raw)
Future commits will remove the TBMIterateResult from BitmapHeapNext() --
pushing it into the table AM-specific code. So, the table AM must inform
BitmapHeapNext() whether or not the current block is lossy or exact for
the purposes of the counters used in EXPLAIN.
---
src/backend/access/heap/heapam_handler.c | 5 ++++-
src/backend/executor/nodeBitmapHeapscan.c | 10 +++++-----
src/include/access/tableam.h | 14 ++++++++++----
3 files changed, 19 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index c9b9b4c00f1..10c1c3b616b 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2112,7 +2112,8 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths,
static bool
heapam_scan_bitmap_next_block(TableScanDesc scan,
- TBMIterateResult *tbmres)
+ TBMIterateResult *tbmres,
+ bool *lossy)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
BlockNumber block = tbmres->blockno;
@@ -2240,6 +2241,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
+ *lossy = tbmres->ntuples < 0;
+
return ntup > 0;
}
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index a6b98fa12a1..b2397fe2054 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -211,7 +211,7 @@ BitmapHeapNext(BitmapHeapScanState *node)
for (;;)
{
- bool valid;
+ bool valid, lossy;
CHECK_FOR_INTERRUPTS();
@@ -232,12 +232,12 @@ BitmapHeapNext(BitmapHeapScanState *node)
BitmapAdjustPrefetchIterator(node, tbmres->blockno);
- valid = table_scan_bitmap_next_block(scan, tbmres);
+ valid = table_scan_bitmap_next_block(scan, tbmres, &lossy);
- if (tbmres->ntuples >= 0)
- node->exact_pages++;
- else
+ if (lossy)
node->lossy_pages++;
+ else
+ node->exact_pages++;
if (!valid)
{
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index c43a8b3dea5..f1d0d4b78e3 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -796,6 +796,9 @@ typedef struct TableAmRoutine
* on the page have to be returned, otherwise the tuples at offsets in
* `tbmres->offsets` need to be returned.
*
+ * lossy indicates whether or not the block's representation in the bitmap
+ * is lossy or exact.
+ *
* XXX: Currently this may only be implemented if the AM uses md.c as its
* storage manager, and uses ItemPointer->ip_blkid in a manner that maps
* blockids directly to the underlying storage. nodeBitmapHeapscan.c
@@ -811,7 +814,8 @@ typedef struct TableAmRoutine
* scan_bitmap_next_tuple need to exist, or neither.
*/
bool (*scan_bitmap_next_block) (TableScanDesc scan,
- struct TBMIterateResult *tbmres);
+ struct TBMIterateResult *tbmres,
+ bool *lossy);
/*
* Fetch the next tuple of a bitmap table scan into `slot` and return true
@@ -1951,14 +1955,16 @@ table_relation_estimate_size(Relation rel, int32 *attr_widths,
* Prepare to fetch / check / return tuples from `tbmres->blockno` as part of
* a bitmap table scan. `scan` needs to have been started via
* table_beginscan_bm(). Returns false if there are no tuples to be found on
- * the page, true otherwise.
+ * the page, true otherwise. lossy is set to true if bitmap is lossy for the
+ * selected block and false otherwise.
*
* Note, this is an optionally implemented function, therefore should only be
* used after verifying the presence (at plan time or such).
*/
static inline bool
table_scan_bitmap_next_block(TableScanDesc scan,
- struct TBMIterateResult *tbmres)
+ struct TBMIterateResult *tbmres,
+ bool *lossy)
{
/*
* We don't expect direct calls to table_scan_bitmap_next_block with valid
@@ -1969,7 +1975,7 @@ table_scan_bitmap_next_block(TableScanDesc scan,
elog(ERROR, "unexpected table_scan_bitmap_next_block call during logical decoding");
return scan->rs_rd->rd_tableam->scan_bitmap_next_block(scan,
- tbmres);
+ tbmres, lossy);
}
/*
--
2.40.1
--kqqpqghcwbcc3dt5
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0007-Reduce-scope-of-BitmapHeapScan-tbmiterator-local-.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH v9 06/17] table_scan_bitmap_next_block() returns lossy or exact
@ 2024-02-27 01:34 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Melanie Plageman @ 2024-02-27 01:34 UTC (permalink / raw)
Future commits will remove the TBMIterateResult from BitmapHeapNext() --
pushing it into the table AM-specific code. So, the table AM must inform
BitmapHeapNext() whether or not the current block is lossy or exact for
the purposes of the counters used in EXPLAIN.
---
src/backend/access/heap/heapam_handler.c | 5 ++++-
src/backend/executor/nodeBitmapHeapscan.c | 10 +++++-----
src/include/access/tableam.h | 14 ++++++++++----
3 files changed, 19 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 7fdccaf613..849cac3947 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2114,7 +2114,8 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths,
static bool
heapam_scan_bitmap_next_block(TableScanDesc scan,
- TBMIterateResult *tbmres)
+ TBMIterateResult *tbmres,
+ bool *lossy)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
BlockNumber block = tbmres->blockno;
@@ -2242,6 +2243,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
+ *lossy = tbmres->ntuples < 0;
+
return ntup > 0;
}
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 404de0595e..c95e3412da 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -211,7 +211,7 @@ BitmapHeapNext(BitmapHeapScanState *node)
for (;;)
{
- bool valid;
+ bool valid, lossy;
CHECK_FOR_INTERRUPTS();
@@ -232,12 +232,12 @@ BitmapHeapNext(BitmapHeapScanState *node)
BitmapAdjustPrefetchIterator(node, tbmres->blockno);
- valid = table_scan_bitmap_next_block(scan, tbmres);
+ valid = table_scan_bitmap_next_block(scan, tbmres, &lossy);
- if (tbmres->ntuples >= 0)
- node->exact_pages++;
- else
+ if (lossy)
node->lossy_pages++;
+ else
+ node->exact_pages++;
if (!valid)
{
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index 1bc5f7c057..b9ba4f9fb3 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -804,6 +804,9 @@ typedef struct TableAmRoutine
* on the page have to be returned, otherwise the tuples at offsets in
* `tbmres->offsets` need to be returned.
*
+ * lossy indicates whether or not the block's representation in the bitmap
+ * is lossy or exact.
+ *
* XXX: Currently this may only be implemented if the AM uses md.c as its
* storage manager, and uses ItemPointer->ip_blkid in a manner that maps
* blockids directly to the underlying storage. nodeBitmapHeapscan.c
@@ -819,7 +822,8 @@ typedef struct TableAmRoutine
* scan_bitmap_next_tuple need to exist, or neither.
*/
bool (*scan_bitmap_next_block) (TableScanDesc scan,
- struct TBMIterateResult *tbmres);
+ struct TBMIterateResult *tbmres,
+ bool *lossy);
/*
* Fetch the next tuple of a bitmap table scan into `slot` and return true
@@ -1988,14 +1992,16 @@ table_relation_estimate_size(Relation rel, int32 *attr_widths,
* Prepare to fetch / check / return tuples from `tbmres->blockno` as part of
* a bitmap table scan. `scan` needs to have been started via
* table_beginscan_bm(). Returns false if there are no tuples to be found on
- * the page, true otherwise.
+ * the page, true otherwise. lossy is set to true if bitmap is lossy for the
+ * selected block and false otherwise.
*
* Note, this is an optionally implemented function, therefore should only be
* used after verifying the presence (at plan time or such).
*/
static inline bool
table_scan_bitmap_next_block(TableScanDesc scan,
- struct TBMIterateResult *tbmres)
+ struct TBMIterateResult *tbmres,
+ bool *lossy)
{
/*
* We don't expect direct calls to table_scan_bitmap_next_block with valid
@@ -2006,7 +2012,7 @@ table_scan_bitmap_next_block(TableScanDesc scan,
elog(ERROR, "unexpected table_scan_bitmap_next_block call during logical decoding");
return scan->rs_rd->rd_tableam->scan_bitmap_next_block(scan,
- tbmres);
+ tbmres, lossy);
}
/*
--
2.40.1
--7mdtsjmrzitrgzgx
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v9-0007-Reduce-scope-of-BitmapHeapScan-tbmiterator-local-.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH v7 06/13] table_scan_bitmap_next_block() returns lossy or exact
@ 2024-02-27 01:34 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Melanie Plageman @ 2024-02-27 01:34 UTC (permalink / raw)
Future commits will remove the TBMIterateResult from BitmapHeapNext() --
pushing it into the table AM-specific code. So, the table AM must inform
BitmapHeapNext() whether or not the current block is lossy or exact for
the purposes of the counters used in EXPLAIN.
---
src/backend/access/heap/heapam_handler.c | 5 ++++-
src/backend/executor/nodeBitmapHeapscan.c | 10 +++++-----
src/include/access/tableam.h | 14 ++++++++++----
3 files changed, 19 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index c9b9b4c00f1..10c1c3b616b 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2112,7 +2112,8 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths,
static bool
heapam_scan_bitmap_next_block(TableScanDesc scan,
- TBMIterateResult *tbmres)
+ TBMIterateResult *tbmres,
+ bool *lossy)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
BlockNumber block = tbmres->blockno;
@@ -2240,6 +2241,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
+ *lossy = tbmres->ntuples < 0;
+
return ntup > 0;
}
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index a6b98fa12a1..b2397fe2054 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -211,7 +211,7 @@ BitmapHeapNext(BitmapHeapScanState *node)
for (;;)
{
- bool valid;
+ bool valid, lossy;
CHECK_FOR_INTERRUPTS();
@@ -232,12 +232,12 @@ BitmapHeapNext(BitmapHeapScanState *node)
BitmapAdjustPrefetchIterator(node, tbmres->blockno);
- valid = table_scan_bitmap_next_block(scan, tbmres);
+ valid = table_scan_bitmap_next_block(scan, tbmres, &lossy);
- if (tbmres->ntuples >= 0)
- node->exact_pages++;
- else
+ if (lossy)
node->lossy_pages++;
+ else
+ node->exact_pages++;
if (!valid)
{
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index c43a8b3dea5..f1d0d4b78e3 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -796,6 +796,9 @@ typedef struct TableAmRoutine
* on the page have to be returned, otherwise the tuples at offsets in
* `tbmres->offsets` need to be returned.
*
+ * lossy indicates whether or not the block's representation in the bitmap
+ * is lossy or exact.
+ *
* XXX: Currently this may only be implemented if the AM uses md.c as its
* storage manager, and uses ItemPointer->ip_blkid in a manner that maps
* blockids directly to the underlying storage. nodeBitmapHeapscan.c
@@ -811,7 +814,8 @@ typedef struct TableAmRoutine
* scan_bitmap_next_tuple need to exist, or neither.
*/
bool (*scan_bitmap_next_block) (TableScanDesc scan,
- struct TBMIterateResult *tbmres);
+ struct TBMIterateResult *tbmres,
+ bool *lossy);
/*
* Fetch the next tuple of a bitmap table scan into `slot` and return true
@@ -1951,14 +1955,16 @@ table_relation_estimate_size(Relation rel, int32 *attr_widths,
* Prepare to fetch / check / return tuples from `tbmres->blockno` as part of
* a bitmap table scan. `scan` needs to have been started via
* table_beginscan_bm(). Returns false if there are no tuples to be found on
- * the page, true otherwise.
+ * the page, true otherwise. lossy is set to true if bitmap is lossy for the
+ * selected block and false otherwise.
*
* Note, this is an optionally implemented function, therefore should only be
* used after verifying the presence (at plan time or such).
*/
static inline bool
table_scan_bitmap_next_block(TableScanDesc scan,
- struct TBMIterateResult *tbmres)
+ struct TBMIterateResult *tbmres,
+ bool *lossy)
{
/*
* We don't expect direct calls to table_scan_bitmap_next_block with valid
@@ -1969,7 +1975,7 @@ table_scan_bitmap_next_block(TableScanDesc scan,
elog(ERROR, "unexpected table_scan_bitmap_next_block call during logical decoding");
return scan->rs_rd->rd_tableam->scan_bitmap_next_block(scan,
- tbmres);
+ tbmres, lossy);
}
/*
--
2.40.1
--kqqpqghcwbcc3dt5
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0007-Reduce-scope-of-BitmapHeapScan-tbmiterator-local-.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH v8 06/17] table_scan_bitmap_next_block() returns lossy or exact
@ 2024-02-27 01:34 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Melanie Plageman @ 2024-02-27 01:34 UTC (permalink / raw)
Future commits will remove the TBMIterateResult from BitmapHeapNext() --
pushing it into the table AM-specific code. So, the table AM must inform
BitmapHeapNext() whether or not the current block is lossy or exact for
the purposes of the counters used in EXPLAIN.
---
src/backend/access/heap/heapam_handler.c | 5 ++++-
src/backend/executor/nodeBitmapHeapscan.c | 10 +++++-----
src/include/access/tableam.h | 14 ++++++++++----
3 files changed, 19 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 7fdccaf613..849cac3947 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2114,7 +2114,8 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths,
static bool
heapam_scan_bitmap_next_block(TableScanDesc scan,
- TBMIterateResult *tbmres)
+ TBMIterateResult *tbmres,
+ bool *lossy)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
BlockNumber block = tbmres->blockno;
@@ -2242,6 +2243,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
+ *lossy = tbmres->ntuples < 0;
+
return ntup > 0;
}
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 404de0595e..c95e3412da 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -211,7 +211,7 @@ BitmapHeapNext(BitmapHeapScanState *node)
for (;;)
{
- bool valid;
+ bool valid, lossy;
CHECK_FOR_INTERRUPTS();
@@ -232,12 +232,12 @@ BitmapHeapNext(BitmapHeapScanState *node)
BitmapAdjustPrefetchIterator(node, tbmres->blockno);
- valid = table_scan_bitmap_next_block(scan, tbmres);
+ valid = table_scan_bitmap_next_block(scan, tbmres, &lossy);
- if (tbmres->ntuples >= 0)
- node->exact_pages++;
- else
+ if (lossy)
node->lossy_pages++;
+ else
+ node->exact_pages++;
if (!valid)
{
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index 1bc5f7c057..b9ba4f9fb3 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -804,6 +804,9 @@ typedef struct TableAmRoutine
* on the page have to be returned, otherwise the tuples at offsets in
* `tbmres->offsets` need to be returned.
*
+ * lossy indicates whether or not the block's representation in the bitmap
+ * is lossy or exact.
+ *
* XXX: Currently this may only be implemented if the AM uses md.c as its
* storage manager, and uses ItemPointer->ip_blkid in a manner that maps
* blockids directly to the underlying storage. nodeBitmapHeapscan.c
@@ -819,7 +822,8 @@ typedef struct TableAmRoutine
* scan_bitmap_next_tuple need to exist, or neither.
*/
bool (*scan_bitmap_next_block) (TableScanDesc scan,
- struct TBMIterateResult *tbmres);
+ struct TBMIterateResult *tbmres,
+ bool *lossy);
/*
* Fetch the next tuple of a bitmap table scan into `slot` and return true
@@ -1988,14 +1992,16 @@ table_relation_estimate_size(Relation rel, int32 *attr_widths,
* Prepare to fetch / check / return tuples from `tbmres->blockno` as part of
* a bitmap table scan. `scan` needs to have been started via
* table_beginscan_bm(). Returns false if there are no tuples to be found on
- * the page, true otherwise.
+ * the page, true otherwise. lossy is set to true if bitmap is lossy for the
+ * selected block and false otherwise.
*
* Note, this is an optionally implemented function, therefore should only be
* used after verifying the presence (at plan time or such).
*/
static inline bool
table_scan_bitmap_next_block(TableScanDesc scan,
- struct TBMIterateResult *tbmres)
+ struct TBMIterateResult *tbmres,
+ bool *lossy)
{
/*
* We don't expect direct calls to table_scan_bitmap_next_block with valid
@@ -2006,7 +2012,7 @@ table_scan_bitmap_next_block(TableScanDesc scan,
elog(ERROR, "unexpected table_scan_bitmap_next_block call during logical decoding");
return scan->rs_rd->rd_tableam->scan_bitmap_next_block(scan,
- tbmres);
+ tbmres, lossy);
}
/*
--
2.40.1
--xqq4defy3uncu6k6
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v8-0007-Reduce-scope-of-BitmapHeapScan-tbmiterator-local-.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH v11 06/17] table_scan_bitmap_next_block() returns lossy or exact
@ 2024-02-27 01:34 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Melanie Plageman @ 2024-02-27 01:34 UTC (permalink / raw)
Future commits will remove the TBMIterateResult from BitmapHeapNext() --
pushing it into the table AM-specific code. So, the table AM must inform
BitmapHeapNext() whether or not the current block is lossy or exact for
the purposes of the counters used in EXPLAIN.
---
src/backend/access/heap/heapam_handler.c | 5 ++++-
src/backend/executor/nodeBitmapHeapscan.c | 10 +++++-----
src/include/access/tableam.h | 14 ++++++++++----
3 files changed, 19 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 5ba8cb3657..ce47a158ae 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2172,7 +2172,8 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths,
static bool
heapam_scan_bitmap_next_block(TableScanDesc scan,
- TBMIterateResult *tbmres)
+ TBMIterateResult *tbmres,
+ bool *lossy)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
BlockNumber block = tbmres->blockno;
@@ -2300,6 +2301,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
+ *lossy = tbmres->ntuples < 0;
+
return ntup > 0;
}
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 404de0595e..c95e3412da 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -211,7 +211,7 @@ BitmapHeapNext(BitmapHeapScanState *node)
for (;;)
{
- bool valid;
+ bool valid, lossy;
CHECK_FOR_INTERRUPTS();
@@ -232,12 +232,12 @@ BitmapHeapNext(BitmapHeapScanState *node)
BitmapAdjustPrefetchIterator(node, tbmres->blockno);
- valid = table_scan_bitmap_next_block(scan, tbmres);
+ valid = table_scan_bitmap_next_block(scan, tbmres, &lossy);
- if (tbmres->ntuples >= 0)
- node->exact_pages++;
- else
+ if (lossy)
node->lossy_pages++;
+ else
+ node->exact_pages++;
if (!valid)
{
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index ae758857bd..5c2a7b7422 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -815,6 +815,9 @@ typedef struct TableAmRoutine
* on the page have to be returned, otherwise the tuples at offsets in
* `tbmres->offsets` need to be returned.
*
+ * lossy indicates whether or not the block's representation in the bitmap
+ * is lossy or exact.
+ *
* XXX: Currently this may only be implemented if the AM uses md.c as its
* storage manager, and uses ItemPointer->ip_blkid in a manner that maps
* blockids directly to the underlying storage. nodeBitmapHeapscan.c
@@ -830,7 +833,8 @@ typedef struct TableAmRoutine
* scan_bitmap_next_tuple need to exist, or neither.
*/
bool (*scan_bitmap_next_block) (TableScanDesc scan,
- struct TBMIterateResult *tbmres);
+ struct TBMIterateResult *tbmres,
+ bool *lossy);
/*
* Fetch the next tuple of a bitmap table scan into `slot` and return true
@@ -2023,14 +2027,16 @@ table_relation_estimate_size(Relation rel, int32 *attr_widths,
* Prepare to fetch / check / return tuples from `tbmres->blockno` as part of
* a bitmap table scan. `scan` needs to have been started via
* table_beginscan_bm(). Returns false if there are no tuples to be found on
- * the page, true otherwise.
+ * the page, true otherwise. lossy is set to true if bitmap is lossy for the
+ * selected block and false otherwise.
*
* Note, this is an optionally implemented function, therefore should only be
* used after verifying the presence (at plan time or such).
*/
static inline bool
table_scan_bitmap_next_block(TableScanDesc scan,
- struct TBMIterateResult *tbmres)
+ struct TBMIterateResult *tbmres,
+ bool *lossy)
{
/*
* We don't expect direct calls to table_scan_bitmap_next_block with valid
@@ -2041,7 +2047,7 @@ table_scan_bitmap_next_block(TableScanDesc scan,
elog(ERROR, "unexpected table_scan_bitmap_next_block call during logical decoding");
return scan->rs_rd->rd_tableam->scan_bitmap_next_block(scan,
- tbmres);
+ tbmres, lossy);
}
/*
--
2.40.1
--owzzsiozz6hgpp7e
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v11-0007-Reduce-scope-of-BitmapHeapScan-tbmiterator-local.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* [PATCH v8 06/17] table_scan_bitmap_next_block() returns lossy or exact
@ 2024-02-27 01:34 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Melanie Plageman @ 2024-02-27 01:34 UTC (permalink / raw)
Future commits will remove the TBMIterateResult from BitmapHeapNext() --
pushing it into the table AM-specific code. So, the table AM must inform
BitmapHeapNext() whether or not the current block is lossy or exact for
the purposes of the counters used in EXPLAIN.
---
src/backend/access/heap/heapam_handler.c | 5 ++++-
src/backend/executor/nodeBitmapHeapscan.c | 10 +++++-----
src/include/access/tableam.h | 14 ++++++++++----
3 files changed, 19 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 7fdccaf613..849cac3947 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2114,7 +2114,8 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths,
static bool
heapam_scan_bitmap_next_block(TableScanDesc scan,
- TBMIterateResult *tbmres)
+ TBMIterateResult *tbmres,
+ bool *lossy)
{
HeapScanDesc hscan = (HeapScanDesc) scan;
BlockNumber block = tbmres->blockno;
@@ -2242,6 +2243,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan,
Assert(ntup <= MaxHeapTuplesPerPage);
hscan->rs_ntuples = ntup;
+ *lossy = tbmres->ntuples < 0;
+
return ntup > 0;
}
diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c
index 404de0595e..c95e3412da 100644
--- a/src/backend/executor/nodeBitmapHeapscan.c
+++ b/src/backend/executor/nodeBitmapHeapscan.c
@@ -211,7 +211,7 @@ BitmapHeapNext(BitmapHeapScanState *node)
for (;;)
{
- bool valid;
+ bool valid, lossy;
CHECK_FOR_INTERRUPTS();
@@ -232,12 +232,12 @@ BitmapHeapNext(BitmapHeapScanState *node)
BitmapAdjustPrefetchIterator(node, tbmres->blockno);
- valid = table_scan_bitmap_next_block(scan, tbmres);
+ valid = table_scan_bitmap_next_block(scan, tbmres, &lossy);
- if (tbmres->ntuples >= 0)
- node->exact_pages++;
- else
+ if (lossy)
node->lossy_pages++;
+ else
+ node->exact_pages++;
if (!valid)
{
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index 1bc5f7c057..b9ba4f9fb3 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -804,6 +804,9 @@ typedef struct TableAmRoutine
* on the page have to be returned, otherwise the tuples at offsets in
* `tbmres->offsets` need to be returned.
*
+ * lossy indicates whether or not the block's representation in the bitmap
+ * is lossy or exact.
+ *
* XXX: Currently this may only be implemented if the AM uses md.c as its
* storage manager, and uses ItemPointer->ip_blkid in a manner that maps
* blockids directly to the underlying storage. nodeBitmapHeapscan.c
@@ -819,7 +822,8 @@ typedef struct TableAmRoutine
* scan_bitmap_next_tuple need to exist, or neither.
*/
bool (*scan_bitmap_next_block) (TableScanDesc scan,
- struct TBMIterateResult *tbmres);
+ struct TBMIterateResult *tbmres,
+ bool *lossy);
/*
* Fetch the next tuple of a bitmap table scan into `slot` and return true
@@ -1988,14 +1992,16 @@ table_relation_estimate_size(Relation rel, int32 *attr_widths,
* Prepare to fetch / check / return tuples from `tbmres->blockno` as part of
* a bitmap table scan. `scan` needs to have been started via
* table_beginscan_bm(). Returns false if there are no tuples to be found on
- * the page, true otherwise.
+ * the page, true otherwise. lossy is set to true if bitmap is lossy for the
+ * selected block and false otherwise.
*
* Note, this is an optionally implemented function, therefore should only be
* used after verifying the presence (at plan time or such).
*/
static inline bool
table_scan_bitmap_next_block(TableScanDesc scan,
- struct TBMIterateResult *tbmres)
+ struct TBMIterateResult *tbmres,
+ bool *lossy)
{
/*
* We don't expect direct calls to table_scan_bitmap_next_block with valid
@@ -2006,7 +2012,7 @@ table_scan_bitmap_next_block(TableScanDesc scan,
elog(ERROR, "unexpected table_scan_bitmap_next_block call during logical decoding");
return scan->rs_rd->rd_tableam->scan_bitmap_next_block(scan,
- tbmres);
+ tbmres, lossy);
}
/*
--
2.40.1
--xqq4defy3uncu6k6
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v8-0007-Reduce-scope-of-BitmapHeapScan-tbmiterator-local-.patch"
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Conflict detection and logging in logical replication
@ 2024-07-11 05:03 shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
0 siblings, 1 reply; 50+ messages in thread
From: shveta malik @ 2024-07-11 05:03 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Nisha Moond <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Amit Kapila <[email protected]>; Ashutosh Bapat <[email protected]>; shveta malik <[email protected]>
On Thu, Jul 11, 2024 at 7:47 AM Zhijie Hou (Fujitsu)
<[email protected]> wrote:
>
> On Wednesday, July 10, 2024 5:39 PM shveta malik <[email protected]> wrote:
> >
> > On Wed, Jul 3, 2024 at 8:31 AM Zhijie Hou (Fujitsu) <[email protected]>
> > wrote:
> > >
> > > On Wednesday, June 26, 2024 10:58 AM Zhijie Hou (Fujitsu)
> > <[email protected]> wrote:
> > > >
> > >
> > > Hi,
> > >
> > > As suggested by Sawada-san in another thread[1].
> > >
> > > I am attaching the V4 patch set which tracks the delete_differ
> > > conflict in logical replication.
> > >
> > > delete_differ means that the replicated DELETE is deleting a row that
> > > was modified by a different origin.
> > >
> >
> > Thanks for the patch. I am still in process of review but please find few
> > comments:
>
> Thanks for the comments!
>
> > 1) When I try to *insert* primary/unique key on pub, which already exists on
> > sub, conflict gets detected. But when I try to *update* primary/unique key to a
> > value on pub which already exists on sub, conflict is not detected. I get the
> > error:
> >
> > 2024-07-10 14:21:09.976 IST [647678] ERROR: duplicate key value violates
> > unique constraint "t1_pkey"
> > 2024-07-10 14:21:09.976 IST [647678] DETAIL: Key (pk)=(4) already exists.
>
> Yes, I think the detection of this conflict is not added with the
> intention to control the size of the patch in the first version.
>
> >
> > This is because such conflict detection needs detection of constraint violation
> > using the *new value* rather than *existing* value during UPDATE. INSERT
> > conflict detection takes care of this case i.e. the columns of incoming row are
> > considered as new values and it tries to see if all unique indexes are okay to
> > digest such new values (all incoming columns) but update's logic is different.
> > It searches based on oldTuple *only* and thus above detection is missing.
>
> I think the logic is the same if we want to detect the unique violation
> for UDPATE, we need to check if the new value of the UPDATE violates any
> unique constraints same as the detection of insert_exists (e.g. check
> the conflict around ExecInsertIndexTuples())
>
> >
> > Shall we support such detection? If not, is it worth docuementing?
>
> I am personally OK to support this detection.
+1. I think it should not be a complex or too big change.
> And
> I think it's already documented that we only detect unique violation for
> insert which mean update conflict is not detected.
>
> > 2)
> > Another case which might confuse user:
> >
> > CREATE TABLE t1 (pk integer primary key, val1 integer, val2 integer);
> >
> > On PUB: insert into t1 values(1,10,10); insert into t1 values(2,20,20);
> >
> > On SUB: update t1 set pk=3 where pk=2;
> >
> > Data on PUB: {1,10,10}, {2,20,20}
> > Data on SUB: {1,10,10}, {3,20,20}
> >
> > Now on PUB: update t1 set val1=200 where val1=20;
> >
> > On Sub, I get this:
> > 2024-07-10 14:44:00.160 IST [648287] LOG: conflict update_missing detected
> > on relation "public.t1"
> > 2024-07-10 14:44:00.160 IST [648287] DETAIL: Did not find the row to be
> > updated.
> > 2024-07-10 14:44:00.160 IST [648287] CONTEXT: processing remote data for
> > replication origin "pg_16389" during message type "UPDATE" for replication
> > target relation "public.t1" in transaction 760, finished at 0/156D658
> >
> > To user, it could be quite confusing, as val1=20 exists on sub but still he gets
> > update_missing conflict and the 'DETAIL' is not sufficient to give the clarity. I
> > think on HEAD as well (have not tested), we will get same behavior i.e. update
> > will be ignored as we make search based on RI (pk in this case). So we are not
> > worsening the situation, but now since we are detecting conflict, is it possible
> > to give better details in 'DETAIL' section indicating what is actually missing?
>
> I think It's doable to report the row value that cannot be found in the local
> relation, but the concern is the potential risk of exposing some
> sensitive data in the log. This may be OK, as we are already reporting the
> key value for constraints violation, so if others also agree, we can add
> the row value in the DETAIL as well.
Okay, let's see what others say. JFYI, the same situation holds valid
for delete_missing case.
I have one concern about how we deal with conflicts. As for
insert_exists, we keep on erroring out while raising conflict, until
it is manually resolved:
ERROR: conflict insert_exists detected
But for other cases, we just log conflict and either skip or apply the
operation. I
LOG: conflict update_differ detected
DETAIL: Updating a row that was modified by a different origin
I know that it is no different than HEAD. But now since we are logging
conflicts explicitly, we should call out default behavior on each
conflict. I see some incomplete and scattered info in '31.5.
Conflicts' section saying that:
"When replicating UPDATE or DELETE operations, missing data will not
produce a conflict and such operations will simply be skipped."
(lets say it as pt a)
Also some more info in a later section saying (pt b):
:A conflict will produce an error and will stop the replication; it
must be resolved manually by the user."
My suggestions:
1) in point a above, shall we have:
missing data or differing data (i.e. somehow reword to accommodate
update_differ and delete_differ cases)
2) Now since we have a section explaining conflicts detected and
logged with detect_conflict=true, shall we mention default behaviour
with each?
insert_exists: error will be raised until resolved manually.
update_differ: update will be applied
update_missing: update will be skipped
delete_missing: delete will be skipped
delete_differ: delete will be applied.
thanks
Shveta
^ permalink raw reply [nested|flat] 50+ messages in thread
* RE: Conflict detection and logging in logical replication
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
@ 2024-07-18 02:22 ` Zhijie Hou (Fujitsu) <[email protected]>
2024-07-18 05:50 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-19 08:36 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-24 04:50 ` Re: Conflict detection and logging in logical replication Nisha Moond <[email protected]>
0 siblings, 3 replies; 50+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-07-18 02:22 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Nisha Moond <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Amit Kapila <[email protected]>; Ashutosh Bapat <[email protected]>
On Thursday, July 11, 2024 1:03 PM shveta malik <[email protected]> wrote:
Hi,
Thanks for the comments!
>
> I have one concern about how we deal with conflicts. As for insert_exists, we
> keep on erroring out while raising conflict, until it is manually resolved:
> ERROR: conflict insert_exists detected
>
> But for other cases, we just log conflict and either skip or apply the operation. I
> LOG: conflict update_differ detected
> DETAIL: Updating a row that was modified by a different origin
>
> I know that it is no different than HEAD. But now since we are logging conflicts
> explicitly, we should call out default behavior on each conflict. I see some
> incomplete and scattered info in '31.5.
> Conflicts' section saying that:
> "When replicating UPDATE or DELETE operations, missing data will not
> produce a conflict and such operations will simply be skipped."
> (lets say it as pt a)
>
> Also some more info in a later section saying (pt b):
> :A conflict will produce an error and will stop the replication; it must be resolved
> manually by the user."
>
> My suggestions:
> 1) in point a above, shall we have:
> missing data or differing data (i.e. somehow reword to accommodate
> update_differ and delete_differ cases)
I am not sure if rewording existing words is better. I feel adding a link to
let user refer to the detect_conflict section for the all the
conflicts is sufficient, so did like that.
>
> 2)
> monitoring.sgml: Below are my suggestions, please change if you feel apt.
>
> 2a) insert_exists_count : Number of times inserting a row that violates a NOT
> DEFERRABLE unique constraint while applying changes. Suggestion: Number of
> times a row insertion violated a NOT DEFERRABLE unique constraint while
> applying changes.
>
> 2b) update_differ_count : Number of times updating a row that was previously
> modified by another source while applying changes. Suggestion: Number of times
> update was performed on a row that was previously modified by another source
> while applying changes.
>
> 2c) delete_differ_count: Number of times deleting a row that was previously
> modified by another source while applying changes. Suggestion: Number of times
> delete was performed on a row that was previously modified by another source
> while applying changes.
I am a bit unsure which one is better, so I didn't change in this version.
>
> 5)
> conflict.h:CONFLICT_NUM_TYPES
>
> --Shall the macro be CONFLICT_TYPES_NUM instead?
I think the current style followed existing ones(e.g. IOOP_NUM_TYPES,
BACKEND_NUM_TYPES, IOOBJECT_NUM_TYPES ...), so I didn't change this.
Attach the V5 patch set which changed the following:
1. addressed shveta's comments which are not mentioned above.
2. support update_exists conflict which indicates
that the updated value of a row violates the unique constraint.
Best Regards,
Hou zj
Attachments:
[application/octet-stream] v5-0002-Collect-statistics-about-conflicts-in-logical-rep.patch (23.5K, ../../OS0PR01MB57166C2566E00676649CF48B94AC2@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v5-0002-Collect-statistics-about-conflicts-in-logical-rep.patch)
download | inline diff:
From 4485d3ac013dcd38ed98bc9afb93b1b4e7ae54c4 Mon Sep 17 00:00:00 2001
From: Hou Zhijie <[email protected]>
Date: Wed, 3 Jul 2024 10:34:10 +0800
Subject: [PATCH v5 2/2] Collect statistics about conflicts in logical
replication
This commit adds columns in view pg_stat_subscription_stats to show
information about the conflict which occur during the application of
logical replication changes. Currently, the following columns are added.
insert_exists_counts:
Number of times inserting a row that iolates a NOT DEFERRABLE unique constraint.
insert_exists_counts:
Number of times that the updated value of a row violates a NOT DEFERRABLE unique constraint.
update_differ_counts:
Number of times updating a row that was previously modified by another origin.
update_missing_counts:
Number of times that the tuple to be updated is missing.
delete_missing_counts:
Number of times that the tuple to be deleted is missing.
delete_differ_counts:
Number of times deleting a row that was previously modified by another origin.
The conflicts will be tracked only when detect_conflict option of the
subscription is enabled. Additionally, update_differ and delete_differ
can be detected only when track_commit_timestamp is enabled.
---
doc/src/sgml/monitoring.sgml | 78 +++++++++++-
doc/src/sgml/ref/create_subscription.sgml | 5 +-
src/backend/catalog/system_views.sql | 6 +
src/backend/replication/logical/conflict.c | 4 +
.../utils/activity/pgstat_subscription.c | 17 +++
src/backend/utils/adt/pgstatfuncs.c | 35 ++++-
src/include/catalog/pg_proc.dat | 6 +-
src/include/pgstat.h | 4 +
src/include/replication/conflict.h | 7 +
src/test/regress/expected/rules.out | 8 +-
src/test/subscription/t/026_stats.pl | 120 +++++++++++++++++-
11 files changed, 269 insertions(+), 21 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 55417a6fa9..abe28068a8 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -507,7 +507,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<row>
<entry><structname>pg_stat_subscription_stats</structname><indexterm><primary>pg_stat_subscription_stats</primary></indexterm></entry>
- <entry>One row per subscription, showing statistics about errors.
+ <entry>One row per subscription, showing statistics about errors and conflicts.
See <link linkend="monitoring-pg-stat-subscription-stats">
<structname>pg_stat_subscription_stats</structname></link> for details.
</entry>
@@ -2171,6 +2171,82 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>insert_exists_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times inserting a row that violates a
+ <literal>NOT DEFERRABLE</literal> unique constraint while applying
+ changes. This conflict is counted only if
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ is enabled
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>update_exists_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times that the updated value of a row violates a
+ <literal>NOT DEFERRABLE</literal> unique constraint while applying
+ changes. This conflict is counted only if
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ is enabled
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>update_differ_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times updating a row that was previously modified by another
+ source while applying changes. This conflict is counted only when the
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ option of the subscription and <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ are enabled
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>update_missing_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times that the tuple to be updated was not found while applying
+ changes. This conflict is counted only if
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ is enabled
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>delete_missing_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times that the tuple to be deleted was not found while applying
+ changes. This conflict is counted only if
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ is enabled
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>delete_differ_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times deleting a row that was previously modified by another
+ source while applying changes. This conflict is counted only when the
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ option of the subscription and <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ are enabled
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>stats_reset</structfield> <type>timestamp with time zone</type>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index eb51805e81..04f3ba6e9a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -437,8 +437,9 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
The default is <literal>false</literal>.
</para>
<para>
- When conflict detection is enabled, additional logging is triggered
- in the following scenarios:
+ When conflict detection is enabled, additional logging is triggered and
+ the conflict statistics are collected(displayed in the
+ <link linkend="monitoring-pg-stat-subscription-stats"><structname>pg_stat_subscription_stats</structname></link> view) in the following scenarios:
<variablelist>
<varlistentry>
<term><literal>insert_exists</literal></term>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index d084bfc48a..5244d8e356 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1366,6 +1366,12 @@ CREATE VIEW pg_stat_subscription_stats AS
s.subname,
ss.apply_error_count,
ss.sync_error_count,
+ ss.insert_exists_count,
+ ss.update_exists_count,
+ ss.update_differ_count,
+ ss.update_missing_count,
+ ss.delete_missing_count,
+ ss.delete_differ_count,
ss.stats_reset
FROM pg_subscription as s,
pg_stat_get_subscription_stats(s.oid) as ss;
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index b7dc73cfce..4873639dbf 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -15,8 +15,10 @@
#include "postgres.h"
#include "access/commit_ts.h"
+#include "pgstat.h"
#include "replication/conflict.h"
#include "replication/origin.h"
+#include "replication/worker_internal.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
@@ -78,6 +80,8 @@ ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
RepOriginId localorigin, TimestampTz localts,
TupleTableSlot *conflictslot)
{
+ pgstat_report_subscription_conflict(MySubscription->oid, type);
+
ereport(elevel,
errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
errmsg("conflict %s detected on relation \"%s.%s\"",
diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c
index d9af8de658..e06c92727e 100644
--- a/src/backend/utils/activity/pgstat_subscription.c
+++ b/src/backend/utils/activity/pgstat_subscription.c
@@ -39,6 +39,21 @@ pgstat_report_subscription_error(Oid subid, bool is_apply_error)
pending->sync_error_count++;
}
+/*
+ * Report a subscription conflict.
+ */
+void
+pgstat_report_subscription_conflict(Oid subid, ConflictType type)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStat_BackendSubEntry *pending;
+
+ entry_ref = pgstat_prep_pending_entry(PGSTAT_KIND_SUBSCRIPTION,
+ InvalidOid, subid, NULL);
+ pending = entry_ref->pending;
+ pending->conflict_count[type]++;
+}
+
/*
* Report creating the subscription.
*/
@@ -101,6 +116,8 @@ pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
#define SUB_ACC(fld) shsubent->stats.fld += localent->fld
SUB_ACC(apply_error_count);
SUB_ACC(sync_error_count);
+ for (int i = 0; i < CONFLICT_NUM_TYPES; i++)
+ SUB_ACC(conflict_count[i]);
#undef SUB_ACC
pgstat_unlock_entry(entry_ref);
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 3876339ee1..850b20509f 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1966,13 +1966,14 @@ pg_stat_get_replication_slot(PG_FUNCTION_ARGS)
Datum
pg_stat_get_subscription_stats(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_SUBSCRIPTION_STATS_COLS 4
+#define PG_STAT_GET_SUBSCRIPTION_STATS_COLS 10
Oid subid = PG_GETARG_OID(0);
TupleDesc tupdesc;
Datum values[PG_STAT_GET_SUBSCRIPTION_STATS_COLS] = {0};
bool nulls[PG_STAT_GET_SUBSCRIPTION_STATS_COLS] = {0};
PgStat_StatSubEntry *subentry;
PgStat_StatSubEntry allzero;
+ int i = 0;
/* Get subscription stats */
subentry = pgstat_fetch_stat_subscription(subid);
@@ -1985,7 +1986,19 @@ pg_stat_get_subscription_stats(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 3, "sync_error_count",
INT8OID, -1, 0);
- TupleDescInitEntry(tupdesc, (AttrNumber) 4, "stats_reset",
+ TupleDescInitEntry(tupdesc, (AttrNumber) 4, "insert_exists_count",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 5, "update_exists_count",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 6, "update_differ_count",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 7, "update_missing_count",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 8, "delete_missing_count",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 9, "delete_differ_count",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 10, "stats_reset",
TIMESTAMPTZOID, -1, 0);
BlessTupleDesc(tupdesc);
@@ -1997,19 +2010,27 @@ pg_stat_get_subscription_stats(PG_FUNCTION_ARGS)
}
/* subid */
- values[0] = ObjectIdGetDatum(subid);
+ values[i++] = ObjectIdGetDatum(subid);
/* apply_error_count */
- values[1] = Int64GetDatum(subentry->apply_error_count);
+ values[i++] = Int64GetDatum(subentry->apply_error_count);
/* sync_error_count */
- values[2] = Int64GetDatum(subentry->sync_error_count);
+ values[i++] = Int64GetDatum(subentry->sync_error_count);
+
+ /* conflict count */
+ for (int nconflict = 0; nconflict < CONFLICT_NUM_TYPES; nconflict++)
+ values[i + nconflict] = Int64GetDatum(subentry->conflict_count[nconflict]);
+
+ i += CONFLICT_NUM_TYPES;
/* stats_reset */
if (subentry->stat_reset_timestamp == 0)
- nulls[3] = true;
+ nulls[i] = true;
else
- values[3] = TimestampTzGetDatum(subentry->stat_reset_timestamp);
+ values[i] = TimestampTzGetDatum(subentry->stat_reset_timestamp);
+
+ Assert(i + 1 == PG_STAT_GET_SUBSCRIPTION_STATS_COLS);
/* Returns the record as Datum */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 73d9cf8582..6848096b47 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5505,9 +5505,9 @@
{ oid => '6231', descr => 'statistics: information about subscription stats',
proname => 'pg_stat_get_subscription_stats', provolatile => 's',
proparallel => 'r', prorettype => 'record', proargtypes => 'oid',
- proallargtypes => '{oid,oid,int8,int8,timestamptz}',
- proargmodes => '{i,o,o,o,o}',
- proargnames => '{subid,subid,apply_error_count,sync_error_count,stats_reset}',
+ proallargtypes => '{oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,timestamptz}',
+ proargmodes => '{i,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{subid,subid,apply_error_count,sync_error_count,insert_exists_count,update_exists_count,update_differ_count,update_missing_count,delete_missing_count,delete_differ_count,stats_reset}',
prosrc => 'pg_stat_get_subscription_stats' },
{ oid => '6118', descr => 'statistics: information about subscription',
proname => 'pg_stat_get_subscription', prorows => '10', proisstrict => 'f',
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 6b99bb8aad..ad6619bcd0 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -14,6 +14,7 @@
#include "datatype/timestamp.h"
#include "portability/instr_time.h"
#include "postmaster/pgarch.h" /* for MAX_XFN_CHARS */
+#include "replication/conflict.h"
#include "utils/backend_progress.h" /* for backward compatibility */
#include "utils/backend_status.h" /* for backward compatibility */
#include "utils/relcache.h"
@@ -135,6 +136,7 @@ typedef struct PgStat_BackendSubEntry
{
PgStat_Counter apply_error_count;
PgStat_Counter sync_error_count;
+ PgStat_Counter conflict_count[CONFLICT_NUM_TYPES];
} PgStat_BackendSubEntry;
/* ----------
@@ -393,6 +395,7 @@ typedef struct PgStat_StatSubEntry
{
PgStat_Counter apply_error_count;
PgStat_Counter sync_error_count;
+ PgStat_Counter conflict_count[CONFLICT_NUM_TYPES];
TimestampTz stat_reset_timestamp;
} PgStat_StatSubEntry;
@@ -695,6 +698,7 @@ extern PgStat_SLRUStats *pgstat_fetch_slru(void);
*/
extern void pgstat_report_subscription_error(Oid subid, bool is_apply_error);
+extern void pgstat_report_subscription_conflict(Oid subid, ConflictType conflict);
extern void pgstat_create_subscription(Oid subid);
extern void pgstat_drop_subscription(Oid subid);
extern PgStat_StatSubEntry *pgstat_fetch_stat_subscription(Oid subid);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 3a7260d3c1..b5e9c79100 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -17,6 +17,11 @@
/*
* Conflict types that could be encountered when applying remote changes.
+ *
+ * This enum is used in statistics collection (see
+ * PgStat_StatSubEntry::conflict_count) as well, therefore, when adding new
+ * values or reordering existing ones, ensure to review and potentially adjust
+ * the corresponding statistics collection codes.
*/
typedef enum
{
@@ -39,6 +44,8 @@ typedef enum
CT_DELETE_DIFFER,
} ConflictType;
+#define CONFLICT_NUM_TYPES (CT_DELETE_DIFFER + 1)
+
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
extern void ReportApplyConflict(int elevel, ConflictType type,
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 4c789279e5..3fa03b4d76 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2139,9 +2139,15 @@ pg_stat_subscription_stats| SELECT ss.subid,
s.subname,
ss.apply_error_count,
ss.sync_error_count,
+ ss.insert_exists_count,
+ ss.update_exists_count,
+ ss.update_differ_count,
+ ss.update_missing_count,
+ ss.delete_missing_count,
+ ss.delete_differ_count,
ss.stats_reset
FROM pg_subscription s,
- LATERAL pg_stat_get_subscription_stats(s.oid) ss(subid, apply_error_count, sync_error_count, stats_reset);
+ LATERAL pg_stat_get_subscription_stats(s.oid) ss(subid, apply_error_count, sync_error_count, insert_exists_count, update_exists_count, update_differ_count, update_missing_count, delete_missing_count, delete_differ_count, stats_reset);
pg_stat_sys_indexes| SELECT relid,
indexrelid,
schemaname,
diff --git a/src/test/subscription/t/026_stats.pl b/src/test/subscription/t/026_stats.pl
index fb3e5629b3..deaf71275d 100644
--- a/src/test/subscription/t/026_stats.pl
+++ b/src/test/subscription/t/026_stats.pl
@@ -16,6 +16,7 @@ $node_publisher->start;
# Create subscriber node.
my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf', 'track_commit_timestamp = on');
$node_subscriber->start;
@@ -30,6 +31,7 @@ sub create_sub_pub_w_errors
qq[
BEGIN;
CREATE TABLE $table_name(a int);
+ ALTER TABLE $table_name REPLICA IDENTITY FULL;
INSERT INTO $table_name VALUES (1);
COMMIT;
]);
@@ -53,7 +55,7 @@ sub create_sub_pub_w_errors
# infinite error loop due to violating the unique constraint.
my $sub_name = $table_name . '_sub';
$node_subscriber->safe_psql($db,
- qq(CREATE SUBSCRIPTION $sub_name CONNECTION '$publisher_connstr' PUBLICATION $pub_name)
+ qq(CREATE SUBSCRIPTION $sub_name CONNECTION '$publisher_connstr' PUBLICATION $pub_name WITH (detect_conflict = on))
);
$node_publisher->wait_for_catchup($sub_name);
@@ -95,7 +97,7 @@ sub create_sub_pub_w_errors
$node_subscriber->poll_query_until(
$db,
qq[
- SELECT apply_error_count > 0
+ SELECT apply_error_count > 0 AND insert_exists_count > 0
FROM pg_stat_subscription_stats
WHERE subname = '$sub_name'
])
@@ -105,6 +107,85 @@ sub create_sub_pub_w_errors
# Truncate test table so that apply worker can continue.
$node_subscriber->safe_psql($db, qq(TRUNCATE $table_name));
+ # Insert a row on the subscriber.
+ $node_subscriber->safe_psql($db, qq(INSERT INTO $table_name VALUES (2)));
+
+ # Update data from test table on the publisher, raising an error on the
+ # subscriber due to violation of the unique constraint on test table.
+ $node_publisher->safe_psql($db, qq(UPDATE $table_name SET a = 2;));
+
+ # Wait for the apply error to be reported.
+ $node_subscriber->poll_query_until(
+ $db,
+ qq[
+ SELECT update_exists_count > 0
+ FROM pg_stat_subscription_stats
+ WHERE subname = '$sub_name'
+ ])
+ or die
+ qq(Timed out while waiting for update_exists conflict for subscription '$sub_name');
+
+ # Truncate test table so that the update will be skipped and the test can
+ # continue.
+ $node_subscriber->safe_psql($db, qq(TRUNCATE $table_name));
+
+ # delete the data from test table on the publisher. The delete should be
+ # skipped on the subscriber as there are no data in the test table.
+ $node_publisher->safe_psql($db, qq(DELETE FROM $table_name;));
+
+ # Wait for the tuple missing to be reported.
+ $node_subscriber->poll_query_until(
+ $db,
+ qq[
+ SELECT update_missing_count > 0 AND delete_missing_count > 0
+ FROM pg_stat_subscription_stats
+ WHERE subname = '$sub_name'
+ ])
+ or die
+ qq(Timed out while waiting for tuple missing conflict for subscription '$sub_name');
+
+ # Prepare data for further tests.
+ $node_publisher->safe_psql($db, qq(INSERT INTO $table_name VALUES (1)));
+ $node_publisher->wait_for_catchup($sub_name);
+ $node_subscriber->safe_psql($db, qq(
+ TRUNCATE $table_name;
+ INSERT INTO $table_name VALUES (1);
+ ));
+
+ # Update data from test table on the publisher, updating a row on the
+ # subscriber that was modified by a different origin.
+ $node_publisher->safe_psql($db, qq(UPDATE $table_name SET a = 2;));
+
+ $node_subscriber->poll_query_until(
+ $db,
+ qq[
+ SELECT update_differ_count > 0
+ FROM pg_stat_subscription_stats
+ WHERE subname = '$sub_name'
+ ])
+ or die
+ qq(Timed out while waiting for update_differ conflict for subscription '$sub_name');
+
+ # Prepare data for further tests.
+ $node_subscriber->safe_psql($db, qq(
+ TRUNCATE $table_name;
+ INSERT INTO $table_name VALUES (2);
+ ));
+
+ # Delete data to test table on the publisher, deleting a row on the
+ # subscriber that was modified by a different origin.
+ $node_publisher->safe_psql($db, qq(DELETE FROM $table_name;));
+
+ $node_subscriber->poll_query_until(
+ $db,
+ qq[
+ SELECT delete_differ_count > 0
+ FROM pg_stat_subscription_stats
+ WHERE subname = '$sub_name'
+ ])
+ or die
+ qq(Timed out while waiting for delete_differ conflict for subscription '$sub_name');
+
return ($pub_name, $sub_name);
}
@@ -128,11 +209,16 @@ is( $node_subscriber->safe_psql(
$db,
qq(SELECT apply_error_count > 0,
sync_error_count > 0,
+ insert_exists_count > 0,
+ update_differ_count > 0,
+ update_missing_count > 0,
+ delete_missing_count > 0,
+ delete_differ_count > 0,
stats_reset IS NULL
FROM pg_stat_subscription_stats
WHERE subname = '$sub1_name')
),
- qq(t|t|t),
+ qq(t|t|t|t|t|t|t|t),
qq(Check that apply errors and sync errors are both > 0 and stats_reset is NULL for subscription '$sub1_name'.)
);
@@ -146,11 +232,16 @@ is( $node_subscriber->safe_psql(
$db,
qq(SELECT apply_error_count = 0,
sync_error_count = 0,
+ insert_exists_count = 0,
+ update_differ_count = 0,
+ update_missing_count = 0,
+ delete_missing_count = 0,
+ delete_differ_count = 0,
stats_reset IS NOT NULL
FROM pg_stat_subscription_stats
WHERE subname = '$sub1_name')
),
- qq(t|t|t),
+ qq(t|t|t|t|t|t|t|t),
qq(Confirm that apply errors and sync errors are both 0 and stats_reset is not NULL after reset for subscription '$sub1_name'.)
);
@@ -186,11 +277,16 @@ is( $node_subscriber->safe_psql(
$db,
qq(SELECT apply_error_count > 0,
sync_error_count > 0,
+ insert_exists_count > 0,
+ update_differ_count > 0,
+ update_missing_count > 0,
+ delete_missing_count > 0,
+ delete_differ_count > 0,
stats_reset IS NULL
FROM pg_stat_subscription_stats
WHERE subname = '$sub2_name')
),
- qq(t|t|t),
+ qq(t|t|t|t|t|t|t|t),
qq(Confirm that apply errors and sync errors are both > 0 and stats_reset is NULL for sub '$sub2_name'.)
);
@@ -203,11 +299,16 @@ is( $node_subscriber->safe_psql(
$db,
qq(SELECT apply_error_count = 0,
sync_error_count = 0,
+ insert_exists_count = 0,
+ update_differ_count = 0,
+ update_missing_count = 0,
+ delete_missing_count = 0,
+ delete_differ_count = 0,
stats_reset IS NOT NULL
FROM pg_stat_subscription_stats
WHERE subname = '$sub1_name')
),
- qq(t|t|t),
+ qq(t|t|t|t|t|t|t|t),
qq(Confirm that apply errors and sync errors are both 0 and stats_reset is not NULL for sub '$sub1_name' after reset.)
);
@@ -215,11 +316,16 @@ is( $node_subscriber->safe_psql(
$db,
qq(SELECT apply_error_count = 0,
sync_error_count = 0,
+ insert_exists_count = 0,
+ update_differ_count = 0,
+ update_missing_count = 0,
+ delete_missing_count = 0,
+ delete_differ_count = 0,
stats_reset IS NOT NULL
FROM pg_stat_subscription_stats
WHERE subname = '$sub2_name')
),
- qq(t|t|t),
+ qq(t|t|t|t|t|t|t|t),
qq(Confirm that apply errors and sync errors are both 0 and stats_reset is not NULL for sub '$sub2_name' after reset.)
);
--
2.30.0.windows.2
[application/octet-stream] v5-0001-Detect-and-log-conflicts-in-logical-replication.patch (98.9K, ../../OS0PR01MB57166C2566E00676649CF48B94AC2@OS0PR01MB5716.jpnprd01.prod.outlook.com/3-v5-0001-Detect-and-log-conflicts-in-logical-replication.patch)
download | inline diff:
From 9268838cca52c51a218618287f436fd09a1615a1 Mon Sep 17 00:00:00 2001
From: Hou Zhijie <[email protected]>
Date: Fri, 31 May 2024 14:20:45 +0530
Subject: [PATCH v5 1/2] Detect and log conflicts in logical replication
This patch adds a new parameter detect_conflict for CREATE and ALTER
subscription commands. This new parameter will decide if subscription will
go for confict detection. By default, conflict detection will be off for a
subscription.
When conflict detection is enabled, additional logging is triggered in the
following conflict scenarios:
insert_exists: Inserting a row that violates a NOT DEFERRABLE unique constraint.
update_exists: The updated row value violates a NOT DEFERRABLE unique constraint.
update_differ: Updating a row that was previously modified by another origin.
update_missing: The tuple to be updated is missing.
delete_missing: The tuple to be deleted is missing.
delete_differ: Deleting a row that was previously modified by another origin.
For insert_exists conflict, the log can include origin and commit
timestamp details of the conflicting key with track_commit_timestamp
enabled.
update_differ and delete_differ conflicts can only be detected when
track_commit_timestamp is enabled.
---
doc/src/sgml/catalogs.sgml | 9 +
doc/src/sgml/logical-replication.sgml | 21 ++-
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 84 +++++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/subscriptioncmds.c | 31 +++-
src/backend/executor/execIndexing.c | 14 +-
src/backend/executor/execReplication.c | 142 +++++++++++++-
src/backend/replication/logical/Makefile | 1 +
src/backend/replication/logical/conflict.c | 194 ++++++++++++++++++++
src/backend/replication/logical/meson.build | 1 +
src/backend/replication/logical/worker.c | 84 +++++++--
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 6 +-
src/bin/psql/tab-complete.c | 14 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/conflict.h | 50 +++++
src/test/regress/expected/subscription.out | 176 ++++++++++--------
src/test/regress/sql/subscription.sql | 15 ++
src/test/subscription/t/001_rep_changes.pl | 15 +-
src/test/subscription/t/013_partition.pl | 68 ++++---
src/test/subscription/t/029_on_error.pl | 5 +-
src/test/subscription/t/030_origin.pl | 43 +++++
src/tools/pgindent/typedefs.list | 1 +
26 files changed, 850 insertions(+), 155 deletions(-)
create mode 100644 src/backend/replication/logical/conflict.c
create mode 100644 src/include/replication/conflict.h
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index b654fae1b2..b042a5a94a 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8035,6 +8035,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdetectconflict</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription is enabled for conflict detection.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index ccdd24312b..1c37f63e12 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1565,7 +1565,8 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
stop. This is referred to as a <firstterm>conflict</firstterm>. When
replicating <command>UPDATE</command> or <command>DELETE</command>
operations, missing data will not produce a conflict and such operations
- will simply be skipped.
+ will simply be skipped. Please refer to <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ for all the conflicts that will be logged when enabling <literal>detect_conflict</literal>.
</para>
<para>
@@ -1634,6 +1635,24 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ...
SKIP</command></link>.
</para>
+
+ <para>
+ Enabling both <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ and <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ on the subscriber can provide additional details regarding conflicting
+ rows, such as their origin and commit timestamp, in case of a unique
+ constraint violation conflict:
+<screen>
+ERROR: conflict insert_exists detected on relation "public.t"
+DETAIL: Key (a)=(1) already exists in unique index "t_pkey", which was modified by origin 0 in transaction 740 at 2024-06-26 10:47:04.727375+08.
+CONTEXT: processing remote data for replication origin "pg_16389" during message type "INSERT" for replication target relation "public.t" in transaction 740, finished at 0/14F7EC0
+</screen>
+ Users can use these information to make decisions on whether to retain
+ the local change or adopt the remote alteration. For instance, the
+ origin in above log indicates that the existing row was modified by a
+ local change, users can manually perform a remote-change-win resolution
+ by deleting the local row.
+ </para>
</sect1>
<sect1 id="logical-replication-restrictions">
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 476f195622..5f6b83e415 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -228,8 +228,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<link linkend="sql-createsubscription-params-with-disable-on-error"><literal>disable_on_error</literal></link>,
<link linkend="sql-createsubscription-params-with-password-required"><literal>password_required</literal></link>,
<link linkend="sql-createsubscription-params-with-run-as-owner"><literal>run_as_owner</literal></link>,
- <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>, and
- <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>.
+ <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>,
+ <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>, and
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>.
Only a superuser can set <literal>password_required = false</literal>.
</para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..eb51805e81 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,90 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-detect-conflict">
+ <term><literal>detect_conflict</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription is enabled for conflict detection.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ When conflict detection is enabled, additional logging is triggered
+ in the following scenarios:
+ <variablelist>
+ <varlistentry>
+ <term><literal>insert_exists</literal></term>
+ <listitem>
+ <para>
+ Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint. Note that to obtain the origin and commit
+ timestamp details of the conflicting key in the log, ensure that
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. In this scenario, an error will be raised until the
+ conflict is resolved manually.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_exists</literal></term>
+ <listitem>
+ <para>
+ The updated value of a row violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint. Note that to obtain the origin and commit
+ timestamp details of the conflicting key in the log, ensure that
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. In this scenario, an error will be raised until the
+ conflict is resolved manually.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_differ</literal></term>
+ <listitem>
+ <para>
+ Updating a row that was previously modified by another origin.
+ Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. Currenly, the update is always applied regardless of
+ the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be updated was not found. The update will simply be
+ skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be deleted was not found. The delete will simply be
+ skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_differ</literal></term>
+ <listitem>
+ <para>
+ Deleting a row that was previously modified by another origin. Note that this
+ conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. Currenly, the delete is always applied regardless of
+ the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..5a423f4fb0 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->detectconflict = subform->subdetectconflict;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 19cabc9a47..d084bfc48a 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1356,7 +1356,8 @@ REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
subbinary, substream, subtwophasestate, subdisableonerr,
subpasswordrequired, subrunasowner, subfailover,
- subslotname, subsynccommit, subpublications, suborigin)
+ subdetectconflict, subslotname, subsynccommit,
+ subpublications, suborigin)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_stats AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 16d83b3253..512b4273ae 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -70,8 +70,9 @@
#define SUBOPT_PASSWORD_REQUIRED 0x00000800
#define SUBOPT_RUN_AS_OWNER 0x00001000
#define SUBOPT_FAILOVER 0x00002000
-#define SUBOPT_LSN 0x00004000
-#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_DETECT_CONFLICT 0x00004000
+#define SUBOPT_LSN 0x00008000
+#define SUBOPT_ORIGIN 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -97,6 +98,7 @@ typedef struct SubOpts
bool passwordrequired;
bool runasowner;
bool failover;
+ bool detectconflict;
char *origin;
XLogRecPtr lsn;
} SubOpts;
@@ -159,6 +161,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->runasowner = false;
if (IsSet(supported_opts, SUBOPT_FAILOVER))
opts->failover = false;
+ if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT))
+ opts->detectconflict = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
@@ -316,6 +320,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_FAILOVER;
opts->failover = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT) &&
+ strcmp(defel->defname, "detect_conflict") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DETECT_CONFLICT))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DETECT_CONFLICT;
+ opts->detectconflict = defGetBoolean(defel);
+ }
else if (IsSet(supported_opts, SUBOPT_ORIGIN) &&
strcmp(defel->defname, "origin") == 0)
{
@@ -603,7 +616,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -710,6 +724,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_subpasswordrequired - 1] = BoolGetDatum(opts.passwordrequired);
values[Anum_pg_subscription_subrunasowner - 1] = BoolGetDatum(opts.runasowner);
values[Anum_pg_subscription_subfailover - 1] = BoolGetDatum(opts.failover);
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -1148,7 +1164,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR |
SUBOPT_PASSWORD_REQUIRED |
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
- SUBOPT_ORIGIN);
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1258,6 +1274,13 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_subfailover - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DETECT_CONFLICT))
+ {
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
+ replaces[Anum_pg_subscription_subdetectconflict - 1] = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
{
values[Anum_pg_subscription_suborigin - 1] =
diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c
index 9f05b3654c..ef522778a2 100644
--- a/src/backend/executor/execIndexing.c
+++ b/src/backend/executor/execIndexing.c
@@ -207,8 +207,9 @@ ExecOpenIndices(ResultRelInfo *resultRelInfo, bool speculative)
ii = BuildIndexInfo(indexDesc);
/*
- * If the indexes are to be used for speculative insertion, add extra
- * information required by unique index entries.
+ * If the indexes are to be used for speculative insertion or conflict
+ * detection in logical replication, add extra information required by
+ * unique index entries.
*/
if (speculative && ii->ii_Unique)
BuildSpeculativeIndexInfo(indexDesc, ii);
@@ -521,6 +522,11 @@ ExecInsertIndexTuples(ResultRelInfo *resultRelInfo,
* possible that a conflicting tuple is inserted immediately
* after this returns. But this can be used for a pre-check
* before insertion.
+ *
+ * If the 'slot' holds a tuple with valid tid, this tuple will
+ * be ignored when checking conflict. This can help in scenarios
+ * where we want to re-check for conflicts after inserting a
+ * tuple.
* ----------------------------------------------------------------
*/
bool
@@ -536,11 +542,9 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
ExprContext *econtext;
Datum values[INDEX_MAX_KEYS];
bool isnull[INDEX_MAX_KEYS];
- ItemPointerData invalidItemPtr;
bool checkedIndex = false;
ItemPointerSetInvalid(conflictTid);
- ItemPointerSetInvalid(&invalidItemPtr);
/*
* Get information from the result relation info structure.
@@ -629,7 +633,7 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
satisfiesConstraint =
check_exclusion_or_unique_constraint(heapRelation, indexRelation,
- indexInfo, &invalidItemPtr,
+ indexInfo, &slot->tts_tid,
values, isnull, estate, false,
CEOUC_WAIT, true,
conflictTid);
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index d0a89cd577..0680bc86fd 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -23,6 +23,7 @@
#include "commands/trigger.h"
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
+#include "replication/conflict.h"
#include "replication/logicalrelation.h"
#include "storage/lmgr.h"
#include "utils/builtins.h"
@@ -480,6 +481,121 @@ retry:
return found;
}
+/*
+ * Find the tuple that violates the passed in unique index constraint
+ * (conflictindex).
+ *
+ * Return false if there is no conflict and *conflictslot is set to NULL.
+ * Otherwise return true, and the conflicting tuple is locked and returned
+ * in *conflictslot.
+ */
+static bool
+FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
+ Oid conflictindex, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot)
+{
+ Relation rel = resultRelInfo->ri_RelationDesc;
+ ItemPointerData conflictTid;
+ TM_FailureData tmfd;
+ TM_Result res;
+
+ *conflictslot = NULL;
+
+retry:
+ if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
+ &conflictTid, list_make1_oid(conflictindex)))
+ {
+ if (*conflictslot)
+ ExecDropSingleTupleTableSlot(*conflictslot);
+
+ *conflictslot = NULL;
+ return false;
+ }
+
+ *conflictslot = table_slot_create(rel, NULL);
+
+ PushActiveSnapshot(GetLatestSnapshot());
+
+ res = table_tuple_lock(rel, &conflictTid, GetLatestSnapshot(),
+ *conflictslot,
+ GetCurrentCommandId(false),
+ LockTupleShare,
+ LockWaitBlock,
+ 0 /* don't follow updates */ ,
+ &tmfd);
+
+ PopActiveSnapshot();
+
+ switch (res)
+ {
+ case TM_Ok:
+ break;
+ case TM_Updated:
+ /* XXX: Improve handling here */
+ if (ItemPointerIndicatesMovedPartitions(&tmfd.ctid))
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("tuple to be locked was already moved to another partition due to concurrent update, retrying")));
+ else
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent update, retrying")));
+ goto retry;
+ case TM_Deleted:
+ /* XXX: Improve handling here */
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent delete, retrying")));
+ goto retry;
+ case TM_Invisible:
+ elog(ERROR, "attempted to lock invisible tuple");
+ break;
+ default:
+ elog(ERROR, "unexpected table_tuple_lock status: %u", res);
+ break;
+ }
+
+ return true;
+}
+
+/*
+ * Re-check all the unique indexes in 'recheckIndexes' to see if there are
+ * potential conflicts with the tuple in 'slot'.
+ *
+ * This function is invoked after inserting or updating a tuple that detected
+ * potential conflict tuples. It attempts to find the tuple that conflicts with
+ * the provided tuple. This operation may seem redundant with the unique
+ * violation check of indexam, but since we call this function only when we are
+ * detecting conflict in logical replication and encountering potential
+ * conflicts with any unique index constraints (which should not be frequent),
+ * so it's ok. Moreover, upon detecting a conflict, we will report an ERROR and
+ * restart the logical replication, so the additional cost of finding the tuple
+ * should be acceptable.
+ */
+static void
+ReCheckConflictIndexes(ResultRelInfo *resultRelInfo, EState *estate,
+ ConflictType type, List *recheckIndexes,
+ TupleTableSlot *slot)
+{
+ /* Re-check all the unique indexes for potential conflicts */
+ foreach_oid(uniqueidx, resultRelInfo->ri_onConflictArbiterIndexes)
+ {
+ TupleTableSlot *conflictslot;
+
+ if (list_member_oid(recheckIndexes, uniqueidx) &&
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(ERROR, type, resultRelInfo->ri_RelationDesc, uniqueidx,
+ xmin, origin, committs, conflictslot);
+ }
+ }
+}
+
/*
* Insert tuple represented in the slot to the relation, update the indexes,
* and execute any constraints and per-row triggers.
@@ -509,6 +625,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (!skip_tuple)
{
List *recheckIndexes = NIL;
+ List *conflictindexes;
+ bool conflict = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -525,10 +643,17 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
- slot, estate, false, false,
- NULL, NIL, false);
+ slot, estate, false,
+ conflictindexes, &conflict,
+ conflictindexes, false);
+
+ if (conflict)
+ ReCheckConflictIndexes(resultRelInfo, estate, CT_INSERT_EXISTS,
+ recheckIndexes, slot);
/* AFTER ROW INSERT Triggers */
ExecARInsertTriggers(estate, resultRelInfo, slot,
@@ -577,6 +702,8 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
{
List *recheckIndexes = NIL;
TU_UpdateIndexes update_indexes;
+ List *conflictindexes;
+ bool conflict = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -593,12 +720,19 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
simple_table_tuple_update(rel, tid, slot, estate->es_snapshot,
&update_indexes);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
if (resultRelInfo->ri_NumIndices > 0 && (update_indexes != TU_None))
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
- slot, estate, true, false,
- NULL, NIL,
+ slot, estate, true,
+ conflictindexes,
+ &conflict, conflictindexes,
(update_indexes == TU_Summarizing));
+ if (conflict)
+ ReCheckConflictIndexes(resultRelInfo, estate, CT_UPDATE_EXISTS,
+ recheckIndexes, slot);
+
/* AFTER ROW UPDATE Triggers */
ExecARUpdateTriggers(estate, resultRelInfo,
NULL, NULL,
diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile
index ba03eeff1c..1e08bbbd4e 100644
--- a/src/backend/replication/logical/Makefile
+++ b/src/backend/replication/logical/Makefile
@@ -16,6 +16,7 @@ override CPPFLAGS := -I$(srcdir) $(CPPFLAGS)
OBJS = \
applyparallelworker.o \
+ conflict.o \
decode.o \
launcher.o \
logical.o \
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
new file mode 100644
index 0000000000..b7dc73cfce
--- /dev/null
+++ b/src/backend/replication/logical/conflict.c
@@ -0,0 +1,194 @@
+/*-------------------------------------------------------------------------
+ * conflict.c
+ * Functionality for detecting and logging conflicts.
+ *
+ * Copyright (c) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/replication/logical/conflict.c
+ *
+ * This file contains the code for detecting and logging conflicts on
+ * the subscriber during logical replication.
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/commit_ts.h"
+#include "replication/conflict.h"
+#include "replication/origin.h"
+#include "utils/lsyscache.h"
+#include "utils/rel.h"
+
+const char *const ConflictTypeNames[] = {
+ [CT_INSERT_EXISTS] = "insert_exists",
+ [CT_UPDATE_EXISTS] = "update_exists",
+ [CT_UPDATE_DIFFER] = "update_differ",
+ [CT_UPDATE_MISSING] = "update_missing",
+ [CT_UPDATE_MISSING] = "update_missing",
+ [CT_DELETE_MISSING] = "delete_missing",
+ [CT_DELETE_DIFFER] = "delete_differ"
+};
+
+static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
+static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin,
+ RepOriginId localorigin,
+ TimestampTz localts,
+ TupleTableSlot *conflictslot);
+
+/*
+ * Get the xmin and commit timestamp data (origin and timestamp) associated
+ * with the provided local tuple.
+ *
+ * Return true if the commit timestamp data was found, false otherwise.
+ */
+bool
+GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts)
+{
+ Datum xminDatum;
+ bool isnull;
+
+ xminDatum = slot_getsysattr(localslot, MinTransactionIdAttributeNumber,
+ &isnull);
+ *xmin = DatumGetTransactionId(xminDatum);
+ Assert(!isnull);
+
+ /*
+ * The commit timestamp data is not available if track_commit_timestamp is
+ * disabled.
+ */
+ if (!track_commit_timestamp)
+ {
+ *localorigin = InvalidRepOriginId;
+ *localts = 0;
+ return false;
+ }
+
+ return TransactionIdGetCommitTsData(*xmin, localts, localorigin);
+}
+
+/*
+ * Report a conflict when applying remote changes.
+ */
+void
+ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot)
+{
+ ereport(elevel,
+ errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
+ errmsg("conflict %s detected on relation \"%s.%s\"",
+ ConflictTypeNames[type],
+ get_namespace_name(RelationGetNamespace(localrel)),
+ RelationGetRelationName(localrel)),
+ errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
+ localts, conflictslot));
+}
+
+/*
+ * Find all unique indexes to check for a conflict and store them into
+ * ResultRelInfo.
+ */
+void
+InitConflictIndexes(ResultRelInfo *relInfo)
+{
+ List *uniqueIndexes = NIL;
+
+ for (int i = 0; i < relInfo->ri_NumIndices; i++)
+ {
+ Relation indexRelation = relInfo->ri_IndexRelationDescs[i];
+
+ if (indexRelation == NULL)
+ continue;
+
+ /* Detect conflict only for unique indexes */
+ if (!relInfo->ri_IndexRelationInfo[i]->ii_Unique)
+ continue;
+
+ /* Don't support conflict detection for deferrable index */
+ if (!indexRelation->rd_index->indimmediate)
+ continue;
+
+ uniqueIndexes = lappend_oid(uniqueIndexes,
+ RelationGetRelid(indexRelation));
+ }
+
+ relInfo->ri_onConflictArbiterIndexes = uniqueIndexes;
+}
+
+/*
+ * Add an errdetail() line showing conflict detail.
+ */
+static int
+errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot)
+{
+ switch (type)
+ {
+ case CT_INSERT_EXISTS:
+ case CT_UPDATE_EXISTS:
+ {
+ /*
+ * Bulid the index value string. If the return value is NULL,
+ * it indicates that the current user lacks permissions to
+ * view all the columns involved.
+ */
+ char *index_value = build_index_value_desc(conflictidx,
+ conflictslot);
+
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
+ case CT_UPDATE_DIFFER:
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ case CT_UPDATE_MISSING:
+ return errdetail("Did not find the row to be updated.");
+ case CT_DELETE_MISSING:
+ return errdetail("Did not find the row to be deleted.");
+ case CT_DELETE_DIFFER:
+ return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ }
+
+ return 0; /* silence compiler warning */
+}
+
+/*
+ * Helper functions to construct a string describing the contents of an index
+ * entry. See BuildIndexValueDescription for details.
+ */
+static char *
+build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
+{
+ char *conflict_row;
+ Relation indexDesc;
+
+ if (!conflictslot)
+ return NULL;
+
+ /* Assume the index has been locked */
+ indexDesc = index_open(indexoid, NoLock);
+
+ slot_getallattrs(conflictslot);
+
+ conflict_row = BuildIndexValueDescription(indexDesc,
+ conflictslot->tts_values,
+ conflictslot->tts_isnull);
+
+ index_close(indexDesc, NoLock);
+
+ return conflict_row;
+}
diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build
index 3dec36a6de..3d36249d8a 100644
--- a/src/backend/replication/logical/meson.build
+++ b/src/backend/replication/logical/meson.build
@@ -2,6 +2,7 @@
backend_sources += files(
'applyparallelworker.c',
+ 'conflict.c',
'decode.c',
'launcher.c',
'logical.c',
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index c0bda6269b..c49240a6a4 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -167,6 +167,7 @@
#include "postmaster/bgworker.h"
#include "postmaster/interrupt.h"
#include "postmaster/walwriter.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalproto.h"
#include "replication/logicalrelation.h"
@@ -2461,7 +2462,10 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
EState *estate = edata->estate;
/* We must open indexes here. */
- ExecOpenIndices(relinfo, false);
+ ExecOpenIndices(relinfo, MySubscription->detectconflict);
+
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
@@ -2649,7 +2653,7 @@ apply_handle_update_internal(ApplyExecutionData *edata,
MemoryContext oldctx;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
- ExecOpenIndices(relinfo, false);
+ ExecOpenIndices(relinfo, MySubscription->detectconflict);
found = FindReplTupleInLocalRel(edata, localrel,
&relmapentry->remoterel,
@@ -2664,6 +2668,20 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2671,6 +2689,9 @@ apply_handle_update_internal(ApplyExecutionData *edata,
EvalPlanQualSetSlot(&epqstate, remoteslot);
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
+
/* Do the actual update. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
@@ -2681,13 +2702,10 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2810,6 +2828,20 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/* If found delete it. */
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_DELETE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
EvalPlanQualSetSlot(&epqstate, localslot);
/* Do the actual delete. */
@@ -2821,13 +2853,10 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/*
* The tuple to be deleted could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be deleted "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2994,6 +3023,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ResultRelInfo *partrelinfo_new;
Relation partrel_new;
bool found;
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3005,16 +3037,28 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing
* except for emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation's partition \"%s\"",
- RelationGetRelationName(partrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING,
+ partrel, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+
return;
}
+ /*
+ * If conflict detection is enabled, check whether the local
+ * tuple was modified by a different origin. If detected,
+ * report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, partrel,
+ InvalidOid, localxmin, localorigin,
+ localts, NULL);
+
/*
* Apply the update to the local tuple, putting the result in
* remoteslot_part.
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index b8b1888bd3..829f9b3e88 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4760,6 +4760,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subdetectconflict;
int i,
ntups;
@@ -4832,11 +4833,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subdetectconflict\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdetectconflict\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4875,6 +4882,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subdetectconflict = PQfnumber(res, "subdetectconflict");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4921,6 +4929,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subdetectconflict =
+ pg_strdup(PQgetvalue(res, i, i_subdetectconflict));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5161,6 +5171,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subfailover, "t") == 0)
appendPQExpBufferStr(query, ", failover = true");
+ if (strcmp(subinfo->subdetectconflict, "t") == 0)
+ appendPQExpBufferStr(query, ", detect_conflict = true");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e5870a9..bbd7cbeff6 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subdetectconflict;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f234c..fef1ad0d70 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6607,6 +6607,10 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subdetectconflict AS \"%s\"\n",
+ gettext_noop("Detect conflict"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index d453e224d9..219fac7e71 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1946,9 +1946,10 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "disable_on_error", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "detect_conflict", "disable_on_error",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit");
/* ALTER SUBSCRIPTION <name> SKIP ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SKIP", "("))
COMPLETE_WITH("lsn");
@@ -3363,9 +3364,10 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit", "two_phase");
+ "detect_conflict", "disable_on_error", "enabled",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit", "two_phase");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..17daf11dc7 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subdetectconflict; /* True if replication should perform
+ * conflict detection */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -151,6 +154,7 @@ typedef struct Subscription
* (i.e. the main slot and the table sync
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool detectconflict; /* True if conflict detection is enabled */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
new file mode 100644
index 0000000000..3a7260d3c1
--- /dev/null
+++ b/src/include/replication/conflict.h
@@ -0,0 +1,50 @@
+/*-------------------------------------------------------------------------
+ * conflict.h
+ * Exports for conflict detection and log
+ *
+ * Copyright (c) 2012-2024, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef CONFLICT_H
+#define CONFLICT_H
+
+#include "access/xlogdefs.h"
+#include "executor/tuptable.h"
+#include "nodes/execnodes.h"
+#include "utils/relcache.h"
+#include "utils/timestamp.h"
+
+/*
+ * Conflict types that could be encountered when applying remote changes.
+ */
+typedef enum
+{
+ /* The row to be inserted violates unique constraint */
+ CT_INSERT_EXISTS,
+
+ /* The updated row value violates unique constraint */
+ CT_UPDATE_EXISTS,
+
+ /* The row to be updated was modified by a different origin */
+ CT_UPDATE_DIFFER,
+
+ /* The row to be updated is missing */
+ CT_UPDATE_MISSING,
+
+ /* The row to be deleted is missing */
+ CT_DELETE_MISSING,
+
+ /* The row to be deleted was modified by a different origin */
+ CT_DELETE_DIFFER,
+} ConflictType;
+
+extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts);
+extern void ReportApplyConflict(int elevel, ConflictType type,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot);
+extern void InitConflictIndexes(ResultRelInfo *relInfo);
+
+#endif
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 5c2f1ee517..3d8b8c5d32 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -116,18 +116,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +145,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +157,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +176,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +188,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +223,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +255,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +279,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +314,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +332,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,10 +371,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -383,10 +383,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,10 +396,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -412,18 +412,42 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+ERROR: detect_conflict requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | t | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 3e5ba4cb8c..a77b196704 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -290,6 +290,21 @@ ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- let's do some tests with pg_create_subscription rather than superuser
SET SESSION AUTHORIZATION regress_subscription_user3;
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 471e981962..78c0307165 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -331,11 +331,12 @@ is( $result, qq(1|bar
2|baz),
'update works with REPLICA IDENTITY FULL and a primary key');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1");
-$node_subscriber->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (detect_conflict = true)"
+);
$node_subscriber->safe_psql('postgres', "DELETE FROM tab_full_pk");
@@ -352,10 +353,10 @@ $node_publisher->wait_for_catchup('tap_sub');
my $logfile = slurp_file($node_subscriber->logfile, $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation "tab_full_pk"/,
+ qr/conflict update_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be updated./m,
'update target row is missing');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab_full_pk"/,
+ qr/conflict delete_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be deleted./m,
'delete target row is missing');
$node_subscriber->append_conf('postgresql.conf',
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 29580525a9..7a66a06b51 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -343,12 +343,12 @@ $result =
$node_subscriber2->safe_psql('postgres', "SELECT a FROM tab1 ORDER BY 1");
is($result, qq(), 'truncate of tab1 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = true)"
+);
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 VALUES (1, 'foo'), (4, 'bar'), (10, 'baz')");
@@ -372,21 +372,21 @@ $node_publisher->wait_for_catchup('sub2');
my $logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab1_2_2"/,
+ qr/conflict update_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_1"/,
+ qr/conflict delete_missing detected on relation "public.tab1_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_2_2"/,
+ qr/conflict delete_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_def"/,
+ qr/conflict delete_missing detected on relation "public.tab1_def".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_def');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = false)"
+);
# Tests for replication using root table identity and schema
@@ -773,12 +773,12 @@ pub_tab2|3|yyy
pub_tab2|5|zzz
xxx_c|6|aaa), 'inserts into tab2 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = true)"
+);
$node_subscriber1->safe_psql('postgres', "DELETE FROM tab2");
@@ -796,15 +796,35 @@ $node_publisher->wait_for_catchup('sub2');
$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab2_1"/,
+ qr/conflict update_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab2_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab2_1"/,
+ qr/conflict delete_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab2_1');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+# Enable the track_commit_timestamp to detect the conflict when attempting
+# to update a row that was previously modified by a different origin.
+$node_subscriber1->append_conf('postgresql.conf', 'track_commit_timestamp = on');
+$node_subscriber1->restart;
+
+$node_subscriber1->safe_psql('postgres', "INSERT INTO tab2 VALUES (3, 'yyy')");
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab2 SET b = 'quux' WHERE a = 3");
+
+$node_publisher->wait_for_catchup('sub_viaroot');
+
+$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
+ok( $logfile =~
+ qr/Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/,
+ 'updating a tuple that was modified by a different origin');
+
+# The remaining tests no longer test conflict detection.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = false)"
+);
+
+$node_subscriber1->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_subscriber1->restart;
# Test that replication continues to work correctly after altering the
# partition of a partitioned target table.
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 0ab57a4b5b..496a3c6cd9 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -30,7 +30,7 @@ sub test_skip_lsn
# ERROR with its CONTEXT when retrieving this information.
my $contents = slurp_file($node_subscriber->logfile, $offset);
$contents =~
- qr/duplicate key value violates unique constraint "tbl_pkey".*\n.*DETAIL:.*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
+ qr/conflict insert_exists detected on relation "public.tbl".*\n.*DETAIL:.* Key \(i\)=\(1\) already exists in unique index "tbl_pkey", which was modified by origin \d+ in transaction \d+ at .*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
or die "could not get error-LSN";
my $lsn = $1;
@@ -83,6 +83,7 @@ $node_subscriber->append_conf(
'postgresql.conf',
qq[
max_prepared_transactions = 10
+track_commit_timestamp = on
]);
$node_subscriber->start;
@@ -109,7 +110,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub FOR TABLE tbl");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on)"
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
# Initial synchronization failure causes the subscription to be disabled.
diff --git a/src/test/subscription/t/030_origin.pl b/src/test/subscription/t/030_origin.pl
index 056561f008..8c929c07c7 100644
--- a/src/test/subscription/t/030_origin.pl
+++ b/src/test/subscription/t/030_origin.pl
@@ -27,9 +27,14 @@ my $stderr;
my $node_A = PostgreSQL::Test::Cluster->new('node_A');
$node_A->init(allows_streaming => 'logical');
$node_A->start;
+
# node_B
my $node_B = PostgreSQL::Test::Cluster->new('node_B');
$node_B->init(allows_streaming => 'logical');
+
+# Enable the track_commit_timestamp to detect the conflict when attempting to
+# update a row that was previously modified by a different origin.
+$node_B->append_conf('postgresql.conf', 'track_commit_timestamp = on');
$node_B->start;
# Create table on node_A
@@ -139,6 +144,44 @@ is($result, qq(),
'Remote data originating from another node (not the publisher) is not replicated when origin parameter is none'
);
+###############################################################################
+# Check that the conflict can be detected when attempting to update or
+# delete a row that was previously modified by a different source.
+###############################################################################
+
+$node_B->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_BC SET (detect_conflict = true);
+ DELETE FROM tab;");
+
+$node_A->safe_psql('postgres', "INSERT INTO tab VALUES (32);");
+
+# The update should update the row on node B that was inserted by node A.
+$node_C->safe_psql('postgres', "UPDATE tab SET a = 33 WHERE a = 32;");
+
+$node_B->wait_for_log(
+ qr/Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
+$node_B->safe_psql('postgres', "DELETE FROM tab;");
+$node_A->safe_psql('postgres', "INSERT INTO tab VALUES (33);");
+
+# The delete should remove the row on node B that was inserted by node A.
+$node_C->safe_psql('postgres', "DELETE FROM tab WHERE a = 33;");
+
+$node_B->wait_for_log(
+ qr/Deleting a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
+$node_A->wait_for_catchup($subname_BA);
+$node_B->wait_for_catchup($subname_AB);
+
+# The remaining tests no longer test conflict detection.
+$node_B->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_BC SET (detect_conflict = false);");
+
+$node_B->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_B->restart;
+
###############################################################################
# Specifying origin = NONE indicates that the publisher should only replicate the
# changes that are generated locally from node_B, but in this case since the
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b4d7f9217c..2098ed7467 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -467,6 +467,7 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictType
ConnCacheEntry
ConnCacheKey
ConnParams
--
2.30.0.windows.2
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Conflict detection and logging in logical replication
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
@ 2024-07-18 05:50 ` shveta malik <[email protected]>
2 siblings, 0 replies; 50+ messages in thread
From: shveta malik @ 2024-07-18 05:50 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Nisha Moond <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Amit Kapila <[email protected]>; Ashutosh Bapat <[email protected]>; shveta malik <[email protected]>
On Thu, Jul 18, 2024 at 7:52 AM Zhijie Hou (Fujitsu)
<[email protected]> wrote:
>
> On Thursday, July 11, 2024 1:03 PM shveta malik <[email protected]> wrote:
>
> Hi,
>
> Thanks for the comments!
>
> >
> > I have one concern about how we deal with conflicts. As for insert_exists, we
> > keep on erroring out while raising conflict, until it is manually resolved:
> > ERROR: conflict insert_exists detected
> >
> > But for other cases, we just log conflict and either skip or apply the operation. I
> > LOG: conflict update_differ detected
> > DETAIL: Updating a row that was modified by a different origin
> >
> > I know that it is no different than HEAD. But now since we are logging conflicts
> > explicitly, we should call out default behavior on each conflict. I see some
> > incomplete and scattered info in '31.5.
> > Conflicts' section saying that:
> > "When replicating UPDATE or DELETE operations, missing data will not
> > produce a conflict and such operations will simply be skipped."
> > (lets say it as pt a)
> >
> > Also some more info in a later section saying (pt b):
> > :A conflict will produce an error and will stop the replication; it must be resolved
> > manually by the user."
> >
> > My suggestions:
> > 1) in point a above, shall we have:
> > missing data or differing data (i.e. somehow reword to accommodate
> > update_differ and delete_differ cases)
>
> I am not sure if rewording existing words is better. I feel adding a link to
> let user refer to the detect_conflict section for the all the
> conflicts is sufficient, so did like that.
Agree, it looks better with detect_conflict link.
> >
> > 2)
> > monitoring.sgml: Below are my suggestions, please change if you feel apt.
> >
> > 2a) insert_exists_count : Number of times inserting a row that violates a NOT
> > DEFERRABLE unique constraint while applying changes. Suggestion: Number of
> > times a row insertion violated a NOT DEFERRABLE unique constraint while
> > applying changes.
> >
> > 2b) update_differ_count : Number of times updating a row that was previously
> > modified by another source while applying changes. Suggestion: Number of times
> > update was performed on a row that was previously modified by another source
> > while applying changes.
> >
> > 2c) delete_differ_count: Number of times deleting a row that was previously
> > modified by another source while applying changes. Suggestion: Number of times
> > delete was performed on a row that was previously modified by another source
> > while applying changes.
>
> I am a bit unsure which one is better, so I didn't change in this version.
I still feel the wording is bit unclear/incomplete Also to be
consistent with previous fields (see sync_error_count:Number of times
an error occurred during the initial table synchronization), we should
at-least have it in past tense. Another way of writing could be:
'Number of times inserting a row violated a NOT DEFERRABLE unique
constraint while applying changes.' and likewise for each conflict
field.
> >
> > 5)
> > conflict.h:CONFLICT_NUM_TYPES
> >
> > --Shall the macro be CONFLICT_TYPES_NUM instead?
>
> I think the current style followed existing ones(e.g. IOOP_NUM_TYPES,
> BACKEND_NUM_TYPES, IOOBJECT_NUM_TYPES ...), so I didn't change this.
>
> Attach the V5 patch set which changed the following:
> 1. addressed shveta's comments which are not mentioned above.
> 2. support update_exists conflict which indicates
> that the updated value of a row violates the unique constraint.
Thanks for making the changes.
thanks
Shveta
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Conflict detection and logging in logical replication
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
@ 2024-07-19 08:36 ` shveta malik <[email protected]>
2024-07-22 09:03 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2 siblings, 1 reply; 50+ messages in thread
From: shveta malik @ 2024-07-19 08:36 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Nisha Moond <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Amit Kapila <[email protected]>; Ashutosh Bapat <[email protected]>; shveta malik <[email protected]>
On Thu, Jul 18, 2024 at 7:52 AM Zhijie Hou (Fujitsu)
<[email protected]> wrote:
>
> Attach the V5 patch set which changed the following.
Thanks for the patch. Tested that previous reported issues are fixed.
Please have a look at below scenario, I was expecting it to raise
'update_differ' but it raised both 'update_differ' and 'delete_differ'
together:
-------------------------
Pub:
create table tab (a int not null, b int primary key);
create publication pub1 for table tab;
Sub (partitioned table):
create table tab (a int not null, b int primary key) partition by
range (b);
create table tab_1 partition of tab for values from (minvalue) to
(100);
create table tab_2 partition of tab for values from (100) to
(maxvalue);
create subscription sub1 connection '.......' publication pub1 WITH
(detect_conflict=true);
Pub - insert into tab values (1,1);
Sub - update tab set b=1000 where a=1;
Pub - update tab set b=1000 where a=1; -->update_missing detected
correctly as b=1 will not be found on sub.
Pub - update tab set b=1 where b=1000; -->update_differ expected, but
it gives both update_differ and delete_differ.
-------------------------
Few trivial comments:
1)
Commit msg:
For insert_exists conflict, the log can include origin and commit
timestamp details of the conflicting key with track_commit_timestamp
enabled.
--Please add update_exists as well.
2)
execReplication.c:
Return false if there is no conflict and *conflictslot is set to NULL.
--This gives a feeling that this function will return false if both
the conditions are true. But instead first one is the condition, while
the second is action. Better to rephrase to:
Returns false if there is no conflict. Sets *conflictslot to NULL in
such a case.
Or
Sets *conflictslot to NULL and returns false in case of no conflict.
3)
FindConflictTuple() shares some code parts with
RelationFindReplTupleByIndex() and RelationFindReplTupleSeq() for
checking status in 'res'. Is it worth making a function to be used in
all three.
thanks
Shveta
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Conflict detection and logging in logical replication
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-19 08:36 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
@ 2024-07-22 09:03 ` shveta malik <[email protected]>
2024-07-25 06:34 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
0 siblings, 1 reply; 50+ messages in thread
From: shveta malik @ 2024-07-22 09:03 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Nisha Moond <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Amit Kapila <[email protected]>; Ashutosh Bapat <[email protected]>; shveta malik <[email protected]>
On Fri, Jul 19, 2024 at 2:06 PM shveta malik <[email protected]> wrote:
>
> On Thu, Jul 18, 2024 at 7:52 AM Zhijie Hou (Fujitsu)
> <[email protected]> wrote:
> >
> > Attach the V5 patch set which changed the following.
>
Please find last batch of comments on v5:
patch001:
1)
create subscription sub1 ... (detect_conflict=true);
I think it will be good to give WARNING here indicating that
detect_conflict is enabled but track_commit_timestamp is disabled and
thus few conflicts detection may not work. (Rephrase as apt)
2)
013_partition.pl: Since we have added update_differ testcase here, we
shall add delete_differ as well. And in some file where appropriate,
we shall add update_exists test as well.
3)
013_partition.pl (#799,802):
For update_missing and delete_missing, we have log verification format
as 'qr/conflict delete_missing/update_missing detected on relation '.
But for update_differ, we do not capture "conflict update_differ
detected on relation ...". We capture only the DETAIL.
I think we should be consistent and capture conflict name here as well.
patch002:
4)
pg_stat_get_subscription_stats():
---------
/* conflict count */
for (int nconflict = 0; nconflict < CONFLICT_NUM_TYPES; nconflict++)
values[i + nconflict] = Int64GetDatum(subentry->conflict_count[nconflict]);
i += CONFLICT_NUM_TYPES;
---------
Can't we do values[i++] here as well (instead of values[i +
nconflict])? Then we don't need to do 'i += CONFLICT_NUM_TYPES'.
5)
026_stats.pl:
Wherever we are checking this: 'Apply and Sync errors are > 0 and
reset timestamp is NULL', we need to check update_exssts_count as well
along with other counts.
thanks
Shveta
^ permalink raw reply [nested|flat] 50+ messages in thread
* RE: Conflict detection and logging in logical replication
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-19 08:36 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-22 09:03 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
@ 2024-07-25 06:34 ` Zhijie Hou (Fujitsu) <[email protected]>
2024-07-25 10:42 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
2024-07-26 09:33 ` Re: Conflict detection and logging in logical replication Nisha Moond <[email protected]>
0 siblings, 2 replies; 50+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-07-25 06:34 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Nisha Moond <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Amit Kapila <[email protected]>; Ashutosh Bapat <[email protected]>
On Monday, July 22, 2024 5:03 PM shveta malik <[email protected]> wrote:
>
> On Fri, Jul 19, 2024 at 2:06 PM shveta malik <[email protected]> wrote:
> >
> > On Thu, Jul 18, 2024 at 7:52 AM Zhijie Hou (Fujitsu)
> > <[email protected]> wrote:
> > >
> > > Attach the V5 patch set which changed the following.
> >
>
> Please find last batch of comments on v5:
Thanks Shveta and Nisha for giving comments!
>
>
> 2)
> 013_partition.pl: Since we have added update_differ testcase here, we shall
> add delete_differ as well.
I didn't add tests for delete_differ in partition test, because I think the main
codes and functionality of delete_differ have been tested in 030_origin.pl.
The test for update_differ is needed because the patch adds new codes in
partition code path to report this conflict.
Here is the V6 patch set which addressed Shveta and Nisha's comments
in [1][2][3][4].
[1] https://www.postgresql.org/message-id/CAJpy0uDWdw2W-S8boFU0KOcZjw0%2BsFFgLrHYrr1TROtrcTPZMg%40mail.g...
[2] https://www.postgresql.org/message-id/CAJpy0uDGJXdVCGoaRHP-5G0pL0zhuZaRJSqxOxs%3DCNsSwc%2BSJQ%40mail...
[3] https://www.postgresql.org/message-id/CAJpy0uC%2B1puapWdOnAMSS%3DQUp_1jj3GfAeivE0JRWbpqrUy%3Dug%40ma...
[4] https://www.postgresql.org/message-id/CABdArM6%2BN1Xy_%2BtK%2Bu-H%3DsCB%2B92rAUh8qH6GDsB%2B1naKzgGKz...
Best Regards,
Hou zj
Attachments:
[application/octet-stream] v6-0002-Collect-statistics-about-conflicts-in-logical-rep.patch (23.7K, ../../TYAPR01MB5724B31BE097CFECF8A7D0E794AB2@TYAPR01MB5724.jpnprd01.prod.outlook.com/2-v6-0002-Collect-statistics-about-conflicts-in-logical-rep.patch)
download | inline diff:
From 30379de51c6f67c65244c772dfcdadc87986e77e Mon Sep 17 00:00:00 2001
From: Hou Zhijie <[email protected]>
Date: Wed, 3 Jul 2024 10:34:10 +0800
Subject: [PATCH v6 2/2] Collect statistics about conflicts in logical
replication
This commit adds columns in view pg_stat_subscription_stats to show
information about the conflict which occur during the application of
logical replication changes. Currently, the following columns are added.
insert_exists_count:
Number of times a row insertion violated a NOT DEFERRABLE unique constraint.
update_exists_count:
Number of times that the updated value of a row violates a NOT DEFERRABLE unique constraint.
update_differ_count:
Number of times an update was performed on a row that was previously modified by another origin.
update_missing_count:
Number of times that the tuple to be updated is missing.
delete_missing_count:
Number of times that the tuple to be deleted is missing.
delete_differ_count:
Number of times a delete was performed on a row that was previously modified by another origin.
The conflicts will be tracked only when detect_conflict option of the
subscription is enabled. Additionally, update_differ and delete_differ
can be detected only when track_commit_timestamp is enabled.
---
doc/src/sgml/monitoring.sgml | 80 ++++++++++-
doc/src/sgml/ref/create_subscription.sgml | 5 +-
src/backend/catalog/system_views.sql | 6 +
src/backend/replication/logical/conflict.c | 4 +
.../utils/activity/pgstat_subscription.c | 17 +++
src/backend/utils/adt/pgstatfuncs.c | 33 ++++-
src/include/catalog/pg_proc.dat | 6 +-
src/include/pgstat.h | 4 +
src/include/replication/conflict.h | 7 +
src/test/regress/expected/rules.out | 8 +-
src/test/subscription/t/026_stats.pl | 125 +++++++++++++++++-
11 files changed, 274 insertions(+), 21 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 55417a6fa9..e3f8f3708b 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -507,7 +507,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<row>
<entry><structname>pg_stat_subscription_stats</structname><indexterm><primary>pg_stat_subscription_stats</primary></indexterm></entry>
- <entry>One row per subscription, showing statistics about errors.
+ <entry>One row per subscription, showing statistics about errors and conflicts.
See <link linkend="monitoring-pg-stat-subscription-stats">
<structname>pg_stat_subscription_stats</structname></link> for details.
</entry>
@@ -2171,6 +2171,84 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>insert_exists_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times a row insertion violated a
+ <literal>NOT DEFERRABLE</literal> unique constraint while applying
+ changes. This conflict is counted only if
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ is enabled
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>update_exists_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times that the updated value of a row violates a
+ <literal>NOT DEFERRABLE</literal> unique constraint while applying
+ changes. This conflict is counted only if
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ is enabled
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>update_differ_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times an update was performed on a row that was previously
+ modified by another source while applying changes. This conflict is
+ counted only when the
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ option of the subscription and <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ are enabled
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>update_missing_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times that the tuple to be updated was not found while applying
+ changes. This conflict is counted only if
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ is enabled
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>delete_missing_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times that the tuple to be deleted was not found while applying
+ changes. This conflict is counted only if
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ is enabled
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>delete_differ_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times a delete was performed on a row that was previously
+ modified by another source while applying changes. This conflict is
+ counted only when the
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ option of the subscription and <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ are enabled
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>stats_reset</structfield> <type>timestamp with time zone</type>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index eb51805e81..04f3ba6e9a 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -437,8 +437,9 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
The default is <literal>false</literal>.
</para>
<para>
- When conflict detection is enabled, additional logging is triggered
- in the following scenarios:
+ When conflict detection is enabled, additional logging is triggered and
+ the conflict statistics are collected(displayed in the
+ <link linkend="monitoring-pg-stat-subscription-stats"><structname>pg_stat_subscription_stats</structname></link> view) in the following scenarios:
<variablelist>
<varlistentry>
<term><literal>insert_exists</literal></term>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index d084bfc48a..5244d8e356 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1366,6 +1366,12 @@ CREATE VIEW pg_stat_subscription_stats AS
s.subname,
ss.apply_error_count,
ss.sync_error_count,
+ ss.insert_exists_count,
+ ss.update_exists_count,
+ ss.update_differ_count,
+ ss.update_missing_count,
+ ss.delete_missing_count,
+ ss.delete_differ_count,
ss.stats_reset
FROM pg_subscription as s,
pg_stat_get_subscription_stats(s.oid) as ss;
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 4918011a7f..6e2f15cd7d 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -15,8 +15,10 @@
#include "postgres.h"
#include "access/commit_ts.h"
+#include "pgstat.h"
#include "replication/conflict.h"
#include "replication/origin.h"
+#include "replication/worker_internal.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
@@ -77,6 +79,8 @@ ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
RepOriginId localorigin, TimestampTz localts,
TupleTableSlot *conflictslot)
{
+ pgstat_report_subscription_conflict(MySubscription->oid, type);
+
ereport(elevel,
errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
errmsg("conflict %s detected on relation \"%s.%s\"",
diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c
index d9af8de658..e06c92727e 100644
--- a/src/backend/utils/activity/pgstat_subscription.c
+++ b/src/backend/utils/activity/pgstat_subscription.c
@@ -39,6 +39,21 @@ pgstat_report_subscription_error(Oid subid, bool is_apply_error)
pending->sync_error_count++;
}
+/*
+ * Report a subscription conflict.
+ */
+void
+pgstat_report_subscription_conflict(Oid subid, ConflictType type)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStat_BackendSubEntry *pending;
+
+ entry_ref = pgstat_prep_pending_entry(PGSTAT_KIND_SUBSCRIPTION,
+ InvalidOid, subid, NULL);
+ pending = entry_ref->pending;
+ pending->conflict_count[type]++;
+}
+
/*
* Report creating the subscription.
*/
@@ -101,6 +116,8 @@ pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
#define SUB_ACC(fld) shsubent->stats.fld += localent->fld
SUB_ACC(apply_error_count);
SUB_ACC(sync_error_count);
+ for (int i = 0; i < CONFLICT_NUM_TYPES; i++)
+ SUB_ACC(conflict_count[i]);
#undef SUB_ACC
pgstat_unlock_entry(entry_ref);
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 3876339ee1..e36ddb4cac 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1966,13 +1966,14 @@ pg_stat_get_replication_slot(PG_FUNCTION_ARGS)
Datum
pg_stat_get_subscription_stats(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_SUBSCRIPTION_STATS_COLS 4
+#define PG_STAT_GET_SUBSCRIPTION_STATS_COLS 10
Oid subid = PG_GETARG_OID(0);
TupleDesc tupdesc;
Datum values[PG_STAT_GET_SUBSCRIPTION_STATS_COLS] = {0};
bool nulls[PG_STAT_GET_SUBSCRIPTION_STATS_COLS] = {0};
PgStat_StatSubEntry *subentry;
PgStat_StatSubEntry allzero;
+ int i = 0;
/* Get subscription stats */
subentry = pgstat_fetch_stat_subscription(subid);
@@ -1985,7 +1986,19 @@ pg_stat_get_subscription_stats(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 3, "sync_error_count",
INT8OID, -1, 0);
- TupleDescInitEntry(tupdesc, (AttrNumber) 4, "stats_reset",
+ TupleDescInitEntry(tupdesc, (AttrNumber) 4, "insert_exists_count",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 5, "update_exists_count",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 6, "update_differ_count",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 7, "update_missing_count",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 8, "delete_missing_count",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 9, "delete_differ_count",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 10, "stats_reset",
TIMESTAMPTZOID, -1, 0);
BlessTupleDesc(tupdesc);
@@ -1997,19 +2010,25 @@ pg_stat_get_subscription_stats(PG_FUNCTION_ARGS)
}
/* subid */
- values[0] = ObjectIdGetDatum(subid);
+ values[i++] = ObjectIdGetDatum(subid);
/* apply_error_count */
- values[1] = Int64GetDatum(subentry->apply_error_count);
+ values[i++] = Int64GetDatum(subentry->apply_error_count);
/* sync_error_count */
- values[2] = Int64GetDatum(subentry->sync_error_count);
+ values[i++] = Int64GetDatum(subentry->sync_error_count);
+
+ /* conflict count */
+ for (int nconflict = 0; nconflict < CONFLICT_NUM_TYPES; nconflict++)
+ values[i++] = Int64GetDatum(subentry->conflict_count[nconflict]);
/* stats_reset */
if (subentry->stat_reset_timestamp == 0)
- nulls[3] = true;
+ nulls[i] = true;
else
- values[3] = TimestampTzGetDatum(subentry->stat_reset_timestamp);
+ values[i] = TimestampTzGetDatum(subentry->stat_reset_timestamp);
+
+ Assert(i + 1 == PG_STAT_GET_SUBSCRIPTION_STATS_COLS);
/* Returns the record as Datum */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 73d9cf8582..6848096b47 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5505,9 +5505,9 @@
{ oid => '6231', descr => 'statistics: information about subscription stats',
proname => 'pg_stat_get_subscription_stats', provolatile => 's',
proparallel => 'r', prorettype => 'record', proargtypes => 'oid',
- proallargtypes => '{oid,oid,int8,int8,timestamptz}',
- proargmodes => '{i,o,o,o,o}',
- proargnames => '{subid,subid,apply_error_count,sync_error_count,stats_reset}',
+ proallargtypes => '{oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,timestamptz}',
+ proargmodes => '{i,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{subid,subid,apply_error_count,sync_error_count,insert_exists_count,update_exists_count,update_differ_count,update_missing_count,delete_missing_count,delete_differ_count,stats_reset}',
prosrc => 'pg_stat_get_subscription_stats' },
{ oid => '6118', descr => 'statistics: information about subscription',
proname => 'pg_stat_get_subscription', prorows => '10', proisstrict => 'f',
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 6b99bb8aad..ad6619bcd0 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -14,6 +14,7 @@
#include "datatype/timestamp.h"
#include "portability/instr_time.h"
#include "postmaster/pgarch.h" /* for MAX_XFN_CHARS */
+#include "replication/conflict.h"
#include "utils/backend_progress.h" /* for backward compatibility */
#include "utils/backend_status.h" /* for backward compatibility */
#include "utils/relcache.h"
@@ -135,6 +136,7 @@ typedef struct PgStat_BackendSubEntry
{
PgStat_Counter apply_error_count;
PgStat_Counter sync_error_count;
+ PgStat_Counter conflict_count[CONFLICT_NUM_TYPES];
} PgStat_BackendSubEntry;
/* ----------
@@ -393,6 +395,7 @@ typedef struct PgStat_StatSubEntry
{
PgStat_Counter apply_error_count;
PgStat_Counter sync_error_count;
+ PgStat_Counter conflict_count[CONFLICT_NUM_TYPES];
TimestampTz stat_reset_timestamp;
} PgStat_StatSubEntry;
@@ -695,6 +698,7 @@ extern PgStat_SLRUStats *pgstat_fetch_slru(void);
*/
extern void pgstat_report_subscription_error(Oid subid, bool is_apply_error);
+extern void pgstat_report_subscription_conflict(Oid subid, ConflictType conflict);
extern void pgstat_create_subscription(Oid subid);
extern void pgstat_drop_subscription(Oid subid);
extern PgStat_StatSubEntry *pgstat_fetch_stat_subscription(Oid subid);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 3a7260d3c1..b5e9c79100 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -17,6 +17,11 @@
/*
* Conflict types that could be encountered when applying remote changes.
+ *
+ * This enum is used in statistics collection (see
+ * PgStat_StatSubEntry::conflict_count) as well, therefore, when adding new
+ * values or reordering existing ones, ensure to review and potentially adjust
+ * the corresponding statistics collection codes.
*/
typedef enum
{
@@ -39,6 +44,8 @@ typedef enum
CT_DELETE_DIFFER,
} ConflictType;
+#define CONFLICT_NUM_TYPES (CT_DELETE_DIFFER + 1)
+
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
extern void ReportApplyConflict(int elevel, ConflictType type,
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 4c789279e5..3fa03b4d76 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2139,9 +2139,15 @@ pg_stat_subscription_stats| SELECT ss.subid,
s.subname,
ss.apply_error_count,
ss.sync_error_count,
+ ss.insert_exists_count,
+ ss.update_exists_count,
+ ss.update_differ_count,
+ ss.update_missing_count,
+ ss.delete_missing_count,
+ ss.delete_differ_count,
ss.stats_reset
FROM pg_subscription s,
- LATERAL pg_stat_get_subscription_stats(s.oid) ss(subid, apply_error_count, sync_error_count, stats_reset);
+ LATERAL pg_stat_get_subscription_stats(s.oid) ss(subid, apply_error_count, sync_error_count, insert_exists_count, update_exists_count, update_differ_count, update_missing_count, delete_missing_count, delete_differ_count, stats_reset);
pg_stat_sys_indexes| SELECT relid,
indexrelid,
schemaname,
diff --git a/src/test/subscription/t/026_stats.pl b/src/test/subscription/t/026_stats.pl
index fb3e5629b3..56eafa5ba6 100644
--- a/src/test/subscription/t/026_stats.pl
+++ b/src/test/subscription/t/026_stats.pl
@@ -16,6 +16,7 @@ $node_publisher->start;
# Create subscriber node.
my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf', 'track_commit_timestamp = on');
$node_subscriber->start;
@@ -30,6 +31,7 @@ sub create_sub_pub_w_errors
qq[
BEGIN;
CREATE TABLE $table_name(a int);
+ ALTER TABLE $table_name REPLICA IDENTITY FULL;
INSERT INTO $table_name VALUES (1);
COMMIT;
]);
@@ -53,7 +55,7 @@ sub create_sub_pub_w_errors
# infinite error loop due to violating the unique constraint.
my $sub_name = $table_name . '_sub';
$node_subscriber->safe_psql($db,
- qq(CREATE SUBSCRIPTION $sub_name CONNECTION '$publisher_connstr' PUBLICATION $pub_name)
+ qq(CREATE SUBSCRIPTION $sub_name CONNECTION '$publisher_connstr' PUBLICATION $pub_name WITH (detect_conflict = on))
);
$node_publisher->wait_for_catchup($sub_name);
@@ -95,7 +97,7 @@ sub create_sub_pub_w_errors
$node_subscriber->poll_query_until(
$db,
qq[
- SELECT apply_error_count > 0
+ SELECT apply_error_count > 0 AND insert_exists_count > 0
FROM pg_stat_subscription_stats
WHERE subname = '$sub_name'
])
@@ -105,6 +107,85 @@ sub create_sub_pub_w_errors
# Truncate test table so that apply worker can continue.
$node_subscriber->safe_psql($db, qq(TRUNCATE $table_name));
+ # Insert a row on the subscriber.
+ $node_subscriber->safe_psql($db, qq(INSERT INTO $table_name VALUES (2)));
+
+ # Update data from test table on the publisher, raising an error on the
+ # subscriber due to violation of the unique constraint on test table.
+ $node_publisher->safe_psql($db, qq(UPDATE $table_name SET a = 2;));
+
+ # Wait for the apply error to be reported.
+ $node_subscriber->poll_query_until(
+ $db,
+ qq[
+ SELECT update_exists_count > 0
+ FROM pg_stat_subscription_stats
+ WHERE subname = '$sub_name'
+ ])
+ or die
+ qq(Timed out while waiting for update_exists conflict for subscription '$sub_name');
+
+ # Truncate test table so that the update will be skipped and the test can
+ # continue.
+ $node_subscriber->safe_psql($db, qq(TRUNCATE $table_name));
+
+ # delete the data from test table on the publisher. The delete should be
+ # skipped on the subscriber as there are no data in the test table.
+ $node_publisher->safe_psql($db, qq(DELETE FROM $table_name;));
+
+ # Wait for the tuple missing to be reported.
+ $node_subscriber->poll_query_until(
+ $db,
+ qq[
+ SELECT update_missing_count > 0 AND delete_missing_count > 0
+ FROM pg_stat_subscription_stats
+ WHERE subname = '$sub_name'
+ ])
+ or die
+ qq(Timed out while waiting for tuple missing conflict for subscription '$sub_name');
+
+ # Prepare data for further tests.
+ $node_publisher->safe_psql($db, qq(INSERT INTO $table_name VALUES (1)));
+ $node_publisher->wait_for_catchup($sub_name);
+ $node_subscriber->safe_psql($db, qq(
+ TRUNCATE $table_name;
+ INSERT INTO $table_name VALUES (1);
+ ));
+
+ # Update data from test table on the publisher, updating a row on the
+ # subscriber that was modified by a different origin.
+ $node_publisher->safe_psql($db, qq(UPDATE $table_name SET a = 2;));
+
+ $node_subscriber->poll_query_until(
+ $db,
+ qq[
+ SELECT update_differ_count > 0
+ FROM pg_stat_subscription_stats
+ WHERE subname = '$sub_name'
+ ])
+ or die
+ qq(Timed out while waiting for update_differ conflict for subscription '$sub_name');
+
+ # Prepare data for further tests.
+ $node_subscriber->safe_psql($db, qq(
+ TRUNCATE $table_name;
+ INSERT INTO $table_name VALUES (2);
+ ));
+
+ # Delete data to test table on the publisher, deleting a row on the
+ # subscriber that was modified by a different origin.
+ $node_publisher->safe_psql($db, qq(DELETE FROM $table_name;));
+
+ $node_subscriber->poll_query_until(
+ $db,
+ qq[
+ SELECT delete_differ_count > 0
+ FROM pg_stat_subscription_stats
+ WHERE subname = '$sub_name'
+ ])
+ or die
+ qq(Timed out while waiting for delete_differ conflict for subscription '$sub_name');
+
return ($pub_name, $sub_name);
}
@@ -128,11 +209,17 @@ is( $node_subscriber->safe_psql(
$db,
qq(SELECT apply_error_count > 0,
sync_error_count > 0,
+ insert_exists_count > 0,
+ update_exists_count > 0,
+ update_differ_count > 0,
+ update_missing_count > 0,
+ delete_missing_count > 0,
+ delete_differ_count > 0,
stats_reset IS NULL
FROM pg_stat_subscription_stats
WHERE subname = '$sub1_name')
),
- qq(t|t|t),
+ qq(t|t|t|t|t|t|t|t|t),
qq(Check that apply errors and sync errors are both > 0 and stats_reset is NULL for subscription '$sub1_name'.)
);
@@ -146,11 +233,17 @@ is( $node_subscriber->safe_psql(
$db,
qq(SELECT apply_error_count = 0,
sync_error_count = 0,
+ insert_exists_count = 0,
+ update_exists_count = 0,
+ update_differ_count = 0,
+ update_missing_count = 0,
+ delete_missing_count = 0,
+ delete_differ_count = 0,
stats_reset IS NOT NULL
FROM pg_stat_subscription_stats
WHERE subname = '$sub1_name')
),
- qq(t|t|t),
+ qq(t|t|t|t|t|t|t|t|t),
qq(Confirm that apply errors and sync errors are both 0 and stats_reset is not NULL after reset for subscription '$sub1_name'.)
);
@@ -186,11 +279,17 @@ is( $node_subscriber->safe_psql(
$db,
qq(SELECT apply_error_count > 0,
sync_error_count > 0,
+ insert_exists_count > 0,
+ update_exists_count > 0,
+ update_differ_count > 0,
+ update_missing_count > 0,
+ delete_missing_count > 0,
+ delete_differ_count > 0,
stats_reset IS NULL
FROM pg_stat_subscription_stats
WHERE subname = '$sub2_name')
),
- qq(t|t|t),
+ qq(t|t|t|t|t|t|t|t|t),
qq(Confirm that apply errors and sync errors are both > 0 and stats_reset is NULL for sub '$sub2_name'.)
);
@@ -203,11 +302,17 @@ is( $node_subscriber->safe_psql(
$db,
qq(SELECT apply_error_count = 0,
sync_error_count = 0,
+ insert_exists_count = 0,
+ update_exists_count = 0,
+ update_differ_count = 0,
+ update_missing_count = 0,
+ delete_missing_count = 0,
+ delete_differ_count = 0,
stats_reset IS NOT NULL
FROM pg_stat_subscription_stats
WHERE subname = '$sub1_name')
),
- qq(t|t|t),
+ qq(t|t|t|t|t|t|t|t|t),
qq(Confirm that apply errors and sync errors are both 0 and stats_reset is not NULL for sub '$sub1_name' after reset.)
);
@@ -215,11 +320,17 @@ is( $node_subscriber->safe_psql(
$db,
qq(SELECT apply_error_count = 0,
sync_error_count = 0,
+ insert_exists_count = 0,
+ update_exists_count = 0,
+ update_differ_count = 0,
+ update_missing_count = 0,
+ delete_missing_count = 0,
+ delete_differ_count = 0,
stats_reset IS NOT NULL
FROM pg_stat_subscription_stats
WHERE subname = '$sub2_name')
),
- qq(t|t|t),
+ qq(t|t|t|t|t|t|t|t|t),
qq(Confirm that apply errors and sync errors are both 0 and stats_reset is not NULL for sub '$sub2_name' after reset.)
);
--
2.30.0.windows.2
[application/octet-stream] v6-0001-Detect-and-log-conflicts-in-logical-replication.patch (106.4K, ../../TYAPR01MB5724B31BE097CFECF8A7D0E794AB2@TYAPR01MB5724.jpnprd01.prod.outlook.com/3-v6-0001-Detect-and-log-conflicts-in-logical-replication.patch)
download | inline diff:
From 9b1e0e2442e6a46ef8d910dda98658bcd197ad5c Mon Sep 17 00:00:00 2001
From: Hou Zhijie <[email protected]>
Date: Thu, 25 Jul 2024 10:06:52 +0800
Subject: [PATCH v6 1/2] Detect and log conflicts in logical replication
This patch adds a new parameter detect_conflict for CREATE and ALTER
subscription commands. This new parameter will decide if subscription will
go for confict detection. By default, conflict detection will be off for a
subscription.
When conflict detection is enabled, additional logging is triggered in the
following conflict scenarios:
insert_exists: Inserting a row that violates a NOT DEFERRABLE unique constraint.
update_exists: The updated row value violates a NOT DEFERRABLE unique constraint.
update_differ: Updating a row that was previously modified by another origin.
update_missing: The tuple to be updated is missing.
delete_missing: The tuple to be deleted is missing.
delete_differ: Deleting a row that was previously modified by another origin.
For insert_exists and update_exists conflicts, the log can include origin
and commit timestamp details of the conflicting key with track_commit_timestamp
enabled.
update_differ and delete_differ conflicts can only be detected when
track_commit_timestamp is enabled.
---
doc/src/sgml/catalogs.sgml | 9 +
doc/src/sgml/logical-replication.sgml | 21 +-
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 84 ++++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/subscriptioncmds.c | 54 ++++-
src/backend/executor/execIndexing.c | 14 +-
src/backend/executor/execReplication.c | 221 ++++++++++++++------
src/backend/replication/logical/Makefile | 1 +
src/backend/replication/logical/conflict.c | 193 +++++++++++++++++
src/backend/replication/logical/meson.build | 1 +
src/backend/replication/logical/worker.c | 91 ++++++--
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 6 +-
src/bin/psql/tab-complete.c | 14 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/conflict.h | 50 +++++
src/test/regress/expected/subscription.out | 188 ++++++++++-------
src/test/regress/sql/subscription.sql | 19 ++
src/test/subscription/t/001_rep_changes.pl | 15 +-
src/test/subscription/t/013_partition.pl | 68 +++---
src/test/subscription/t/029_on_error.pl | 13 +-
src/test/subscription/t/030_origin.pl | 43 ++++
src/tools/pgindent/typedefs.list | 1 +
26 files changed, 921 insertions(+), 216 deletions(-)
create mode 100644 src/backend/replication/logical/conflict.c
create mode 100644 src/include/replication/conflict.h
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index b654fae1b2..b042a5a94a 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8035,6 +8035,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdetectconflict</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription is enabled for conflict detection.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index a23a3d57e2..d236d8530c 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1580,7 +1580,8 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
stop. This is referred to as a <firstterm>conflict</firstterm>. When
replicating <command>UPDATE</command> or <command>DELETE</command>
operations, missing data will not produce a conflict and such operations
- will simply be skipped.
+ will simply be skipped. Please refer to <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ for all the conflicts that will be logged when enabling <literal>detect_conflict</literal>.
</para>
<para>
@@ -1649,6 +1650,24 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ...
SKIP</command></link>.
</para>
+
+ <para>
+ Enabling both <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ and <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ on the subscriber can provide additional details regarding conflicting
+ rows, such as their origin and commit timestamp, in case of a unique
+ constraint violation conflict:
+<screen>
+ERROR: conflict insert_exists detected on relation "public.t"
+DETAIL: Key (a)=(1) already exists in unique index "t_pkey", which was modified by origin 0 in transaction 740 at 2024-06-26 10:47:04.727375+08.
+CONTEXT: processing remote data for replication origin "pg_16389" during message type "INSERT" for replication target relation "public.t" in transaction 740, finished at 0/14F7EC0
+</screen>
+ Users can use these information to make decisions on whether to retain
+ the local change or adopt the remote alteration. For instance, the
+ origin in above log indicates that the existing row was modified by a
+ local change, users can manually perform a remote-change-win resolution
+ by deleting the local row.
+ </para>
</sect1>
<sect1 id="logical-replication-restrictions">
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index fdc648d007..dfbe25b59e 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -235,8 +235,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<link linkend="sql-createsubscription-params-with-password-required"><literal>password_required</literal></link>,
<link linkend="sql-createsubscription-params-with-run-as-owner"><literal>run_as_owner</literal></link>,
<link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>,
- <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>, and
- <link linkend="sql-createsubscription-params-with-two-phase"><literal>two_phase</literal></link>.
+ <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>,
+ <link linkend="sql-createsubscription-params-with-two-phase"><literal>two_phase</literal></link>, and
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>.
Only a superuser can set <literal>password_required = false</literal>.
</para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..eb51805e81 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,90 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-detect-conflict">
+ <term><literal>detect_conflict</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription is enabled for conflict detection.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ When conflict detection is enabled, additional logging is triggered
+ in the following scenarios:
+ <variablelist>
+ <varlistentry>
+ <term><literal>insert_exists</literal></term>
+ <listitem>
+ <para>
+ Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint. Note that to obtain the origin and commit
+ timestamp details of the conflicting key in the log, ensure that
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. In this scenario, an error will be raised until the
+ conflict is resolved manually.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_exists</literal></term>
+ <listitem>
+ <para>
+ The updated value of a row violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint. Note that to obtain the origin and commit
+ timestamp details of the conflicting key in the log, ensure that
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. In this scenario, an error will be raised until the
+ conflict is resolved manually.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_differ</literal></term>
+ <listitem>
+ <para>
+ Updating a row that was previously modified by another origin.
+ Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. Currenly, the update is always applied regardless of
+ the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be updated was not found. The update will simply be
+ skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be deleted was not found. The delete will simply be
+ skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_differ</literal></term>
+ <listitem>
+ <para>
+ Deleting a row that was previously modified by another origin. Note that this
+ conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. Currenly, the delete is always applied regardless of
+ the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..5a423f4fb0 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->detectconflict = subform->subdetectconflict;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 19cabc9a47..d084bfc48a 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1356,7 +1356,8 @@ REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
subbinary, substream, subtwophasestate, subdisableonerr,
subpasswordrequired, subrunasowner, subfailover,
- subslotname, subsynccommit, subpublications, suborigin)
+ subdetectconflict, subslotname, subsynccommit,
+ subpublications, suborigin)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_stats AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index d124bfe55c..24f9430fb1 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -14,6 +14,7 @@
#include "postgres.h"
+#include "access/commit_ts.h"
#include "access/htup_details.h"
#include "access/table.h"
#include "access/twophase.h"
@@ -71,8 +72,9 @@
#define SUBOPT_PASSWORD_REQUIRED 0x00000800
#define SUBOPT_RUN_AS_OWNER 0x00001000
#define SUBOPT_FAILOVER 0x00002000
-#define SUBOPT_LSN 0x00004000
-#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_DETECT_CONFLICT 0x00004000
+#define SUBOPT_LSN 0x00008000
+#define SUBOPT_ORIGIN 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -98,6 +100,7 @@ typedef struct SubOpts
bool passwordrequired;
bool runasowner;
bool failover;
+ bool detectconflict;
char *origin;
XLogRecPtr lsn;
} SubOpts;
@@ -112,7 +115,7 @@ static List *merge_publications(List *oldpublist, List *newpublist, bool addpub,
static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err);
static void CheckAlterSubOption(Subscription *sub, const char *option,
bool slot_needs_update, bool isTopLevel);
-
+static void check_conflict_detection(void);
/*
* Common option parsing function for CREATE and ALTER SUBSCRIPTION commands.
@@ -162,6 +165,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->runasowner = false;
if (IsSet(supported_opts, SUBOPT_FAILOVER))
opts->failover = false;
+ if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT))
+ opts->detectconflict = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
@@ -307,6 +312,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_FAILOVER;
opts->failover = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT) &&
+ strcmp(defel->defname, "detect_conflict") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DETECT_CONFLICT))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DETECT_CONFLICT;
+ opts->detectconflict = defGetBoolean(defel);
+ }
else if (IsSet(supported_opts, SUBOPT_ORIGIN) &&
strcmp(defel->defname, "origin") == 0)
{
@@ -594,7 +608,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -639,6 +654,9 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
errmsg("password_required=false is superuser-only"),
errhint("Subscriptions with the password_required option set to false may only be created or modified by the superuser.")));
+ if (opts.detectconflict)
+ check_conflict_detection();
+
/*
* If built with appropriate switch, whine when regression-testing
* conventions for subscription names are violated.
@@ -701,6 +719,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_subpasswordrequired - 1] = BoolGetDatum(opts.passwordrequired);
values[Anum_pg_subscription_subrunasowner - 1] = BoolGetDatum(opts.runasowner);
values[Anum_pg_subscription_subfailover - 1] = BoolGetDatum(opts.failover);
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -1196,7 +1216,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
SUBOPT_DISABLE_ON_ERR |
SUBOPT_PASSWORD_REQUIRED |
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
- SUBOPT_ORIGIN);
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1356,6 +1376,16 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_subfailover - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DETECT_CONFLICT))
+ {
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
+ replaces[Anum_pg_subscription_subdetectconflict - 1] = true;
+
+ if (opts.detectconflict)
+ check_conflict_detection();
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
{
values[Anum_pg_subscription_suborigin - 1] =
@@ -2536,3 +2566,17 @@ defGetStreamingMode(DefElem *def)
def->defname)));
return LOGICALREP_STREAM_OFF; /* keep compiler quiet */
}
+
+/*
+ * Report a warning about incomplete conflict detection if
+ * track_commit_timestamp is disabled.
+ */
+static void
+check_conflict_detection(void)
+{
+ if (!track_commit_timestamp)
+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("conflict detection could be incomplete due to disabled track_commit_timestamp"),
+ errdetail("Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged."));
+}
diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c
index 9f05b3654c..ef522778a2 100644
--- a/src/backend/executor/execIndexing.c
+++ b/src/backend/executor/execIndexing.c
@@ -207,8 +207,9 @@ ExecOpenIndices(ResultRelInfo *resultRelInfo, bool speculative)
ii = BuildIndexInfo(indexDesc);
/*
- * If the indexes are to be used for speculative insertion, add extra
- * information required by unique index entries.
+ * If the indexes are to be used for speculative insertion or conflict
+ * detection in logical replication, add extra information required by
+ * unique index entries.
*/
if (speculative && ii->ii_Unique)
BuildSpeculativeIndexInfo(indexDesc, ii);
@@ -521,6 +522,11 @@ ExecInsertIndexTuples(ResultRelInfo *resultRelInfo,
* possible that a conflicting tuple is inserted immediately
* after this returns. But this can be used for a pre-check
* before insertion.
+ *
+ * If the 'slot' holds a tuple with valid tid, this tuple will
+ * be ignored when checking conflict. This can help in scenarios
+ * where we want to re-check for conflicts after inserting a
+ * tuple.
* ----------------------------------------------------------------
*/
bool
@@ -536,11 +542,9 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
ExprContext *econtext;
Datum values[INDEX_MAX_KEYS];
bool isnull[INDEX_MAX_KEYS];
- ItemPointerData invalidItemPtr;
bool checkedIndex = false;
ItemPointerSetInvalid(conflictTid);
- ItemPointerSetInvalid(&invalidItemPtr);
/*
* Get information from the result relation info structure.
@@ -629,7 +633,7 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
satisfiesConstraint =
check_exclusion_or_unique_constraint(heapRelation, indexRelation,
- indexInfo, &invalidItemPtr,
+ indexInfo, &slot->tts_tid,
values, isnull, estate, false,
CEOUC_WAIT, true,
conflictTid);
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index d0a89cd577..f4e3d2aa09 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -23,6 +23,7 @@
#include "commands/trigger.h"
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
+#include "replication/conflict.h"
#include "replication/logicalrelation.h"
#include "storage/lmgr.h"
#include "utils/builtins.h"
@@ -166,6 +167,51 @@ build_replindex_scan_key(ScanKey skey, Relation rel, Relation idxrel,
return skey_attoff;
}
+
+/*
+ * Helper function to check if it is necessary to re-fetch and lock the tuple
+ * due to concurrent modifications. This function should be called after
+ * invoking table_tuple_lock.
+ */
+static bool
+should_refetch_tuple(TM_Result res, TM_FailureData *tmfd)
+{
+ bool refetch = false;
+
+ switch (res)
+ {
+ case TM_Ok:
+ break;
+ case TM_Updated:
+ /* XXX: Improve handling here */
+ if (ItemPointerIndicatesMovedPartitions(&tmfd->ctid))
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("tuple to be locked was already moved to another partition due to concurrent update, retrying")));
+ else
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent update, retrying")));
+ refetch = true;
+ break;
+ case TM_Deleted:
+ /* XXX: Improve handling here */
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent delete, retrying")));
+ refetch = true;
+ break;
+ case TM_Invisible:
+ elog(ERROR, "attempted to lock invisible tuple");
+ break;
+ default:
+ elog(ERROR, "unexpected table_tuple_lock status: %u", res);
+ break;
+ }
+
+ return refetch;
+}
+
/*
* Search the relation 'rel' for tuple using the index.
*
@@ -260,34 +306,8 @@ retry:
PopActiveSnapshot();
- switch (res)
- {
- case TM_Ok:
- break;
- case TM_Updated:
- /* XXX: Improve handling here */
- if (ItemPointerIndicatesMovedPartitions(&tmfd.ctid))
- ereport(LOG,
- (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
- errmsg("tuple to be locked was already moved to another partition due to concurrent update, retrying")));
- else
- ereport(LOG,
- (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
- errmsg("concurrent update, retrying")));
- goto retry;
- case TM_Deleted:
- /* XXX: Improve handling here */
- ereport(LOG,
- (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
- errmsg("concurrent delete, retrying")));
- goto retry;
- case TM_Invisible:
- elog(ERROR, "attempted to lock invisible tuple");
- break;
- default:
- elog(ERROR, "unexpected table_tuple_lock status: %u", res);
- break;
- }
+ if (should_refetch_tuple(res, &tmfd))
+ goto retry;
}
index_endscan(scan);
@@ -444,34 +464,8 @@ retry:
PopActiveSnapshot();
- switch (res)
- {
- case TM_Ok:
- break;
- case TM_Updated:
- /* XXX: Improve handling here */
- if (ItemPointerIndicatesMovedPartitions(&tmfd.ctid))
- ereport(LOG,
- (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
- errmsg("tuple to be locked was already moved to another partition due to concurrent update, retrying")));
- else
- ereport(LOG,
- (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
- errmsg("concurrent update, retrying")));
- goto retry;
- case TM_Deleted:
- /* XXX: Improve handling here */
- ereport(LOG,
- (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
- errmsg("concurrent delete, retrying")));
- goto retry;
- case TM_Invisible:
- elog(ERROR, "attempted to lock invisible tuple");
- break;
- default:
- elog(ERROR, "unexpected table_tuple_lock status: %u", res);
- break;
- }
+ if (should_refetch_tuple(res, &tmfd))
+ goto retry;
}
table_endscan(scan);
@@ -480,6 +474,95 @@ retry:
return found;
}
+/*
+ * Find the tuple that violates the passed in unique index constraint
+ * (conflictindex).
+ *
+ * If no conflict is found, return false and set *conflictslot to NULL.
+ * Otherwise return true, and the conflicting tuple is locked and returned in
+ * *conflictslot.
+ */
+static bool
+FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
+ Oid conflictindex, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot)
+{
+ Relation rel = resultRelInfo->ri_RelationDesc;
+ ItemPointerData conflictTid;
+ TM_FailureData tmfd;
+ TM_Result res;
+
+ *conflictslot = NULL;
+
+retry:
+ if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
+ &conflictTid, list_make1_oid(conflictindex)))
+ {
+ if (*conflictslot)
+ ExecDropSingleTupleTableSlot(*conflictslot);
+
+ *conflictslot = NULL;
+ return false;
+ }
+
+ *conflictslot = table_slot_create(rel, NULL);
+
+ PushActiveSnapshot(GetLatestSnapshot());
+
+ res = table_tuple_lock(rel, &conflictTid, GetLatestSnapshot(),
+ *conflictslot,
+ GetCurrentCommandId(false),
+ LockTupleShare,
+ LockWaitBlock,
+ 0 /* don't follow updates */ ,
+ &tmfd);
+
+ PopActiveSnapshot();
+
+ if (should_refetch_tuple(res, &tmfd))
+ goto retry;
+
+ return true;
+}
+
+/*
+ * Re-check all the unique indexes in 'recheckIndexes' to see if there are
+ * potential conflicts with the tuple in 'slot'.
+ *
+ * This function is invoked after inserting or updating a tuple that detected
+ * potential conflict tuples. It attempts to find the tuple that conflicts with
+ * the provided tuple. This operation may seem redundant with the unique
+ * violation check of indexam, but since we call this function only when we are
+ * detecting conflict in logical replication and encountering potential
+ * conflicts with any unique index constraints (which should not be frequent),
+ * so it's ok. Moreover, upon detecting a conflict, we will report an ERROR and
+ * restart the logical replication, so the additional cost of finding the tuple
+ * should be acceptable.
+ */
+static void
+ReCheckConflictIndexes(ResultRelInfo *resultRelInfo, EState *estate,
+ ConflictType type, List *recheckIndexes,
+ TupleTableSlot *slot)
+{
+ /* Re-check all the unique indexes for potential conflicts */
+ foreach_oid(uniqueidx, resultRelInfo->ri_onConflictArbiterIndexes)
+ {
+ TupleTableSlot *conflictslot;
+
+ if (list_member_oid(recheckIndexes, uniqueidx) &&
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(ERROR, type, resultRelInfo->ri_RelationDesc, uniqueidx,
+ xmin, origin, committs, conflictslot);
+ }
+ }
+}
+
/*
* Insert tuple represented in the slot to the relation, update the indexes,
* and execute any constraints and per-row triggers.
@@ -509,6 +592,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (!skip_tuple)
{
List *recheckIndexes = NIL;
+ List *conflictindexes;
+ bool conflict = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -525,10 +610,17 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
- slot, estate, false, false,
- NULL, NIL, false);
+ slot, estate, false,
+ conflictindexes, &conflict,
+ conflictindexes, false);
+
+ if (conflict)
+ ReCheckConflictIndexes(resultRelInfo, estate, CT_INSERT_EXISTS,
+ recheckIndexes, slot);
/* AFTER ROW INSERT Triggers */
ExecARInsertTriggers(estate, resultRelInfo, slot,
@@ -577,6 +669,8 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
{
List *recheckIndexes = NIL;
TU_UpdateIndexes update_indexes;
+ List *conflictindexes;
+ bool conflict = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -593,12 +687,19 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
simple_table_tuple_update(rel, tid, slot, estate->es_snapshot,
&update_indexes);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
if (resultRelInfo->ri_NumIndices > 0 && (update_indexes != TU_None))
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
- slot, estate, true, false,
- NULL, NIL,
+ slot, estate, true,
+ conflictindexes,
+ &conflict, conflictindexes,
(update_indexes == TU_Summarizing));
+ if (conflict)
+ ReCheckConflictIndexes(resultRelInfo, estate, CT_UPDATE_EXISTS,
+ recheckIndexes, slot);
+
/* AFTER ROW UPDATE Triggers */
ExecARUpdateTriggers(estate, resultRelInfo,
NULL, NULL,
diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile
index ba03eeff1c..1e08bbbd4e 100644
--- a/src/backend/replication/logical/Makefile
+++ b/src/backend/replication/logical/Makefile
@@ -16,6 +16,7 @@ override CPPFLAGS := -I$(srcdir) $(CPPFLAGS)
OBJS = \
applyparallelworker.o \
+ conflict.o \
decode.o \
launcher.o \
logical.o \
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
new file mode 100644
index 0000000000..4918011a7f
--- /dev/null
+++ b/src/backend/replication/logical/conflict.c
@@ -0,0 +1,193 @@
+/*-------------------------------------------------------------------------
+ * conflict.c
+ * Functionality for detecting and logging conflicts.
+ *
+ * Copyright (c) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/replication/logical/conflict.c
+ *
+ * This file contains the code for detecting and logging conflicts on
+ * the subscriber during logical replication.
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/commit_ts.h"
+#include "replication/conflict.h"
+#include "replication/origin.h"
+#include "utils/lsyscache.h"
+#include "utils/rel.h"
+
+const char *const ConflictTypeNames[] = {
+ [CT_INSERT_EXISTS] = "insert_exists",
+ [CT_UPDATE_EXISTS] = "update_exists",
+ [CT_UPDATE_DIFFER] = "update_differ",
+ [CT_UPDATE_MISSING] = "update_missing",
+ [CT_DELETE_MISSING] = "delete_missing",
+ [CT_DELETE_DIFFER] = "delete_differ"
+};
+
+static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
+static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin,
+ RepOriginId localorigin,
+ TimestampTz localts,
+ TupleTableSlot *conflictslot);
+
+/*
+ * Get the xmin and commit timestamp data (origin and timestamp) associated
+ * with the provided local tuple.
+ *
+ * Return true if the commit timestamp data was found, false otherwise.
+ */
+bool
+GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts)
+{
+ Datum xminDatum;
+ bool isnull;
+
+ xminDatum = slot_getsysattr(localslot, MinTransactionIdAttributeNumber,
+ &isnull);
+ *xmin = DatumGetTransactionId(xminDatum);
+ Assert(!isnull);
+
+ /*
+ * The commit timestamp data is not available if track_commit_timestamp is
+ * disabled.
+ */
+ if (!track_commit_timestamp)
+ {
+ *localorigin = InvalidRepOriginId;
+ *localts = 0;
+ return false;
+ }
+
+ return TransactionIdGetCommitTsData(*xmin, localts, localorigin);
+}
+
+/*
+ * Report a conflict when applying remote changes.
+ */
+void
+ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot)
+{
+ ereport(elevel,
+ errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
+ errmsg("conflict %s detected on relation \"%s.%s\"",
+ ConflictTypeNames[type],
+ get_namespace_name(RelationGetNamespace(localrel)),
+ RelationGetRelationName(localrel)),
+ errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
+ localts, conflictslot));
+}
+
+/*
+ * Find all unique indexes to check for a conflict and store them into
+ * ResultRelInfo.
+ */
+void
+InitConflictIndexes(ResultRelInfo *relInfo)
+{
+ List *uniqueIndexes = NIL;
+
+ for (int i = 0; i < relInfo->ri_NumIndices; i++)
+ {
+ Relation indexRelation = relInfo->ri_IndexRelationDescs[i];
+
+ if (indexRelation == NULL)
+ continue;
+
+ /* Detect conflict only for unique indexes */
+ if (!relInfo->ri_IndexRelationInfo[i]->ii_Unique)
+ continue;
+
+ /* Don't support conflict detection for deferrable index */
+ if (!indexRelation->rd_index->indimmediate)
+ continue;
+
+ uniqueIndexes = lappend_oid(uniqueIndexes,
+ RelationGetRelid(indexRelation));
+ }
+
+ relInfo->ri_onConflictArbiterIndexes = uniqueIndexes;
+}
+
+/*
+ * Add an errdetail() line showing conflict detail.
+ */
+static int
+errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot)
+{
+ switch (type)
+ {
+ case CT_INSERT_EXISTS:
+ case CT_UPDATE_EXISTS:
+ {
+ /*
+ * Bulid the index value string. If the return value is NULL,
+ * it indicates that the current user lacks permissions to
+ * view all the columns involved.
+ */
+ char *index_value = build_index_value_desc(conflictidx,
+ conflictslot);
+
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
+ case CT_UPDATE_DIFFER:
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ case CT_UPDATE_MISSING:
+ return errdetail("Did not find the row to be updated.");
+ case CT_DELETE_MISSING:
+ return errdetail("Did not find the row to be deleted.");
+ case CT_DELETE_DIFFER:
+ return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ }
+
+ return 0; /* silence compiler warning */
+}
+
+/*
+ * Helper functions to construct a string describing the contents of an index
+ * entry. See BuildIndexValueDescription for details.
+ */
+static char *
+build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
+{
+ char *conflict_row;
+ Relation indexDesc;
+
+ if (!conflictslot)
+ return NULL;
+
+ /* Assume the index has been locked */
+ indexDesc = index_open(indexoid, NoLock);
+
+ slot_getallattrs(conflictslot);
+
+ conflict_row = BuildIndexValueDescription(indexDesc,
+ conflictslot->tts_values,
+ conflictslot->tts_isnull);
+
+ index_close(indexDesc, NoLock);
+
+ return conflict_row;
+}
diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build
index 3dec36a6de..3d36249d8a 100644
--- a/src/backend/replication/logical/meson.build
+++ b/src/backend/replication/logical/meson.build
@@ -2,6 +2,7 @@
backend_sources += files(
'applyparallelworker.c',
+ 'conflict.c',
'decode.c',
'launcher.c',
'logical.c',
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ec96b5fe85..1008510240 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -167,6 +167,7 @@
#include "postmaster/bgworker.h"
#include "postmaster/interrupt.h"
#include "postmaster/walwriter.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalproto.h"
#include "replication/logicalrelation.h"
@@ -2458,7 +2459,10 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
EState *estate = edata->estate;
/* We must open indexes here. */
- ExecOpenIndices(relinfo, false);
+ ExecOpenIndices(relinfo, MySubscription->detectconflict);
+
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
@@ -2646,7 +2650,7 @@ apply_handle_update_internal(ApplyExecutionData *edata,
MemoryContext oldctx;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
- ExecOpenIndices(relinfo, false);
+ ExecOpenIndices(relinfo, MySubscription->detectconflict);
found = FindReplTupleInLocalRel(edata, localrel,
&relmapentry->remoterel,
@@ -2661,6 +2665,20 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2668,6 +2686,9 @@ apply_handle_update_internal(ApplyExecutionData *edata,
EvalPlanQualSetSlot(&epqstate, remoteslot);
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
+
/* Do the actual update. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
@@ -2678,13 +2699,10 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2807,6 +2825,25 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/* If found delete it. */
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
+ *
+ * For cross-partition update, we skip detecting the delete_differ
+ * conflict since it should have been done in
+ * apply_handle_tuple_routing().
+ */
+ if (MySubscription->detectconflict &&
+ (!edata->mtstate || edata->mtstate->operation != CMD_UPDATE) &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_DELETE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
EvalPlanQualSetSlot(&epqstate, localslot);
/* Do the actual delete. */
@@ -2818,13 +2855,10 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/*
* The tuple to be deleted could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be deleted "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2991,6 +3025,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ResultRelInfo *partrelinfo_new;
Relation partrel_new;
bool found;
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3002,16 +3039,28 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing
* except for emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation's partition \"%s\"",
- RelationGetRelationName(partrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING,
+ partrel, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+
return;
}
+ /*
+ * If conflict detection is enabled, check whether the local
+ * tuple was modified by a different origin. If detected,
+ * report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, partrel,
+ InvalidOid, localxmin, localorigin,
+ localts, NULL);
+
/*
* Apply the update to the local tuple, putting the result in
* remoteslot_part.
@@ -3039,7 +3088,7 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
EPQState epqstate;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
- ExecOpenIndices(partrelinfo, false);
+ ExecOpenIndices(partrelinfo, MySubscription->detectconflict);
EvalPlanQualSetSlot(&epqstate, remoteslot_part);
TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index b8b1888bd3..829f9b3e88 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4760,6 +4760,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subdetectconflict;
int i,
ntups;
@@ -4832,11 +4833,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subdetectconflict\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdetectconflict\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4875,6 +4882,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subdetectconflict = PQfnumber(res, "subdetectconflict");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4921,6 +4929,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subdetectconflict =
+ pg_strdup(PQgetvalue(res, i, i_subdetectconflict));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5161,6 +5171,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subfailover, "t") == 0)
appendPQExpBufferStr(query, ", failover = true");
+ if (strcmp(subinfo->subdetectconflict, "t") == 0)
+ appendPQExpBufferStr(query, ", detect_conflict = true");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e5870a9..bbd7cbeff6 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subdetectconflict;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f234c..fef1ad0d70 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6607,6 +6607,10 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subdetectconflict AS \"%s\"\n",
+ gettext_noop("Detect conflict"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 891face1b6..086e135f65 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1946,9 +1946,10 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "disable_on_error", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit", "two_phase");
+ COMPLETE_WITH("binary", "detect_conflict", "disable_on_error",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit", "two_phase");
/* ALTER SUBSCRIPTION <name> SKIP ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SKIP", "("))
COMPLETE_WITH("lsn");
@@ -3363,9 +3364,10 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit", "two_phase");
+ "detect_conflict", "disable_on_error", "enabled",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit", "two_phase");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..17daf11dc7 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subdetectconflict; /* True if replication should perform
+ * conflict detection */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -151,6 +154,7 @@ typedef struct Subscription
* (i.e. the main slot and the table sync
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool detectconflict; /* True if conflict detection is enabled */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
new file mode 100644
index 0000000000..3a7260d3c1
--- /dev/null
+++ b/src/include/replication/conflict.h
@@ -0,0 +1,50 @@
+/*-------------------------------------------------------------------------
+ * conflict.h
+ * Exports for conflict detection and log
+ *
+ * Copyright (c) 2012-2024, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef CONFLICT_H
+#define CONFLICT_H
+
+#include "access/xlogdefs.h"
+#include "executor/tuptable.h"
+#include "nodes/execnodes.h"
+#include "utils/relcache.h"
+#include "utils/timestamp.h"
+
+/*
+ * Conflict types that could be encountered when applying remote changes.
+ */
+typedef enum
+{
+ /* The row to be inserted violates unique constraint */
+ CT_INSERT_EXISTS,
+
+ /* The updated row value violates unique constraint */
+ CT_UPDATE_EXISTS,
+
+ /* The row to be updated was modified by a different origin */
+ CT_UPDATE_DIFFER,
+
+ /* The row to be updated is missing */
+ CT_UPDATE_MISSING,
+
+ /* The row to be deleted is missing */
+ CT_DELETE_MISSING,
+
+ /* The row to be deleted was modified by a different origin */
+ CT_DELETE_DIFFER,
+} ConflictType;
+
+extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts);
+extern void ReportApplyConflict(int elevel, ConflictType type,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot);
+extern void InitConflictIndexes(ResultRelInfo *relInfo);
+
+#endif
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1685..c5ccc7b7bf 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -116,18 +116,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +145,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +157,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +176,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +188,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +223,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +255,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +279,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +314,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +332,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,19 +371,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- we can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -393,10 +393,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -409,18 +409,54 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+ERROR: detect_conflict requires a Boolean value
+-- now it works, but will report a warning due to disabled track_commit_timestamp
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+WARNING: conflict detection could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | t | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = true);
+WARNING: conflict detection could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | t | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7037..b3f2ab1684 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -287,6 +287,25 @@ ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+
+-- now it works, but will report a warning due to disabled track_commit_timestamp
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- let's do some tests with pg_create_subscription rather than superuser
SET SESSION AUTHORIZATION regress_subscription_user3;
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 471e981962..78c0307165 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -331,11 +331,12 @@ is( $result, qq(1|bar
2|baz),
'update works with REPLICA IDENTITY FULL and a primary key');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1");
-$node_subscriber->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (detect_conflict = true)"
+);
$node_subscriber->safe_psql('postgres', "DELETE FROM tab_full_pk");
@@ -352,10 +353,10 @@ $node_publisher->wait_for_catchup('tap_sub');
my $logfile = slurp_file($node_subscriber->logfile, $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation "tab_full_pk"/,
+ qr/conflict update_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be updated./m,
'update target row is missing');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab_full_pk"/,
+ qr/conflict delete_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be deleted./m,
'delete target row is missing');
$node_subscriber->append_conf('postgresql.conf',
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 29580525a9..7a66a06b51 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -343,12 +343,12 @@ $result =
$node_subscriber2->safe_psql('postgres', "SELECT a FROM tab1 ORDER BY 1");
is($result, qq(), 'truncate of tab1 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = true)"
+);
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 VALUES (1, 'foo'), (4, 'bar'), (10, 'baz')");
@@ -372,21 +372,21 @@ $node_publisher->wait_for_catchup('sub2');
my $logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab1_2_2"/,
+ qr/conflict update_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_1"/,
+ qr/conflict delete_missing detected on relation "public.tab1_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_2_2"/,
+ qr/conflict delete_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_def"/,
+ qr/conflict delete_missing detected on relation "public.tab1_def".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_def');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = false)"
+);
# Tests for replication using root table identity and schema
@@ -773,12 +773,12 @@ pub_tab2|3|yyy
pub_tab2|5|zzz
xxx_c|6|aaa), 'inserts into tab2 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = true)"
+);
$node_subscriber1->safe_psql('postgres', "DELETE FROM tab2");
@@ -796,15 +796,35 @@ $node_publisher->wait_for_catchup('sub2');
$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab2_1"/,
+ qr/conflict update_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab2_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab2_1"/,
+ qr/conflict delete_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab2_1');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+# Enable the track_commit_timestamp to detect the conflict when attempting
+# to update a row that was previously modified by a different origin.
+$node_subscriber1->append_conf('postgresql.conf', 'track_commit_timestamp = on');
+$node_subscriber1->restart;
+
+$node_subscriber1->safe_psql('postgres', "INSERT INTO tab2 VALUES (3, 'yyy')");
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab2 SET b = 'quux' WHERE a = 3");
+
+$node_publisher->wait_for_catchup('sub_viaroot');
+
+$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
+ok( $logfile =~
+ qr/Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/,
+ 'updating a tuple that was modified by a different origin');
+
+# The remaining tests no longer test conflict detection.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = false)"
+);
+
+$node_subscriber1->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_subscriber1->restart;
# Test that replication continues to work correctly after altering the
# partition of a partitioned target table.
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 0ab57a4b5b..ec1a48d0f6 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -30,7 +30,7 @@ sub test_skip_lsn
# ERROR with its CONTEXT when retrieving this information.
my $contents = slurp_file($node_subscriber->logfile, $offset);
$contents =~
- qr/duplicate key value violates unique constraint "tbl_pkey".*\n.*DETAIL:.*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
+ qr/conflict .* detected on relation "public.tbl".*\n.*DETAIL:.* Key \(i\)=\(\d+\) already exists in unique index "tbl_pkey", which was modified by origin \d+ in transaction \d+ at .*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
or die "could not get error-LSN";
my $lsn = $1;
@@ -83,6 +83,7 @@ $node_subscriber->append_conf(
'postgresql.conf',
qq[
max_prepared_transactions = 10
+track_commit_timestamp = on
]);
$node_subscriber->start;
@@ -93,6 +94,7 @@ $node_publisher->safe_psql(
'postgres',
qq[
CREATE TABLE tbl (i INT, t BYTEA);
+ALTER TABLE tbl REPLICA IDENTITY FULL;
INSERT INTO tbl VALUES (1, NULL);
]);
$node_subscriber->safe_psql(
@@ -109,7 +111,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub FOR TABLE tbl");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on)"
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
# Initial synchronization failure causes the subscription to be disabled.
@@ -144,13 +146,14 @@ COMMIT;
test_skip_lsn($node_publisher, $node_subscriber,
"(2, NULL)", "2", "test skipping transaction");
-# Test for PREPARE and COMMIT PREPARED. Insert the same data to tbl and
-# PREPARE the transaction, raising an error. Then skip the transaction.
+# Test for PREPARE and COMMIT PREPARED. Update the data and PREPARE the
+# transaction, raising an error on the subscriber due to violation of the
+# unique constraint on tbl. Then skip the transaction.
$node_publisher->safe_psql(
'postgres',
qq[
BEGIN;
-INSERT INTO tbl VALUES (1, NULL);
+UPDATE tbl SET i = 2;
PREPARE TRANSACTION 'gtx';
COMMIT PREPARED 'gtx';
]);
diff --git a/src/test/subscription/t/030_origin.pl b/src/test/subscription/t/030_origin.pl
index 056561f008..97c2057275 100644
--- a/src/test/subscription/t/030_origin.pl
+++ b/src/test/subscription/t/030_origin.pl
@@ -27,9 +27,14 @@ my $stderr;
my $node_A = PostgreSQL::Test::Cluster->new('node_A');
$node_A->init(allows_streaming => 'logical');
$node_A->start;
+
# node_B
my $node_B = PostgreSQL::Test::Cluster->new('node_B');
$node_B->init(allows_streaming => 'logical');
+
+# Enable the track_commit_timestamp to detect the conflict when attempting to
+# update a row that was previously modified by a different origin.
+$node_B->append_conf('postgresql.conf', 'track_commit_timestamp = on');
$node_B->start;
# Create table on node_A
@@ -139,6 +144,44 @@ is($result, qq(),
'Remote data originating from another node (not the publisher) is not replicated when origin parameter is none'
);
+###############################################################################
+# Check that the conflict can be detected when attempting to update or
+# delete a row that was previously modified by a different source.
+###############################################################################
+
+$node_B->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_BC SET (detect_conflict = true);
+ DELETE FROM tab;");
+
+$node_A->safe_psql('postgres', "INSERT INTO tab VALUES (32);");
+
+# The update should update the row on node B that was inserted by node A.
+$node_C->safe_psql('postgres', "UPDATE tab SET a = 33 WHERE a = 32;");
+
+$node_B->wait_for_log(
+ qr/conflict update_differ detected on relation "public.tab".*\n.*DETAIL:.* Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
+$node_B->safe_psql('postgres', "DELETE FROM tab;");
+$node_A->safe_psql('postgres', "INSERT INTO tab VALUES (33);");
+
+# The delete should remove the row on node B that was inserted by node A.
+$node_C->safe_psql('postgres', "DELETE FROM tab WHERE a = 33;");
+
+$node_B->wait_for_log(
+ qr/conflict delete_differ detected on relation "public.tab".*\n.*DETAIL:.* Deleting a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
+$node_A->wait_for_catchup($subname_BA);
+$node_B->wait_for_catchup($subname_AB);
+
+# The remaining tests no longer test conflict detection.
+$node_B->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_BC SET (detect_conflict = false);");
+
+$node_B->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_B->restart;
+
###############################################################################
# Specifying origin = NONE indicates that the publisher should only replicate the
# changes that are generated locally from node_B, but in this case since the
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b4d7f9217c..2098ed7467 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -467,6 +467,7 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictType
ConnCacheEntry
ConnCacheKey
ConnParams
--
2.30.0.windows.2
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Conflict detection and logging in logical replication
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-19 08:36 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-22 09:03 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-25 06:34 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
@ 2024-07-25 10:42 ` Amit Kapila <[email protected]>
2024-07-26 11:33 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
1 sibling, 1 reply; 50+ messages in thread
From: Amit Kapila @ 2024-07-25 10:42 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: shveta malik <[email protected]>; Nisha Moond <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Ashutosh Bapat <[email protected]>
On Thu, Jul 25, 2024 at 12:04 PM Zhijie Hou (Fujitsu)
<[email protected]> wrote:
>
> Here is the V6 patch set which addressed Shveta and Nisha's comments
> in [1][2][3][4].
>
Do we need an option detect_conflict for logging conflicts? The
possible reason to include such an option is to avoid any overhead
during apply due to conflict detection. IIUC, to detect some of the
conflicts like update_differ and delete_differ, we would need to fetch
commit_ts information which could be costly but we do that only when
GUC track_commit_timestamp is enabled which would anyway have overhead
on its own. Can we do performance testing to see how much additional
overhead we have due to fetching commit_ts information during conflict
detection?
The other time we need to enquire commit_ts is to log the conflict
detection information which is an ERROR path, so performance shouldn't
matter in this case.
In general, it would be good to enable conflict detection/logging by
default but if it has overhead then we can consider adding this new
option. Anyway, adding an option could be a separate patch (at least
for review), let the first patch be the core code of conflict
detection and logging.
minor cosmetic comments:
1.
+static void
+check_conflict_detection(void)
+{
+ if (!track_commit_timestamp)
+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("conflict detection could be incomplete due to disabled
track_commit_timestamp"),
+ errdetail("Conflicts update_differ and delete_differ cannot be
detected, and the origin and commit timestamp for the local row will
not be logged."));
+}
The errdetail string is too long. It would be better to split it into
multiple rows.
2.
-
+static void check_conflict_detection(void);
Spurious line removal.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Conflict detection and logging in logical replication
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-19 08:36 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-22 09:03 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-25 06:34 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-25 10:42 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
@ 2024-07-26 11:33 ` Amit Kapila <[email protected]>
2024-07-29 06:14 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
0 siblings, 1 reply; 50+ messages in thread
From: Amit Kapila @ 2024-07-26 11:33 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: shveta malik <[email protected]>; Nisha Moond <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Ashutosh Bapat <[email protected]>
On Thu, Jul 25, 2024 at 4:12 PM Amit Kapila <[email protected]> wrote:
>
> On Thu, Jul 25, 2024 at 12:04 PM Zhijie Hou (Fujitsu)
> <[email protected]> wrote:
> >
> > Here is the V6 patch set which addressed Shveta and Nisha's comments
> > in [1][2][3][4].
> >
>
> Do we need an option detect_conflict for logging conflicts? The
> possible reason to include such an option is to avoid any overhead
> during apply due to conflict detection. IIUC, to detect some of the
> conflicts like update_differ and delete_differ, we would need to fetch
> commit_ts information which could be costly but we do that only when
> GUC track_commit_timestamp is enabled which would anyway have overhead
> on its own. Can we do performance testing to see how much additional
> overhead we have due to fetching commit_ts information during conflict
> detection?
>
> The other time we need to enquire commit_ts is to log the conflict
> detection information which is an ERROR path, so performance shouldn't
> matter in this case.
>
> In general, it would be good to enable conflict detection/logging by
> default but if it has overhead then we can consider adding this new
> option. Anyway, adding an option could be a separate patch (at least
> for review), let the first patch be the core code of conflict
> detection and logging.
>
> minor cosmetic comments:
> 1.
> +static void
> +check_conflict_detection(void)
> +{
> + if (!track_commit_timestamp)
> + ereport(WARNING,
> + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("conflict detection could be incomplete due to disabled
> track_commit_timestamp"),
> + errdetail("Conflicts update_differ and delete_differ cannot be
> detected, and the origin and commit timestamp for the local row will
> not be logged."));
> +}
>
> The errdetail string is too long. It would be better to split it into
> multiple rows.
>
> 2.
> -
> +static void check_conflict_detection(void);
>
> Spurious line removal.
>
A few more comments:
1.
For duplicate key, the patch reports conflict as following:
ERROR: conflict insert_exists detected on relation "public.t1"
2024-07-26 11:06:34.570 IST [27800] DETAIL: Key (c1)=(1) already
exists in unique index "t1_pkey", which was modified by origin 1 in
transaction 770 at 2024-07-26 09:16:47.79805+05:30.
2024-07-26 11:06:34.570 IST [27800] CONTEXT: processing remote data
for replication origin "pg_16387" during message type "INSERT" for
replication target relation "public.t1" in transaction 742, finished
at 0/151A108
In detail, it is better to display the origin name instead of the
origin id. This will be similar to what we do in CONTEXT information.
2.
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
- slot, estate, false, false,
- NULL, NIL, false);
+ slot, estate, false,
+ conflictindexes, &conflict,
It is better to use true/false for the bool parameter (something like
conflictindexes ? true : false). That will make the code easier to
follow.
3. The need for ReCheckConflictIndexes() is not clear from comments.
Can you please add a few comments to explain this?
4.
- will simply be skipped.
+ will simply be skipped. Please refer to <link
linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ for all the conflicts that will be logged when enabling
<literal>detect_conflict</literal>.
</para>
It would be easier to read the patch if you move <link .. to the next line.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 50+ messages in thread
* RE: Conflict detection and logging in logical replication
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-19 08:36 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-22 09:03 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-25 06:34 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-25 10:42 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
2024-07-26 11:33 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
@ 2024-07-29 06:14 ` Zhijie Hou (Fujitsu) <[email protected]>
2024-07-29 09:24 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
2024-07-29 10:59 ` Re: Conflict detection and logging in logical replication Dilip Kumar <[email protected]>
0 siblings, 2 replies; 50+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-07-29 06:14 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: shveta malik <[email protected]>; Nisha Moond <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Ashutosh Bapat <[email protected]>
On Friday, July 26, 2024 7:34 PM Amit Kapila <[email protected]> wrote:
>
> On Thu, Jul 25, 2024 at 4:12 PM Amit Kapila <[email protected]>
> wrote:
> > >
> A few more comments:
Thanks for the comments.
> 1.
> For duplicate key, the patch reports conflict as following:
> ERROR: conflict insert_exists detected on relation "public.t1"
> 2024-07-26 11:06:34.570 IST [27800] DETAIL: Key (c1)=(1) already exists in
> unique index "t1_pkey", which was modified by origin 1 in transaction 770 at
> 2024-07-26 09:16:47.79805+05:30.
> 2024-07-26 11:06:34.570 IST [27800] CONTEXT: processing remote data for
> replication origin "pg_16387" during message type "INSERT" for replication
> target relation "public.t1" in transaction 742, finished at 0/151A108
>
> In detail, it is better to display the origin name instead of the origin id. This will
> be similar to what we do in CONTEXT information.
Agreed. Before modifying this, I'd like to confirm the message style in the
cases where origin id may not have a corresponding origin name (e.g., if the
data was modified locally (id = 0), or if the origin that modified the data has
been dropped). I thought of two styles:
1)
- for local change: "xxx was modified by a different origin \"(local)\" in transaction 123 at 2024.."
- for dropped origin: "xxx was modified by a different origin \"(unknown)\" in transaction 123 at 2024.."
One issue for this style is that user may create an origin with the same name
here (e.g. "(local)" and "(unknown)").
2)
- for local change: "xxx was modified locally in transaction 123 at 2024.."
- for dropped origin: "xxx was modified by an unknown different origin 1234 in transaction 123 at 2024.."
This style slightly modifies the message format. I personally feel 2) maybe
better but am OK for other options as well.
What do you think ?
Here is the V7 patch set that addressed all the comments so far[1][2][3].
The subscription option part is splitted into the separate patch 0002 and
we will decide whether to drop this patch after finishing the perf tests.
Note that I didn't display the tuple value in the message as the discussion
is still ongoing[4].
[1] https://www.postgresql.org/message-id/CAJpy0uDhCnzvNHVYwse%3DKxmOB%3DqtXr6twnDP9xqdzT-oU0OWEQ%40mail...
[2] https://www.postgresql.org/message-id/CAA4eK1%2BCJXKK34zJdEJZf2Mpn5QyMyaZiPDSNS6%3Dkvewr0-pdg%40mail...
[3] https://www.postgresql.org/message-id/CAA4eK1Lmu%3DoVySfGjxEUykCT3FPnL1YFDHKr1ZMwFy7WUgfc6g%40mail.g...
[4] https://www.postgresql.org/message-id/CAA4eK1%2BaK4MLxbfLtp%3DEV5bpvJozKhxGDRS6T9q8sz_s%2BLK3vw%40ma...
Best Regards,
Hou zj
Attachments:
[application/octet-stream] v7-0003-Collect-statistics-about-conflicts-in-logical-rep.patch (23.7K, ../../OS0PR01MB57167251931CCFD703ADCF1694B72@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v7-0003-Collect-statistics-about-conflicts-in-logical-rep.patch)
download | inline diff:
From 32d712bf6428f747912b695d3b4b3a12026fa748 Mon Sep 17 00:00:00 2001
From: Hou Zhijie <[email protected]>
Date: Wed, 3 Jul 2024 10:34:10 +0800
Subject: [PATCH v7 3/3] Collect statistics about conflicts in logical
replication
This commit adds columns in view pg_stat_subscription_stats to show
information about the conflict which occur during the application of
logical replication changes. Currently, the following columns are added.
insert_exists_count:
Number of times a row insertion violated a NOT DEFERRABLE unique constraint.
update_exists_count:
Number of times that the updated value of a row violates a NOT DEFERRABLE unique constraint.
update_differ_count:
Number of times an update was performed on a row that was previously modified by another origin.
update_missing_count:
Number of times that the tuple to be updated is missing.
delete_missing_count:
Number of times that the tuple to be deleted is missing.
delete_differ_count:
Number of times a delete was performed on a row that was previously modified by another origin.
The conflicts will be tracked only when detect_conflict option of the
subscription is enabled. Additionally, update_differ and delete_differ
can be detected only when track_commit_timestamp is enabled.
---
doc/src/sgml/monitoring.sgml | 80 ++++++++++-
doc/src/sgml/ref/create_subscription.sgml | 5 +-
src/backend/catalog/system_views.sql | 6 +
src/backend/replication/logical/conflict.c | 4 +
.../utils/activity/pgstat_subscription.c | 17 +++
src/backend/utils/adt/pgstatfuncs.c | 33 ++++-
src/include/catalog/pg_proc.dat | 6 +-
src/include/pgstat.h | 4 +
src/include/replication/conflict.h | 7 +
src/test/regress/expected/rules.out | 8 +-
src/test/subscription/t/026_stats.pl | 125 +++++++++++++++++-
11 files changed, 274 insertions(+), 21 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 55417a6fa9..7b93334967 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -507,7 +507,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<row>
<entry><structname>pg_stat_subscription_stats</structname><indexterm><primary>pg_stat_subscription_stats</primary></indexterm></entry>
- <entry>One row per subscription, showing statistics about errors.
+ <entry>One row per subscription, showing statistics about errors and conflicts.
See <link linkend="monitoring-pg-stat-subscription-stats">
<structname>pg_stat_subscription_stats</structname></link> for details.
</entry>
@@ -2171,6 +2171,84 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>insert_exists_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times a row insertion violated a
+ <literal>NOT DEFERRABLE</literal> unique constraint while applying
+ changes. This conflict is counted only if
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ is enabled
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>update_exists_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times that the updated value of a row violated a
+ <literal>NOT DEFERRABLE</literal> unique constraint while applying
+ changes. This conflict is counted only if
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ is enabled
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>update_differ_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times an update was performed on a row that was previously
+ modified by another source while applying changes. This conflict is
+ counted only when the
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ option of the subscription and <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ are enabled
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>update_missing_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times that the tuple to be updated was not found while applying
+ changes. This conflict is counted only if
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ is enabled
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>delete_missing_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times that the tuple to be deleted was not found while applying
+ changes. This conflict is counted only if
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ is enabled
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>delete_differ_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of times a delete was performed on a row that was previously
+ modified by another source while applying changes. This conflict is
+ counted only when the
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ option of the subscription and <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ are enabled
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>stats_reset</structfield> <type>timestamp with time zone</type>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index d07201abec..67c7c9fcbc 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -437,8 +437,9 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
The default is <literal>false</literal>.
</para>
<para>
- When conflict detection is enabled, additional logging is triggered
- in the following scenarios:
+ When conflict detection is enabled, additional logging is triggered and
+ the conflict statistics are collected(displayed in the
+ <link linkend="monitoring-pg-stat-subscription-stats"><structname>pg_stat_subscription_stats</structname></link> view) in the following scenarios:
<variablelist>
<varlistentry>
<term><literal>insert_exists</literal></term>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index d084bfc48a..5244d8e356 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1366,6 +1366,12 @@ CREATE VIEW pg_stat_subscription_stats AS
s.subname,
ss.apply_error_count,
ss.sync_error_count,
+ ss.insert_exists_count,
+ ss.update_exists_count,
+ ss.update_differ_count,
+ ss.update_missing_count,
+ ss.delete_missing_count,
+ ss.delete_differ_count,
ss.stats_reset
FROM pg_subscription as s,
pg_stat_get_subscription_stats(s.oid) as ss;
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 4918011a7f..6e2f15cd7d 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -15,8 +15,10 @@
#include "postgres.h"
#include "access/commit_ts.h"
+#include "pgstat.h"
#include "replication/conflict.h"
#include "replication/origin.h"
+#include "replication/worker_internal.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
@@ -77,6 +79,8 @@ ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
RepOriginId localorigin, TimestampTz localts,
TupleTableSlot *conflictslot)
{
+ pgstat_report_subscription_conflict(MySubscription->oid, type);
+
ereport(elevel,
errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
errmsg("conflict %s detected on relation \"%s.%s\"",
diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c
index d9af8de658..e06c92727e 100644
--- a/src/backend/utils/activity/pgstat_subscription.c
+++ b/src/backend/utils/activity/pgstat_subscription.c
@@ -39,6 +39,21 @@ pgstat_report_subscription_error(Oid subid, bool is_apply_error)
pending->sync_error_count++;
}
+/*
+ * Report a subscription conflict.
+ */
+void
+pgstat_report_subscription_conflict(Oid subid, ConflictType type)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStat_BackendSubEntry *pending;
+
+ entry_ref = pgstat_prep_pending_entry(PGSTAT_KIND_SUBSCRIPTION,
+ InvalidOid, subid, NULL);
+ pending = entry_ref->pending;
+ pending->conflict_count[type]++;
+}
+
/*
* Report creating the subscription.
*/
@@ -101,6 +116,8 @@ pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
#define SUB_ACC(fld) shsubent->stats.fld += localent->fld
SUB_ACC(apply_error_count);
SUB_ACC(sync_error_count);
+ for (int i = 0; i < CONFLICT_NUM_TYPES; i++)
+ SUB_ACC(conflict_count[i]);
#undef SUB_ACC
pgstat_unlock_entry(entry_ref);
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 3876339ee1..e36ddb4cac 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1966,13 +1966,14 @@ pg_stat_get_replication_slot(PG_FUNCTION_ARGS)
Datum
pg_stat_get_subscription_stats(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_SUBSCRIPTION_STATS_COLS 4
+#define PG_STAT_GET_SUBSCRIPTION_STATS_COLS 10
Oid subid = PG_GETARG_OID(0);
TupleDesc tupdesc;
Datum values[PG_STAT_GET_SUBSCRIPTION_STATS_COLS] = {0};
bool nulls[PG_STAT_GET_SUBSCRIPTION_STATS_COLS] = {0};
PgStat_StatSubEntry *subentry;
PgStat_StatSubEntry allzero;
+ int i = 0;
/* Get subscription stats */
subentry = pgstat_fetch_stat_subscription(subid);
@@ -1985,7 +1986,19 @@ pg_stat_get_subscription_stats(PG_FUNCTION_ARGS)
INT8OID, -1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 3, "sync_error_count",
INT8OID, -1, 0);
- TupleDescInitEntry(tupdesc, (AttrNumber) 4, "stats_reset",
+ TupleDescInitEntry(tupdesc, (AttrNumber) 4, "insert_exists_count",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 5, "update_exists_count",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 6, "update_differ_count",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 7, "update_missing_count",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 8, "delete_missing_count",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 9, "delete_differ_count",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 10, "stats_reset",
TIMESTAMPTZOID, -1, 0);
BlessTupleDesc(tupdesc);
@@ -1997,19 +2010,25 @@ pg_stat_get_subscription_stats(PG_FUNCTION_ARGS)
}
/* subid */
- values[0] = ObjectIdGetDatum(subid);
+ values[i++] = ObjectIdGetDatum(subid);
/* apply_error_count */
- values[1] = Int64GetDatum(subentry->apply_error_count);
+ values[i++] = Int64GetDatum(subentry->apply_error_count);
/* sync_error_count */
- values[2] = Int64GetDatum(subentry->sync_error_count);
+ values[i++] = Int64GetDatum(subentry->sync_error_count);
+
+ /* conflict count */
+ for (int nconflict = 0; nconflict < CONFLICT_NUM_TYPES; nconflict++)
+ values[i++] = Int64GetDatum(subentry->conflict_count[nconflict]);
/* stats_reset */
if (subentry->stat_reset_timestamp == 0)
- nulls[3] = true;
+ nulls[i] = true;
else
- values[3] = TimestampTzGetDatum(subentry->stat_reset_timestamp);
+ values[i] = TimestampTzGetDatum(subentry->stat_reset_timestamp);
+
+ Assert(i + 1 == PG_STAT_GET_SUBSCRIPTION_STATS_COLS);
/* Returns the record as Datum */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 06b2f4ba66..12976efc11 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5532,9 +5532,9 @@
{ oid => '6231', descr => 'statistics: information about subscription stats',
proname => 'pg_stat_get_subscription_stats', provolatile => 's',
proparallel => 'r', prorettype => 'record', proargtypes => 'oid',
- proallargtypes => '{oid,oid,int8,int8,timestamptz}',
- proargmodes => '{i,o,o,o,o}',
- proargnames => '{subid,subid,apply_error_count,sync_error_count,stats_reset}',
+ proallargtypes => '{oid,oid,int8,int8,int8,int8,int8,int8,int8,int8,timestamptz}',
+ proargmodes => '{i,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{subid,subid,apply_error_count,sync_error_count,insert_exists_count,update_exists_count,update_differ_count,update_missing_count,delete_missing_count,delete_differ_count,stats_reset}',
prosrc => 'pg_stat_get_subscription_stats' },
{ oid => '6118', descr => 'statistics: information about subscription',
proname => 'pg_stat_get_subscription', prorows => '10', proisstrict => 'f',
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 6b99bb8aad..ad6619bcd0 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -14,6 +14,7 @@
#include "datatype/timestamp.h"
#include "portability/instr_time.h"
#include "postmaster/pgarch.h" /* for MAX_XFN_CHARS */
+#include "replication/conflict.h"
#include "utils/backend_progress.h" /* for backward compatibility */
#include "utils/backend_status.h" /* for backward compatibility */
#include "utils/relcache.h"
@@ -135,6 +136,7 @@ typedef struct PgStat_BackendSubEntry
{
PgStat_Counter apply_error_count;
PgStat_Counter sync_error_count;
+ PgStat_Counter conflict_count[CONFLICT_NUM_TYPES];
} PgStat_BackendSubEntry;
/* ----------
@@ -393,6 +395,7 @@ typedef struct PgStat_StatSubEntry
{
PgStat_Counter apply_error_count;
PgStat_Counter sync_error_count;
+ PgStat_Counter conflict_count[CONFLICT_NUM_TYPES];
TimestampTz stat_reset_timestamp;
} PgStat_StatSubEntry;
@@ -695,6 +698,7 @@ extern PgStat_SLRUStats *pgstat_fetch_slru(void);
*/
extern void pgstat_report_subscription_error(Oid subid, bool is_apply_error);
+extern void pgstat_report_subscription_conflict(Oid subid, ConflictType conflict);
extern void pgstat_create_subscription(Oid subid);
extern void pgstat_drop_subscription(Oid subid);
extern PgStat_StatSubEntry *pgstat_fetch_stat_subscription(Oid subid);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 3a7260d3c1..b5e9c79100 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -17,6 +17,11 @@
/*
* Conflict types that could be encountered when applying remote changes.
+ *
+ * This enum is used in statistics collection (see
+ * PgStat_StatSubEntry::conflict_count) as well, therefore, when adding new
+ * values or reordering existing ones, ensure to review and potentially adjust
+ * the corresponding statistics collection codes.
*/
typedef enum
{
@@ -39,6 +44,8 @@ typedef enum
CT_DELETE_DIFFER,
} ConflictType;
+#define CONFLICT_NUM_TYPES (CT_DELETE_DIFFER + 1)
+
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
extern void ReportApplyConflict(int elevel, ConflictType type,
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 5201280669..3c1154b14b 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2140,9 +2140,15 @@ pg_stat_subscription_stats| SELECT ss.subid,
s.subname,
ss.apply_error_count,
ss.sync_error_count,
+ ss.insert_exists_count,
+ ss.update_exists_count,
+ ss.update_differ_count,
+ ss.update_missing_count,
+ ss.delete_missing_count,
+ ss.delete_differ_count,
ss.stats_reset
FROM pg_subscription s,
- LATERAL pg_stat_get_subscription_stats(s.oid) ss(subid, apply_error_count, sync_error_count, stats_reset);
+ LATERAL pg_stat_get_subscription_stats(s.oid) ss(subid, apply_error_count, sync_error_count, insert_exists_count, update_exists_count, update_differ_count, update_missing_count, delete_missing_count, delete_differ_count, stats_reset);
pg_stat_sys_indexes| SELECT relid,
indexrelid,
schemaname,
diff --git a/src/test/subscription/t/026_stats.pl b/src/test/subscription/t/026_stats.pl
index fb3e5629b3..56eafa5ba6 100644
--- a/src/test/subscription/t/026_stats.pl
+++ b/src/test/subscription/t/026_stats.pl
@@ -16,6 +16,7 @@ $node_publisher->start;
# Create subscriber node.
my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf', 'track_commit_timestamp = on');
$node_subscriber->start;
@@ -30,6 +31,7 @@ sub create_sub_pub_w_errors
qq[
BEGIN;
CREATE TABLE $table_name(a int);
+ ALTER TABLE $table_name REPLICA IDENTITY FULL;
INSERT INTO $table_name VALUES (1);
COMMIT;
]);
@@ -53,7 +55,7 @@ sub create_sub_pub_w_errors
# infinite error loop due to violating the unique constraint.
my $sub_name = $table_name . '_sub';
$node_subscriber->safe_psql($db,
- qq(CREATE SUBSCRIPTION $sub_name CONNECTION '$publisher_connstr' PUBLICATION $pub_name)
+ qq(CREATE SUBSCRIPTION $sub_name CONNECTION '$publisher_connstr' PUBLICATION $pub_name WITH (detect_conflict = on))
);
$node_publisher->wait_for_catchup($sub_name);
@@ -95,7 +97,7 @@ sub create_sub_pub_w_errors
$node_subscriber->poll_query_until(
$db,
qq[
- SELECT apply_error_count > 0
+ SELECT apply_error_count > 0 AND insert_exists_count > 0
FROM pg_stat_subscription_stats
WHERE subname = '$sub_name'
])
@@ -105,6 +107,85 @@ sub create_sub_pub_w_errors
# Truncate test table so that apply worker can continue.
$node_subscriber->safe_psql($db, qq(TRUNCATE $table_name));
+ # Insert a row on the subscriber.
+ $node_subscriber->safe_psql($db, qq(INSERT INTO $table_name VALUES (2)));
+
+ # Update data from test table on the publisher, raising an error on the
+ # subscriber due to violation of the unique constraint on test table.
+ $node_publisher->safe_psql($db, qq(UPDATE $table_name SET a = 2;));
+
+ # Wait for the apply error to be reported.
+ $node_subscriber->poll_query_until(
+ $db,
+ qq[
+ SELECT update_exists_count > 0
+ FROM pg_stat_subscription_stats
+ WHERE subname = '$sub_name'
+ ])
+ or die
+ qq(Timed out while waiting for update_exists conflict for subscription '$sub_name');
+
+ # Truncate test table so that the update will be skipped and the test can
+ # continue.
+ $node_subscriber->safe_psql($db, qq(TRUNCATE $table_name));
+
+ # delete the data from test table on the publisher. The delete should be
+ # skipped on the subscriber as there are no data in the test table.
+ $node_publisher->safe_psql($db, qq(DELETE FROM $table_name;));
+
+ # Wait for the tuple missing to be reported.
+ $node_subscriber->poll_query_until(
+ $db,
+ qq[
+ SELECT update_missing_count > 0 AND delete_missing_count > 0
+ FROM pg_stat_subscription_stats
+ WHERE subname = '$sub_name'
+ ])
+ or die
+ qq(Timed out while waiting for tuple missing conflict for subscription '$sub_name');
+
+ # Prepare data for further tests.
+ $node_publisher->safe_psql($db, qq(INSERT INTO $table_name VALUES (1)));
+ $node_publisher->wait_for_catchup($sub_name);
+ $node_subscriber->safe_psql($db, qq(
+ TRUNCATE $table_name;
+ INSERT INTO $table_name VALUES (1);
+ ));
+
+ # Update data from test table on the publisher, updating a row on the
+ # subscriber that was modified by a different origin.
+ $node_publisher->safe_psql($db, qq(UPDATE $table_name SET a = 2;));
+
+ $node_subscriber->poll_query_until(
+ $db,
+ qq[
+ SELECT update_differ_count > 0
+ FROM pg_stat_subscription_stats
+ WHERE subname = '$sub_name'
+ ])
+ or die
+ qq(Timed out while waiting for update_differ conflict for subscription '$sub_name');
+
+ # Prepare data for further tests.
+ $node_subscriber->safe_psql($db, qq(
+ TRUNCATE $table_name;
+ INSERT INTO $table_name VALUES (2);
+ ));
+
+ # Delete data to test table on the publisher, deleting a row on the
+ # subscriber that was modified by a different origin.
+ $node_publisher->safe_psql($db, qq(DELETE FROM $table_name;));
+
+ $node_subscriber->poll_query_until(
+ $db,
+ qq[
+ SELECT delete_differ_count > 0
+ FROM pg_stat_subscription_stats
+ WHERE subname = '$sub_name'
+ ])
+ or die
+ qq(Timed out while waiting for delete_differ conflict for subscription '$sub_name');
+
return ($pub_name, $sub_name);
}
@@ -128,11 +209,17 @@ is( $node_subscriber->safe_psql(
$db,
qq(SELECT apply_error_count > 0,
sync_error_count > 0,
+ insert_exists_count > 0,
+ update_exists_count > 0,
+ update_differ_count > 0,
+ update_missing_count > 0,
+ delete_missing_count > 0,
+ delete_differ_count > 0,
stats_reset IS NULL
FROM pg_stat_subscription_stats
WHERE subname = '$sub1_name')
),
- qq(t|t|t),
+ qq(t|t|t|t|t|t|t|t|t),
qq(Check that apply errors and sync errors are both > 0 and stats_reset is NULL for subscription '$sub1_name'.)
);
@@ -146,11 +233,17 @@ is( $node_subscriber->safe_psql(
$db,
qq(SELECT apply_error_count = 0,
sync_error_count = 0,
+ insert_exists_count = 0,
+ update_exists_count = 0,
+ update_differ_count = 0,
+ update_missing_count = 0,
+ delete_missing_count = 0,
+ delete_differ_count = 0,
stats_reset IS NOT NULL
FROM pg_stat_subscription_stats
WHERE subname = '$sub1_name')
),
- qq(t|t|t),
+ qq(t|t|t|t|t|t|t|t|t),
qq(Confirm that apply errors and sync errors are both 0 and stats_reset is not NULL after reset for subscription '$sub1_name'.)
);
@@ -186,11 +279,17 @@ is( $node_subscriber->safe_psql(
$db,
qq(SELECT apply_error_count > 0,
sync_error_count > 0,
+ insert_exists_count > 0,
+ update_exists_count > 0,
+ update_differ_count > 0,
+ update_missing_count > 0,
+ delete_missing_count > 0,
+ delete_differ_count > 0,
stats_reset IS NULL
FROM pg_stat_subscription_stats
WHERE subname = '$sub2_name')
),
- qq(t|t|t),
+ qq(t|t|t|t|t|t|t|t|t),
qq(Confirm that apply errors and sync errors are both > 0 and stats_reset is NULL for sub '$sub2_name'.)
);
@@ -203,11 +302,17 @@ is( $node_subscriber->safe_psql(
$db,
qq(SELECT apply_error_count = 0,
sync_error_count = 0,
+ insert_exists_count = 0,
+ update_exists_count = 0,
+ update_differ_count = 0,
+ update_missing_count = 0,
+ delete_missing_count = 0,
+ delete_differ_count = 0,
stats_reset IS NOT NULL
FROM pg_stat_subscription_stats
WHERE subname = '$sub1_name')
),
- qq(t|t|t),
+ qq(t|t|t|t|t|t|t|t|t),
qq(Confirm that apply errors and sync errors are both 0 and stats_reset is not NULL for sub '$sub1_name' after reset.)
);
@@ -215,11 +320,17 @@ is( $node_subscriber->safe_psql(
$db,
qq(SELECT apply_error_count = 0,
sync_error_count = 0,
+ insert_exists_count = 0,
+ update_exists_count = 0,
+ update_differ_count = 0,
+ update_missing_count = 0,
+ delete_missing_count = 0,
+ delete_differ_count = 0,
stats_reset IS NOT NULL
FROM pg_stat_subscription_stats
WHERE subname = '$sub2_name')
),
- qq(t|t|t),
+ qq(t|t|t|t|t|t|t|t|t),
qq(Confirm that apply errors and sync errors are both 0 and stats_reset is not NULL for sub '$sub2_name' after reset.)
);
--
2.30.0.windows.2
[application/octet-stream] v7-0001-Detect-and-log-conflicts-in-logical-replication.patch (44.5K, ../../OS0PR01MB57167251931CCFD703ADCF1694B72@OS0PR01MB5716.jpnprd01.prod.outlook.com/3-v7-0001-Detect-and-log-conflicts-in-logical-replication.patch)
download | inline diff:
From 61550286a80ef8fa3abc33c2b49914e28908d1e3 Mon Sep 17 00:00:00 2001
From: Hou Zhijie <[email protected]>
Date: Mon, 29 Jul 2024 11:14:37 +0800
Subject: [PATCH v7 1/3] Detect and log conflicts in logical replication
This patch enables the logical replication worker to provide additional logging
information in the following conflict scenarios while applying changes:
insert_exists: Inserting a row that violates a NOT DEFERRABLE unique constraint.
update_exists: The updated row value violates a NOT DEFERRABLE unique constraint.
update_differ: Updating a row that was previously modified by another origin.
update_missing: The tuple to be updated is missing.
delete_missing: The tuple to be deleted is missing.
delete_differ: Deleting a row that was previously modified by another origin.
For insert_exists and update_exists conflicts, the log can include origin
and commit timestamp details of the conflicting key with track_commit_timestamp
enabled.
update_differ and delete_differ conflicts can only be detected when
track_commit_timestamp is enabled.
---
doc/src/sgml/logical-replication.sgml | 90 +++++++-
src/backend/executor/execIndexing.c | 14 +-
src/backend/executor/execReplication.c | 237 +++++++++++++++-----
src/backend/replication/logical/Makefile | 1 +
src/backend/replication/logical/conflict.c | 193 ++++++++++++++++
src/backend/replication/logical/meson.build | 1 +
src/backend/replication/logical/worker.c | 82 +++++--
src/include/replication/conflict.h | 50 +++++
src/test/subscription/t/001_rep_changes.pl | 10 +-
src/test/subscription/t/013_partition.pl | 51 ++---
src/test/subscription/t/029_on_error.pl | 11 +-
src/test/subscription/t/030_origin.pl | 47 ++++
src/tools/pgindent/typedefs.list | 1 +
13 files changed, 661 insertions(+), 127 deletions(-)
create mode 100644 src/backend/replication/logical/conflict.c
create mode 100644 src/include/replication/conflict.h
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index a23a3d57e2..830c7a9b02 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1583,6 +1583,86 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
will simply be skipped.
</para>
+ <para>
+ Additional logging is triggered in the following <firstterm>conflict</firstterm>
+ scenarios:
+ <variablelist>
+ <varlistentry>
+ <term><literal>insert_exists</literal></term>
+ <listitem>
+ <para>
+ Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint. Note that to obtain the origin and commit
+ timestamp details of the conflicting key in the log, ensure that
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. In this scenario, an error will be raised until the
+ conflict is resolved manually.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_exists</literal></term>
+ <listitem>
+ <para>
+ The updated value of a row violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint. Note that to obtain the origin and commit
+ timestamp details of the conflicting key in the log, ensure that
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. In this scenario, an error will be raised until the
+ conflict is resolved manually. Note that when updating a
+ partitioned table, if the updated row value satisfies another
+ partition constraint resulting in the row being inserted into a
+ new partition, the <literal>insert_exists</literal> conflict may
+ arise if the new row violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_differ</literal></term>
+ <listitem>
+ <para>
+ Updating a row that was previously modified by another origin.
+ Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. Currenly, the update is always applied regardless of
+ the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be updated was not found. The update will simply be
+ skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be deleted was not found. The delete will simply be
+ skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_differ</literal></term>
+ <listitem>
+ <para>
+ Deleting a row that was previously modified by another origin. Note that this
+ conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. Currenly, the delete is always applied regardless of
+ the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
<para>
Logical replication operations are performed with the privileges of the role
which owns the subscription. Permissions failures on target tables will
@@ -1609,8 +1689,8 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
an error, the replication won't proceed, and the logical replication worker will
emit the following kind of message to the subscriber's server log:
<screen>
-ERROR: duplicate key value violates unique constraint "test_pkey"
-DETAIL: Key (c)=(1) already exists.
+ERROR: conflict insert_exists detected on relation "test_pkey"
+DETAIL: Key (a)=(1) already exists in unique index "t_pkey", which was modified by origin 0 in transaction 740 at 2024-06-26 10:47:04.727375+08.
CONTEXT: processing remote data for replication origin "pg_16395" during "INSERT" for replication target relation "public.test" in transaction 725 finished at 0/14C0378
</screen>
The LSN of the transaction that contains the change violating the constraint and
@@ -1636,6 +1716,12 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
Please note that skipping the whole transaction includes skipping changes that
might not violate any constraint. This can easily make the subscriber
inconsistent.
+ The additional details regarding conflicting rows, such as their origin and
+ commit timestamp can be seen in the log as well. Users can use these
+ information to make decisions on whether to retain the local change or adopt
+ the remote alteration. For instance, the origin in above log indicates that
+ the existing row was modified by a local change, users can manually perform
+ a remote-change-win resolution by deleting the local row.
</para>
<para>
diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c
index 9f05b3654c..ef522778a2 100644
--- a/src/backend/executor/execIndexing.c
+++ b/src/backend/executor/execIndexing.c
@@ -207,8 +207,9 @@ ExecOpenIndices(ResultRelInfo *resultRelInfo, bool speculative)
ii = BuildIndexInfo(indexDesc);
/*
- * If the indexes are to be used for speculative insertion, add extra
- * information required by unique index entries.
+ * If the indexes are to be used for speculative insertion or conflict
+ * detection in logical replication, add extra information required by
+ * unique index entries.
*/
if (speculative && ii->ii_Unique)
BuildSpeculativeIndexInfo(indexDesc, ii);
@@ -521,6 +522,11 @@ ExecInsertIndexTuples(ResultRelInfo *resultRelInfo,
* possible that a conflicting tuple is inserted immediately
* after this returns. But this can be used for a pre-check
* before insertion.
+ *
+ * If the 'slot' holds a tuple with valid tid, this tuple will
+ * be ignored when checking conflict. This can help in scenarios
+ * where we want to re-check for conflicts after inserting a
+ * tuple.
* ----------------------------------------------------------------
*/
bool
@@ -536,11 +542,9 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
ExprContext *econtext;
Datum values[INDEX_MAX_KEYS];
bool isnull[INDEX_MAX_KEYS];
- ItemPointerData invalidItemPtr;
bool checkedIndex = false;
ItemPointerSetInvalid(conflictTid);
- ItemPointerSetInvalid(&invalidItemPtr);
/*
* Get information from the result relation info structure.
@@ -629,7 +633,7 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
satisfiesConstraint =
check_exclusion_or_unique_constraint(heapRelation, indexRelation,
- indexInfo, &invalidItemPtr,
+ indexInfo, &slot->tts_tid,
values, isnull, estate, false,
CEOUC_WAIT, true,
conflictTid);
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index d0a89cd577..4af0040411 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -23,6 +23,7 @@
#include "commands/trigger.h"
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
+#include "replication/conflict.h"
#include "replication/logicalrelation.h"
#include "storage/lmgr.h"
#include "utils/builtins.h"
@@ -166,6 +167,51 @@ build_replindex_scan_key(ScanKey skey, Relation rel, Relation idxrel,
return skey_attoff;
}
+
+/*
+ * Helper function to check if it is necessary to re-fetch and lock the tuple
+ * due to concurrent modifications. This function should be called after
+ * invoking table_tuple_lock.
+ */
+static bool
+should_refetch_tuple(TM_Result res, TM_FailureData *tmfd)
+{
+ bool refetch = false;
+
+ switch (res)
+ {
+ case TM_Ok:
+ break;
+ case TM_Updated:
+ /* XXX: Improve handling here */
+ if (ItemPointerIndicatesMovedPartitions(&tmfd->ctid))
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("tuple to be locked was already moved to another partition due to concurrent update, retrying")));
+ else
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent update, retrying")));
+ refetch = true;
+ break;
+ case TM_Deleted:
+ /* XXX: Improve handling here */
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent delete, retrying")));
+ refetch = true;
+ break;
+ case TM_Invisible:
+ elog(ERROR, "attempted to lock invisible tuple");
+ break;
+ default:
+ elog(ERROR, "unexpected table_tuple_lock status: %u", res);
+ break;
+ }
+
+ return refetch;
+}
+
/*
* Search the relation 'rel' for tuple using the index.
*
@@ -260,34 +306,8 @@ retry:
PopActiveSnapshot();
- switch (res)
- {
- case TM_Ok:
- break;
- case TM_Updated:
- /* XXX: Improve handling here */
- if (ItemPointerIndicatesMovedPartitions(&tmfd.ctid))
- ereport(LOG,
- (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
- errmsg("tuple to be locked was already moved to another partition due to concurrent update, retrying")));
- else
- ereport(LOG,
- (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
- errmsg("concurrent update, retrying")));
- goto retry;
- case TM_Deleted:
- /* XXX: Improve handling here */
- ereport(LOG,
- (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
- errmsg("concurrent delete, retrying")));
- goto retry;
- case TM_Invisible:
- elog(ERROR, "attempted to lock invisible tuple");
- break;
- default:
- elog(ERROR, "unexpected table_tuple_lock status: %u", res);
- break;
- }
+ if (should_refetch_tuple(res, &tmfd))
+ goto retry;
}
index_endscan(scan);
@@ -444,34 +464,8 @@ retry:
PopActiveSnapshot();
- switch (res)
- {
- case TM_Ok:
- break;
- case TM_Updated:
- /* XXX: Improve handling here */
- if (ItemPointerIndicatesMovedPartitions(&tmfd.ctid))
- ereport(LOG,
- (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
- errmsg("tuple to be locked was already moved to another partition due to concurrent update, retrying")));
- else
- ereport(LOG,
- (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
- errmsg("concurrent update, retrying")));
- goto retry;
- case TM_Deleted:
- /* XXX: Improve handling here */
- ereport(LOG,
- (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
- errmsg("concurrent delete, retrying")));
- goto retry;
- case TM_Invisible:
- elog(ERROR, "attempted to lock invisible tuple");
- break;
- default:
- elog(ERROR, "unexpected table_tuple_lock status: %u", res);
- break;
- }
+ if (should_refetch_tuple(res, &tmfd))
+ goto retry;
}
table_endscan(scan);
@@ -480,6 +474,95 @@ retry:
return found;
}
+/*
+ * Find the tuple that violates the passed in unique index constraint
+ * (conflictindex).
+ *
+ * If no conflict is found, return false and set *conflictslot to NULL.
+ * Otherwise return true, and the conflicting tuple is locked and returned in
+ * *conflictslot.
+ */
+static bool
+FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
+ Oid conflictindex, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot)
+{
+ Relation rel = resultRelInfo->ri_RelationDesc;
+ ItemPointerData conflictTid;
+ TM_FailureData tmfd;
+ TM_Result res;
+
+ *conflictslot = NULL;
+
+retry:
+ if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
+ &conflictTid, list_make1_oid(conflictindex)))
+ {
+ if (*conflictslot)
+ ExecDropSingleTupleTableSlot(*conflictslot);
+
+ *conflictslot = NULL;
+ return false;
+ }
+
+ *conflictslot = table_slot_create(rel, NULL);
+
+ PushActiveSnapshot(GetLatestSnapshot());
+
+ res = table_tuple_lock(rel, &conflictTid, GetLatestSnapshot(),
+ *conflictslot,
+ GetCurrentCommandId(false),
+ LockTupleShare,
+ LockWaitBlock,
+ 0 /* don't follow updates */ ,
+ &tmfd);
+
+ PopActiveSnapshot();
+
+ if (should_refetch_tuple(res, &tmfd))
+ goto retry;
+
+ return true;
+}
+
+/*
+ * Re-check all the unique indexes in 'recheckIndexes' to see if there are
+ * potential conflicts with the tuple in 'slot'.
+ *
+ * This function is invoked after inserting or updating a tuple that detected
+ * potential conflict tuples. It attempts to find the tuple that conflicts with
+ * the provided tuple. This operation may seem redundant with the unique
+ * violation check of indexam, but since we call this function only when we are
+ * detecting conflict in logical replication and encountering potential
+ * conflicts with any unique index constraints (which should not be frequent),
+ * so it's ok. Moreover, upon detecting a conflict, we will report an ERROR and
+ * restart the logical replication, so the additional cost of finding the tuple
+ * should be acceptable.
+ */
+static void
+ReCheckConflictIndexes(ResultRelInfo *resultRelInfo, EState *estate,
+ ConflictType type, List *recheckIndexes,
+ TupleTableSlot *slot)
+{
+ /* Re-check all the unique indexes for potential conflicts */
+ foreach_oid(uniqueidx, resultRelInfo->ri_onConflictArbiterIndexes)
+ {
+ TupleTableSlot *conflictslot;
+
+ if (list_member_oid(recheckIndexes, uniqueidx) &&
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(ERROR, type, resultRelInfo->ri_RelationDesc, uniqueidx,
+ xmin, origin, committs, conflictslot);
+ }
+ }
+}
+
/*
* Insert tuple represented in the slot to the relation, update the indexes,
* and execute any constraints and per-row triggers.
@@ -509,6 +592,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (!skip_tuple)
{
List *recheckIndexes = NIL;
+ List *conflictindexes;
+ bool conflict = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -525,10 +610,28 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
- slot, estate, false, false,
- NULL, NIL, false);
+ slot, estate, false,
+ conflictindexes ? true : false,
+ &conflict,
+ conflictindexes, false);
+
+ /*
+ * Rechecks the conflict indexes to fetch the conflicting local tuple
+ * and reports the conflict. We perform this check here, instead of
+ * perform an additional index scan before the actual insertion and
+ * reporting the conflict if any conflicting tuples are found. This is
+ * to avoid the overhead of executing the extra scan for each INSERT
+ * operation, even when no conflict arises, which could introduce
+ * significant overhead to replication, particularly in cases where
+ * conflicts are rare.
+ */
+ if (conflict)
+ ReCheckConflictIndexes(resultRelInfo, estate, CT_INSERT_EXISTS,
+ recheckIndexes, slot);
/* AFTER ROW INSERT Triggers */
ExecARInsertTriggers(estate, resultRelInfo, slot,
@@ -577,6 +680,8 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
{
List *recheckIndexes = NIL;
TU_UpdateIndexes update_indexes;
+ List *conflictindexes;
+ bool conflict = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -593,12 +698,24 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
simple_table_tuple_update(rel, tid, slot, estate->es_snapshot,
&update_indexes);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
if (resultRelInfo->ri_NumIndices > 0 && (update_indexes != TU_None))
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
- slot, estate, true, false,
- NULL, NIL,
+ slot, estate, true,
+ conflictindexes ? true : false,
+ &conflict, conflictindexes,
(update_indexes == TU_Summarizing));
+ /*
+ * Refer to the comments above the call to ReCheckConflictIndexes() in
+ * ExecSimpleRelationInsert to understand why this check is done at
+ * this point.
+ */
+ if (conflict)
+ ReCheckConflictIndexes(resultRelInfo, estate, CT_UPDATE_EXISTS,
+ recheckIndexes, slot);
+
/* AFTER ROW UPDATE Triggers */
ExecARUpdateTriggers(estate, resultRelInfo,
NULL, NULL,
diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile
index ba03eeff1c..1e08bbbd4e 100644
--- a/src/backend/replication/logical/Makefile
+++ b/src/backend/replication/logical/Makefile
@@ -16,6 +16,7 @@ override CPPFLAGS := -I$(srcdir) $(CPPFLAGS)
OBJS = \
applyparallelworker.o \
+ conflict.o \
decode.o \
launcher.o \
logical.o \
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
new file mode 100644
index 0000000000..4918011a7f
--- /dev/null
+++ b/src/backend/replication/logical/conflict.c
@@ -0,0 +1,193 @@
+/*-------------------------------------------------------------------------
+ * conflict.c
+ * Functionality for detecting and logging conflicts.
+ *
+ * Copyright (c) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/replication/logical/conflict.c
+ *
+ * This file contains the code for detecting and logging conflicts on
+ * the subscriber during logical replication.
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/commit_ts.h"
+#include "replication/conflict.h"
+#include "replication/origin.h"
+#include "utils/lsyscache.h"
+#include "utils/rel.h"
+
+const char *const ConflictTypeNames[] = {
+ [CT_INSERT_EXISTS] = "insert_exists",
+ [CT_UPDATE_EXISTS] = "update_exists",
+ [CT_UPDATE_DIFFER] = "update_differ",
+ [CT_UPDATE_MISSING] = "update_missing",
+ [CT_DELETE_MISSING] = "delete_missing",
+ [CT_DELETE_DIFFER] = "delete_differ"
+};
+
+static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
+static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin,
+ RepOriginId localorigin,
+ TimestampTz localts,
+ TupleTableSlot *conflictslot);
+
+/*
+ * Get the xmin and commit timestamp data (origin and timestamp) associated
+ * with the provided local tuple.
+ *
+ * Return true if the commit timestamp data was found, false otherwise.
+ */
+bool
+GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts)
+{
+ Datum xminDatum;
+ bool isnull;
+
+ xminDatum = slot_getsysattr(localslot, MinTransactionIdAttributeNumber,
+ &isnull);
+ *xmin = DatumGetTransactionId(xminDatum);
+ Assert(!isnull);
+
+ /*
+ * The commit timestamp data is not available if track_commit_timestamp is
+ * disabled.
+ */
+ if (!track_commit_timestamp)
+ {
+ *localorigin = InvalidRepOriginId;
+ *localts = 0;
+ return false;
+ }
+
+ return TransactionIdGetCommitTsData(*xmin, localts, localorigin);
+}
+
+/*
+ * Report a conflict when applying remote changes.
+ */
+void
+ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot)
+{
+ ereport(elevel,
+ errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
+ errmsg("conflict %s detected on relation \"%s.%s\"",
+ ConflictTypeNames[type],
+ get_namespace_name(RelationGetNamespace(localrel)),
+ RelationGetRelationName(localrel)),
+ errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
+ localts, conflictslot));
+}
+
+/*
+ * Find all unique indexes to check for a conflict and store them into
+ * ResultRelInfo.
+ */
+void
+InitConflictIndexes(ResultRelInfo *relInfo)
+{
+ List *uniqueIndexes = NIL;
+
+ for (int i = 0; i < relInfo->ri_NumIndices; i++)
+ {
+ Relation indexRelation = relInfo->ri_IndexRelationDescs[i];
+
+ if (indexRelation == NULL)
+ continue;
+
+ /* Detect conflict only for unique indexes */
+ if (!relInfo->ri_IndexRelationInfo[i]->ii_Unique)
+ continue;
+
+ /* Don't support conflict detection for deferrable index */
+ if (!indexRelation->rd_index->indimmediate)
+ continue;
+
+ uniqueIndexes = lappend_oid(uniqueIndexes,
+ RelationGetRelid(indexRelation));
+ }
+
+ relInfo->ri_onConflictArbiterIndexes = uniqueIndexes;
+}
+
+/*
+ * Add an errdetail() line showing conflict detail.
+ */
+static int
+errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot)
+{
+ switch (type)
+ {
+ case CT_INSERT_EXISTS:
+ case CT_UPDATE_EXISTS:
+ {
+ /*
+ * Bulid the index value string. If the return value is NULL,
+ * it indicates that the current user lacks permissions to
+ * view all the columns involved.
+ */
+ char *index_value = build_index_value_desc(conflictidx,
+ conflictslot);
+
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
+ case CT_UPDATE_DIFFER:
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ case CT_UPDATE_MISSING:
+ return errdetail("Did not find the row to be updated.");
+ case CT_DELETE_MISSING:
+ return errdetail("Did not find the row to be deleted.");
+ case CT_DELETE_DIFFER:
+ return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ }
+
+ return 0; /* silence compiler warning */
+}
+
+/*
+ * Helper functions to construct a string describing the contents of an index
+ * entry. See BuildIndexValueDescription for details.
+ */
+static char *
+build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
+{
+ char *conflict_row;
+ Relation indexDesc;
+
+ if (!conflictslot)
+ return NULL;
+
+ /* Assume the index has been locked */
+ indexDesc = index_open(indexoid, NoLock);
+
+ slot_getallattrs(conflictslot);
+
+ conflict_row = BuildIndexValueDescription(indexDesc,
+ conflictslot->tts_values,
+ conflictslot->tts_isnull);
+
+ index_close(indexDesc, NoLock);
+
+ return conflict_row;
+}
diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build
index 3dec36a6de..3d36249d8a 100644
--- a/src/backend/replication/logical/meson.build
+++ b/src/backend/replication/logical/meson.build
@@ -2,6 +2,7 @@
backend_sources += files(
'applyparallelworker.c',
+ 'conflict.c',
'decode.c',
'launcher.c',
'logical.c',
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ec96b5fe85..505db3e13c 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -167,6 +167,7 @@
#include "postmaster/bgworker.h"
#include "postmaster/interrupt.h"
#include "postmaster/walwriter.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalproto.h"
#include "replication/logicalrelation.h"
@@ -2458,7 +2459,8 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
EState *estate = edata->estate;
/* We must open indexes here. */
- ExecOpenIndices(relinfo, false);
+ ExecOpenIndices(relinfo, true);
+ InitConflictIndexes(relinfo);
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
@@ -2646,7 +2648,7 @@ apply_handle_update_internal(ApplyExecutionData *edata,
MemoryContext oldctx;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
- ExecOpenIndices(relinfo, false);
+ ExecOpenIndices(relinfo, true);
found = FindReplTupleInLocalRel(edata, localrel,
&relmapentry->remoterel,
@@ -2661,6 +2663,19 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * Check whether the local tuple was modified by a different origin.
+ * If detected, report the conflict.
+ */
+ if (GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2668,6 +2683,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
EvalPlanQualSetSlot(&epqstate, remoteslot);
+ InitConflictIndexes(relinfo);
+
/* Do the actual update. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
@@ -2678,13 +2695,9 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2807,6 +2820,24 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/* If found delete it. */
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * Check whether the local tuple was modified by a different origin.
+ * If detected, report the conflict.
+ *
+ * For cross-partition update, we skip detecting the delete_differ
+ * conflict since it should have been done in
+ * apply_handle_tuple_routing().
+ */
+ if ((!edata->mtstate || edata->mtstate->operation != CMD_UPDATE) &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_DELETE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
EvalPlanQualSetSlot(&epqstate, localslot);
/* Do the actual delete. */
@@ -2818,13 +2849,9 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/*
* The tuple to be deleted could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be deleted "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2991,6 +3018,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ResultRelInfo *partrelinfo_new;
Relation partrel_new;
bool found;
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3002,16 +3032,25 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing
* except for emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation's partition \"%s\"",
- RelationGetRelationName(partrel));
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING,
+ partrel, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+
return;
}
+ /*
+ * Check whether the local tuple was modified by a different
+ * origin. If detected, report the conflict.
+ */
+ if (GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, partrel,
+ InvalidOid, localxmin, localorigin,
+ localts, NULL);
+
/*
* Apply the update to the local tuple, putting the result in
* remoteslot_part.
@@ -3039,7 +3078,8 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
EPQState epqstate;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
- ExecOpenIndices(partrelinfo, false);
+ ExecOpenIndices(partrelinfo, true);
+ InitConflictIndexes(relinfo);
EvalPlanQualSetSlot(&epqstate, remoteslot_part);
TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
new file mode 100644
index 0000000000..3a7260d3c1
--- /dev/null
+++ b/src/include/replication/conflict.h
@@ -0,0 +1,50 @@
+/*-------------------------------------------------------------------------
+ * conflict.h
+ * Exports for conflict detection and log
+ *
+ * Copyright (c) 2012-2024, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef CONFLICT_H
+#define CONFLICT_H
+
+#include "access/xlogdefs.h"
+#include "executor/tuptable.h"
+#include "nodes/execnodes.h"
+#include "utils/relcache.h"
+#include "utils/timestamp.h"
+
+/*
+ * Conflict types that could be encountered when applying remote changes.
+ */
+typedef enum
+{
+ /* The row to be inserted violates unique constraint */
+ CT_INSERT_EXISTS,
+
+ /* The updated row value violates unique constraint */
+ CT_UPDATE_EXISTS,
+
+ /* The row to be updated was modified by a different origin */
+ CT_UPDATE_DIFFER,
+
+ /* The row to be updated is missing */
+ CT_UPDATE_MISSING,
+
+ /* The row to be deleted is missing */
+ CT_DELETE_MISSING,
+
+ /* The row to be deleted was modified by a different origin */
+ CT_DELETE_DIFFER,
+} ConflictType;
+
+extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts);
+extern void ReportApplyConflict(int elevel, ConflictType type,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot);
+extern void InitConflictIndexes(ResultRelInfo *relInfo);
+
+#endif
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 471e981962..79cbed2e5b 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -331,12 +331,6 @@ is( $result, qq(1|bar
2|baz),
'update works with REPLICA IDENTITY FULL and a primary key');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1");
-$node_subscriber->reload;
-
$node_subscriber->safe_psql('postgres', "DELETE FROM tab_full_pk");
# Note that the current location of the log file is not grabbed immediately
@@ -352,10 +346,10 @@ $node_publisher->wait_for_catchup('tap_sub');
my $logfile = slurp_file($node_subscriber->logfile, $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation "tab_full_pk"/,
+ qr/conflict update_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be updated./m,
'update target row is missing');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab_full_pk"/,
+ qr/conflict delete_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be deleted./m,
'delete target row is missing');
$node_subscriber->append_conf('postgresql.conf',
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 29580525a9..8517c189d4 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -343,13 +343,6 @@ $result =
$node_subscriber2->safe_psql('postgres', "SELECT a FROM tab1 ORDER BY 1");
is($result, qq(), 'truncate of tab1 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
-
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 VALUES (1, 'foo'), (4, 'bar'), (10, 'baz')");
@@ -372,22 +365,18 @@ $node_publisher->wait_for_catchup('sub2');
my $logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab1_2_2"/,
+ qr/conflict update_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_1"/,
+ qr/conflict delete_missing detected on relation "public.tab1_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_2_2"/,
+ qr/conflict delete_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_def"/,
+ qr/conflict delete_missing detected on relation "public.tab1_def".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_def');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
-
# Tests for replication using root table identity and schema
# publisher
@@ -773,13 +762,6 @@ pub_tab2|3|yyy
pub_tab2|5|zzz
xxx_c|6|aaa), 'inserts into tab2 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
-
$node_subscriber1->safe_psql('postgres', "DELETE FROM tab2");
# Note that the current location of the log file is not grabbed immediately
@@ -796,15 +778,30 @@ $node_publisher->wait_for_catchup('sub2');
$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab2_1"/,
+ qr/conflict update_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab2_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab2_1"/,
+ qr/conflict delete_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab2_1');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+# Enable the track_commit_timestamp to detect the conflict when attempting
+# to update a row that was previously modified by a different origin.
+$node_subscriber1->append_conf('postgresql.conf', 'track_commit_timestamp = on');
+$node_subscriber1->restart;
+
+$node_subscriber1->safe_psql('postgres', "INSERT INTO tab2 VALUES (3, 'yyy')");
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab2 SET b = 'quux' WHERE a = 3");
+
+$node_publisher->wait_for_catchup('sub_viaroot');
+
+$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
+ok( $logfile =~
+ qr/conflict update_differ detected on relation "public.tab2_1".*\n.*DETAIL:.* Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/,
+ 'updating a tuple that was modified by a different origin');
+
+$node_subscriber1->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_subscriber1->restart;
# Test that replication continues to work correctly after altering the
# partition of a partitioned target table.
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 0ab57a4b5b..3e27a1a7dc 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -30,7 +30,7 @@ sub test_skip_lsn
# ERROR with its CONTEXT when retrieving this information.
my $contents = slurp_file($node_subscriber->logfile, $offset);
$contents =~
- qr/duplicate key value violates unique constraint "tbl_pkey".*\n.*DETAIL:.*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
+ qr/conflict .* detected on relation "public.tbl".*\n.*DETAIL:.* Key \(i\)=\(\d+\) already exists in unique index "tbl_pkey", which was modified by origin \d+ in transaction \d+ at .*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
or die "could not get error-LSN";
my $lsn = $1;
@@ -83,6 +83,7 @@ $node_subscriber->append_conf(
'postgresql.conf',
qq[
max_prepared_transactions = 10
+track_commit_timestamp = on
]);
$node_subscriber->start;
@@ -93,6 +94,7 @@ $node_publisher->safe_psql(
'postgres',
qq[
CREATE TABLE tbl (i INT, t BYTEA);
+ALTER TABLE tbl REPLICA IDENTITY FULL;
INSERT INTO tbl VALUES (1, NULL);
]);
$node_subscriber->safe_psql(
@@ -144,13 +146,14 @@ COMMIT;
test_skip_lsn($node_publisher, $node_subscriber,
"(2, NULL)", "2", "test skipping transaction");
-# Test for PREPARE and COMMIT PREPARED. Insert the same data to tbl and
-# PREPARE the transaction, raising an error. Then skip the transaction.
+# Test for PREPARE and COMMIT PREPARED. Update the data and PREPARE the
+# transaction, raising an error on the subscriber due to violation of the
+# unique constraint on tbl. Then skip the transaction.
$node_publisher->safe_psql(
'postgres',
qq[
BEGIN;
-INSERT INTO tbl VALUES (1, NULL);
+UPDATE tbl SET i = 2;
PREPARE TRANSACTION 'gtx';
COMMIT PREPARED 'gtx';
]);
diff --git a/src/test/subscription/t/030_origin.pl b/src/test/subscription/t/030_origin.pl
index 056561f008..a368818b21 100644
--- a/src/test/subscription/t/030_origin.pl
+++ b/src/test/subscription/t/030_origin.pl
@@ -27,9 +27,14 @@ my $stderr;
my $node_A = PostgreSQL::Test::Cluster->new('node_A');
$node_A->init(allows_streaming => 'logical');
$node_A->start;
+
# node_B
my $node_B = PostgreSQL::Test::Cluster->new('node_B');
$node_B->init(allows_streaming => 'logical');
+
+# Enable the track_commit_timestamp to detect the conflict when attempting to
+# update a row that was previously modified by a different origin.
+$node_B->append_conf('postgresql.conf', 'track_commit_timestamp = on');
$node_B->start;
# Create table on node_A
@@ -139,6 +144,48 @@ is($result, qq(),
'Remote data originating from another node (not the publisher) is not replicated when origin parameter is none'
);
+###############################################################################
+# Check that the conflict can be detected when attempting to update or
+# delete a row that was previously modified by a different source.
+###############################################################################
+
+$node_B->safe_psql('postgres', "DELETE FROM tab;");
+
+$node_A->safe_psql('postgres', "INSERT INTO tab VALUES (32);");
+
+$node_A->wait_for_catchup($subname_BA);
+$node_B->wait_for_catchup($subname_AB);
+
+$result = $node_B->safe_psql('postgres', "SELECT * FROM tab ORDER BY 1;");
+is($result, qq(32), 'The node_A data replicated to node_B');
+
+# The update should update the row on node B that was inserted by node A.
+$node_C->safe_psql('postgres', "UPDATE tab SET a = 33 WHERE a = 32;");
+
+$node_B->wait_for_log(
+ qr/conflict update_differ detected on relation "public.tab".*\n.*DETAIL:.* Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
+$node_B->safe_psql('postgres', "DELETE FROM tab;");
+$node_A->safe_psql('postgres', "INSERT INTO tab VALUES (33);");
+
+$node_A->wait_for_catchup($subname_BA);
+$node_B->wait_for_catchup($subname_AB);
+
+$result = $node_B->safe_psql('postgres', "SELECT * FROM tab ORDER BY 1;");
+is($result, qq(33), 'The node_A data replicated to node_B');
+
+# The delete should remove the row on node B that was inserted by node A.
+$node_C->safe_psql('postgres', "DELETE FROM tab WHERE a = 33;");
+
+$node_B->wait_for_log(
+ qr/conflict delete_differ detected on relation "public.tab".*\n.*DETAIL:.* Deleting a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
+# The remaining tests no longer test conflict detection.
+$node_B->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_B->restart;
+
###############################################################################
# Specifying origin = NONE indicates that the publisher should only replicate the
# changes that are generated locally from node_B, but in this case since the
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3deb6113b8..ae1db1a759 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -467,6 +467,7 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictType
ConnCacheEntry
ConnCacheKey
ConnParams
--
2.30.0.windows.2
[application/octet-stream] v7-0002-Add-a-detect_conflict-option-to-subscriptions.patch (82.0K, ../../OS0PR01MB57167251931CCFD703ADCF1694B72@OS0PR01MB5716.jpnprd01.prod.outlook.com/4-v7-0002-Add-a-detect_conflict-option-to-subscriptions.patch)
download | inline diff:
From 27c7ed549fcf0a79f3ec5a8680178fd5ae863a8d Mon Sep 17 00:00:00 2001
From: Hou Zhijie <[email protected]>
Date: Mon, 29 Jul 2024 14:02:53 +0800
Subject: [PATCH v7 2/3] Add a detect_conflict option to subscriptions
This patch adds a new parameter detect_conflict for CREATE and ALTER
subscription commands. This new parameter will decide if subscription will go
for confict detection and provide additional logging information. To avoid the
potential overhead introduced by conflict detection, detect_conflict will be
off for a subscription by default.
---
doc/src/sgml/catalogs.sgml | 9 +
doc/src/sgml/logical-replication.sgml | 112 +++---------
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 89 ++++++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/subscriptioncmds.c | 54 +++++-
src/backend/replication/logical/worker.c | 58 ++++---
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 6 +-
src/bin/psql/tab-complete.c | 14 +-
src/include/catalog/pg_subscription.h | 4 +
src/test/regress/expected/subscription.out | 188 ++++++++++++---------
src/test/regress/sql/subscription.sql | 19 +++
src/test/subscription/t/001_rep_changes.pl | 7 +
src/test/subscription/t/013_partition.pl | 23 +++
src/test/subscription/t/029_on_error.pl | 2 +-
src/test/subscription/t/030_origin.pl | 7 +-
19 files changed, 413 insertions(+), 206 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index b654fae1b2..b042a5a94a 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8035,6 +8035,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdetectconflict</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription is enabled for conflict detection.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 830c7a9b02..34b420cfcf 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1580,87 +1580,9 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
stop. This is referred to as a <firstterm>conflict</firstterm>. When
replicating <command>UPDATE</command> or <command>DELETE</command>
operations, missing data will not produce a conflict and such operations
- will simply be skipped.
- </para>
-
- <para>
- Additional logging is triggered in the following <firstterm>conflict</firstterm>
- scenarios:
- <variablelist>
- <varlistentry>
- <term><literal>insert_exists</literal></term>
- <listitem>
- <para>
- Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
- unique constraint. Note that to obtain the origin and commit
- timestamp details of the conflicting key in the log, ensure that
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled. In this scenario, an error will be raised until the
- conflict is resolved manually.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>update_exists</literal></term>
- <listitem>
- <para>
- The updated value of a row violates a <literal>NOT DEFERRABLE</literal>
- unique constraint. Note that to obtain the origin and commit
- timestamp details of the conflicting key in the log, ensure that
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled. In this scenario, an error will be raised until the
- conflict is resolved manually. Note that when updating a
- partitioned table, if the updated row value satisfies another
- partition constraint resulting in the row being inserted into a
- new partition, the <literal>insert_exists</literal> conflict may
- arise if the new row violates a <literal>NOT DEFERRABLE</literal>
- unique constraint.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>update_differ</literal></term>
- <listitem>
- <para>
- Updating a row that was previously modified by another origin.
- Note that this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled. Currenly, the update is always applied regardless of
- the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>update_missing</literal></term>
- <listitem>
- <para>
- The tuple to be updated was not found. The update will simply be
- skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>delete_missing</literal></term>
- <listitem>
- <para>
- The tuple to be deleted was not found. The delete will simply be
- skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>delete_differ</literal></term>
- <listitem>
- <para>
- Deleting a row that was previously modified by another origin. Note that this
- conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled. Currenly, the delete is always applied regardless of
- the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
- </variablelist>
+ will simply be skipped. Please refer to
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ for all the conflicts that will be logged when enabling <literal>detect_conflict</literal>.
</para>
<para>
@@ -1689,8 +1611,8 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
an error, the replication won't proceed, and the logical replication worker will
emit the following kind of message to the subscriber's server log:
<screen>
-ERROR: conflict insert_exists detected on relation "test_pkey"
-DETAIL: Key (a)=(1) already exists in unique index "t_pkey", which was modified by origin 0 in transaction 740 at 2024-06-26 10:47:04.727375+08.
+ERROR: duplicate key value violates unique constraint "test_pkey"
+DETAIL: Key (c)=(1) already exists.
CONTEXT: processing remote data for replication origin "pg_16395" during "INSERT" for replication target relation "public.test" in transaction 725 finished at 0/14C0378
</screen>
The LSN of the transaction that contains the change violating the constraint and
@@ -1716,12 +1638,6 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
Please note that skipping the whole transaction includes skipping changes that
might not violate any constraint. This can easily make the subscriber
inconsistent.
- The additional details regarding conflicting rows, such as their origin and
- commit timestamp can be seen in the log as well. Users can use these
- information to make decisions on whether to retain the local change or adopt
- the remote alteration. For instance, the origin in above log indicates that
- the existing row was modified by a local change, users can manually perform
- a remote-change-win resolution by deleting the local row.
</para>
<para>
@@ -1735,6 +1651,24 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ...
SKIP</command></link>.
</para>
+
+ <para>
+ Enabling both <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ and <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ on the subscriber can provide additional details regarding conflicting
+ rows, such as their origin and commit timestamp, in case of a unique
+ constraint violation conflict:
+<screen>
+ERROR: conflict insert_exists detected on relation "public.t"
+DETAIL: Key (a)=(1) already exists in unique index "t_pkey", which was modified by origin 0 in transaction 740 at 2024-06-26 10:47:04.727375+08.
+CONTEXT: processing remote data for replication origin "pg_16389" during message type "INSERT" for replication target relation "public.t" in transaction 740, finished at 0/14F7EC0
+</screen>
+ Users can use these information to make decisions on whether to retain
+ the local change or adopt the remote alteration. For instance, the
+ origin in above log indicates that the existing row was modified by a
+ local change, users can manually perform a remote-change-win resolution
+ by deleting the local row.
+ </para>
</sect1>
<sect1 id="logical-replication-restrictions">
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index fdc648d007..dfbe25b59e 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -235,8 +235,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<link linkend="sql-createsubscription-params-with-password-required"><literal>password_required</literal></link>,
<link linkend="sql-createsubscription-params-with-run-as-owner"><literal>run_as_owner</literal></link>,
<link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>,
- <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>, and
- <link linkend="sql-createsubscription-params-with-two-phase"><literal>two_phase</literal></link>.
+ <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>,
+ <link linkend="sql-createsubscription-params-with-two-phase"><literal>two_phase</literal></link>, and
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>.
Only a superuser can set <literal>password_required = false</literal>.
</para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..d07201abec 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,95 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-detect-conflict">
+ <term><literal>detect_conflict</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription is enabled for conflict detection.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ When conflict detection is enabled, additional logging is triggered
+ in the following scenarios:
+ <variablelist>
+ <varlistentry>
+ <term><literal>insert_exists</literal></term>
+ <listitem>
+ <para>
+ Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint. Note that to obtain the origin and commit
+ timestamp details of the conflicting key in the log, ensure that
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. In this scenario, an error will be raised until the
+ conflict is resolved manually.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_exists</literal></term>
+ <listitem>
+ <para>
+ The updated value of a row violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint. Note that to obtain the origin and commit
+ timestamp details of the conflicting key in the log, ensure that
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. In this scenario, an error will be raised until the
+ conflict is resolved manually. Note that when updating a
+ partitioned table, if the updated row value satisfies another
+ partition constraint resulting in the row being inserted into a
+ new partition, the <literal>insert_exists</literal> conflict may
+ arise if the new row violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_differ</literal></term>
+ <listitem>
+ <para>
+ Updating a row that was previously modified by another origin.
+ Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. Currenly, the update is always applied regardless of
+ the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be updated was not found. The update will simply be
+ skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be deleted was not found. The delete will simply be
+ skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_differ</literal></term>
+ <listitem>
+ <para>
+ Deleting a row that was previously modified by another origin. Note that this
+ conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. Currenly, the delete is always applied regardless of
+ the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..5a423f4fb0 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->detectconflict = subform->subdetectconflict;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 19cabc9a47..d084bfc48a 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1356,7 +1356,8 @@ REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
subbinary, substream, subtwophasestate, subdisableonerr,
subpasswordrequired, subrunasowner, subfailover,
- subslotname, subsynccommit, subpublications, suborigin)
+ subdetectconflict, subslotname, subsynccommit,
+ subpublications, suborigin)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_stats AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index d124bfe55c..a949d246df 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -14,6 +14,7 @@
#include "postgres.h"
+#include "access/commit_ts.h"
#include "access/htup_details.h"
#include "access/table.h"
#include "access/twophase.h"
@@ -71,8 +72,9 @@
#define SUBOPT_PASSWORD_REQUIRED 0x00000800
#define SUBOPT_RUN_AS_OWNER 0x00001000
#define SUBOPT_FAILOVER 0x00002000
-#define SUBOPT_LSN 0x00004000
-#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_DETECT_CONFLICT 0x00004000
+#define SUBOPT_LSN 0x00008000
+#define SUBOPT_ORIGIN 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -98,6 +100,7 @@ typedef struct SubOpts
bool passwordrequired;
bool runasowner;
bool failover;
+ bool detectconflict;
char *origin;
XLogRecPtr lsn;
} SubOpts;
@@ -112,6 +115,7 @@ static List *merge_publications(List *oldpublist, List *newpublist, bool addpub,
static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err);
static void CheckAlterSubOption(Subscription *sub, const char *option,
bool slot_needs_update, bool isTopLevel);
+static void check_conflict_detection(void);
/*
@@ -162,6 +166,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->runasowner = false;
if (IsSet(supported_opts, SUBOPT_FAILOVER))
opts->failover = false;
+ if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT))
+ opts->detectconflict = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
@@ -307,6 +313,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_FAILOVER;
opts->failover = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT) &&
+ strcmp(defel->defname, "detect_conflict") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DETECT_CONFLICT))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DETECT_CONFLICT;
+ opts->detectconflict = defGetBoolean(defel);
+ }
else if (IsSet(supported_opts, SUBOPT_ORIGIN) &&
strcmp(defel->defname, "origin") == 0)
{
@@ -594,7 +609,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -639,6 +655,9 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
errmsg("password_required=false is superuser-only"),
errhint("Subscriptions with the password_required option set to false may only be created or modified by the superuser.")));
+ if (opts.detectconflict)
+ check_conflict_detection();
+
/*
* If built with appropriate switch, whine when regression-testing
* conventions for subscription names are violated.
@@ -701,6 +720,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_subpasswordrequired - 1] = BoolGetDatum(opts.passwordrequired);
values[Anum_pg_subscription_subrunasowner - 1] = BoolGetDatum(opts.runasowner);
values[Anum_pg_subscription_subfailover - 1] = BoolGetDatum(opts.failover);
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -1196,7 +1217,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
SUBOPT_DISABLE_ON_ERR |
SUBOPT_PASSWORD_REQUIRED |
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
- SUBOPT_ORIGIN);
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1356,6 +1377,16 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_subfailover - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DETECT_CONFLICT))
+ {
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
+ replaces[Anum_pg_subscription_subdetectconflict - 1] = true;
+
+ if (opts.detectconflict)
+ check_conflict_detection();
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
{
values[Anum_pg_subscription_suborigin - 1] =
@@ -2536,3 +2567,18 @@ defGetStreamingMode(DefElem *def)
def->defname)));
return LOGICALREP_STREAM_OFF; /* keep compiler quiet */
}
+
+/*
+ * Report a warning about incomplete conflict detection if
+ * track_commit_timestamp is disabled.
+ */
+static void
+check_conflict_detection(void)
+{
+ if (!track_commit_timestamp)
+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("conflict detection could be incomplete due to disabled track_commit_timestamp"),
+ errdetail("Conflicts update_differ and delete_differ cannot be detected, "
+ "and the origin and commit timestamp for the local row will not be logged."));
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 505db3e13c..df9a9eb0d7 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2459,8 +2459,10 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
EState *estate = edata->estate;
/* We must open indexes here. */
- ExecOpenIndices(relinfo, true);
- InitConflictIndexes(relinfo);
+ ExecOpenIndices(relinfo, MySubscription->detectconflict);
+
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
@@ -2648,7 +2650,7 @@ apply_handle_update_internal(ApplyExecutionData *edata,
MemoryContext oldctx;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
- ExecOpenIndices(relinfo, true);
+ ExecOpenIndices(relinfo, MySubscription->detectconflict);
found = FindReplTupleInLocalRel(edata, localrel,
&relmapentry->remoterel,
@@ -2668,10 +2670,11 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TimestampTz localts;
/*
- * Check whether the local tuple was modified by a different origin.
- * If detected, report the conflict.
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
*/
- if (GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
localxmin, localorigin, localts, NULL);
@@ -2683,7 +2686,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
EvalPlanQualSetSlot(&epqstate, remoteslot);
- InitConflictIndexes(relinfo);
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
/* Do the actual update. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
@@ -2696,8 +2700,9 @@ apply_handle_update_internal(ApplyExecutionData *edata,
* The tuple to be updated could not be found. Do nothing except for
* emitting a log message.
*/
- ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
- InvalidTransactionId, InvalidRepOriginId, 0, NULL);
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2825,14 +2830,15 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
TimestampTz localts;
/*
- * Check whether the local tuple was modified by a different origin.
- * If detected, report the conflict.
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
*
* For cross-partition update, we skip detecting the delete_differ
* conflict since it should have been done in
* apply_handle_tuple_routing().
*/
- if ((!edata->mtstate || edata->mtstate->operation != CMD_UPDATE) &&
+ if (MySubscription->detectconflict &&
+ (!edata->mtstate || edata->mtstate->operation != CMD_UPDATE) &&
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
ReportApplyConflict(LOG, CT_DELETE_DIFFER, localrel, InvalidOid,
@@ -2850,8 +2856,9 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
* The tuple to be deleted could not be found. Do nothing except for
* emitting a log message.
*/
- ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
- InvalidTransactionId, InvalidRepOriginId, 0, NULL);
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -3033,19 +3040,22 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
* The tuple to be updated could not be found. Do nothing
* except for emitting a log message.
*/
- ReportApplyConflict(LOG, CT_UPDATE_MISSING,
- partrel, InvalidOid,
- InvalidTransactionId,
- InvalidRepOriginId, 0, NULL);
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING,
+ partrel, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
return;
}
/*
- * Check whether the local tuple was modified by a different
- * origin. If detected, report the conflict.
+ * If conflict detection is enabled, check whether the local
+ * tuple was modified by a different origin. If detected,
+ * report the conflict.
*/
- if (GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
ReportApplyConflict(LOG, CT_UPDATE_DIFFER, partrel,
InvalidOid, localxmin, localorigin,
@@ -3078,8 +3088,10 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
EPQState epqstate;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
- ExecOpenIndices(partrelinfo, true);
- InitConflictIndexes(relinfo);
+ ExecOpenIndices(partrelinfo, MySubscription->detectconflict);
+
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
EvalPlanQualSetSlot(&epqstate, remoteslot_part);
TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index b8b1888bd3..829f9b3e88 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4760,6 +4760,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subdetectconflict;
int i,
ntups;
@@ -4832,11 +4833,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subdetectconflict\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdetectconflict\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4875,6 +4882,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subdetectconflict = PQfnumber(res, "subdetectconflict");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4921,6 +4929,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subdetectconflict =
+ pg_strdup(PQgetvalue(res, i, i_subdetectconflict));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5161,6 +5171,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subfailover, "t") == 0)
appendPQExpBufferStr(query, ", failover = true");
+ if (strcmp(subinfo->subdetectconflict, "t") == 0)
+ appendPQExpBufferStr(query, ", detect_conflict = true");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e5870a9..bbd7cbeff6 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subdetectconflict;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f234c..fef1ad0d70 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6607,6 +6607,10 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subdetectconflict AS \"%s\"\n",
+ gettext_noop("Detect conflict"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 891face1b6..086e135f65 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1946,9 +1946,10 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "disable_on_error", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit", "two_phase");
+ COMPLETE_WITH("binary", "detect_conflict", "disable_on_error",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit", "two_phase");
/* ALTER SUBSCRIPTION <name> SKIP ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SKIP", "("))
COMPLETE_WITH("lsn");
@@ -3363,9 +3364,10 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit", "two_phase");
+ "detect_conflict", "disable_on_error", "enabled",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit", "two_phase");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..17daf11dc7 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subdetectconflict; /* True if replication should perform
+ * conflict detection */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -151,6 +154,7 @@ typedef struct Subscription
* (i.e. the main slot and the table sync
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool detectconflict; /* True if conflict detection is enabled */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1685..118f207df5 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -116,18 +116,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +145,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +157,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +176,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +188,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +223,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +255,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +279,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +314,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +332,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,19 +371,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- we can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -393,10 +393,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -409,18 +409,54 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+ERROR: detect_conflict requires a Boolean value
+-- now it works, but will report a warning due to disabled track_commit_timestamp
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+WARNING: conflict detection could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | t | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = true);
+WARNING: conflict detection could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | t | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7037..b3f2ab1684 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -287,6 +287,25 @@ ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+
+-- now it works, but will report a warning due to disabled track_commit_timestamp
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- let's do some tests with pg_create_subscription rather than superuser
SET SESSION AUTHORIZATION regress_subscription_user3;
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 79cbed2e5b..78c0307165 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -331,6 +331,13 @@ is( $result, qq(1|bar
2|baz),
'update works with REPLICA IDENTITY FULL and a primary key');
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (detect_conflict = true)"
+);
+
$node_subscriber->safe_psql('postgres', "DELETE FROM tab_full_pk");
# Note that the current location of the log file is not grabbed immediately
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 8517c189d4..b3044a2651 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -343,6 +343,13 @@ $result =
$node_subscriber2->safe_psql('postgres', "SELECT a FROM tab1 ORDER BY 1");
is($result, qq(), 'truncate of tab1 replicated');
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = true)"
+);
+
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 VALUES (1, 'foo'), (4, 'bar'), (10, 'baz')");
@@ -377,6 +384,10 @@ ok( $logfile =~
qr/conflict delete_missing detected on relation "public.tab1_def".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_def');
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = false)"
+);
+
# Tests for replication using root table identity and schema
# publisher
@@ -762,6 +773,13 @@ pub_tab2|3|yyy
pub_tab2|5|zzz
xxx_c|6|aaa), 'inserts into tab2 replicated');
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = true)"
+);
+
$node_subscriber1->safe_psql('postgres', "DELETE FROM tab2");
# Note that the current location of the log file is not grabbed immediately
@@ -800,6 +818,11 @@ ok( $logfile =~
qr/conflict update_differ detected on relation "public.tab2_1".*\n.*DETAIL:.* Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/,
'updating a tuple that was modified by a different origin');
+# The remaining tests no longer test conflict detection.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = false)"
+);
+
$node_subscriber1->append_conf('postgresql.conf', 'track_commit_timestamp = off');
$node_subscriber1->restart;
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 3e27a1a7dc..ec1a48d0f6 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -111,7 +111,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub FOR TABLE tbl");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on)"
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
# Initial synchronization failure causes the subscription to be disabled.
diff --git a/src/test/subscription/t/030_origin.pl b/src/test/subscription/t/030_origin.pl
index a368818b21..69a1ef03a1 100644
--- a/src/test/subscription/t/030_origin.pl
+++ b/src/test/subscription/t/030_origin.pl
@@ -149,7 +149,9 @@ is($result, qq(),
# delete a row that was previously modified by a different source.
###############################################################################
-$node_B->safe_psql('postgres', "DELETE FROM tab;");
+$node_B->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_BC SET (detect_conflict = true);
+ DELETE FROM tab;");
$node_A->safe_psql('postgres', "INSERT INTO tab VALUES (32);");
@@ -183,6 +185,9 @@ $node_B->wait_for_log(
);
# The remaining tests no longer test conflict detection.
+$node_B->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_BC SET (detect_conflict = false);");
+
$node_B->append_conf('postgresql.conf', 'track_commit_timestamp = off');
$node_B->restart;
--
2.30.0.windows.2
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Conflict detection and logging in logical replication
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-19 08:36 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-22 09:03 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-25 06:34 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-25 10:42 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
2024-07-26 11:33 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
2024-07-29 06:14 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
@ 2024-07-29 09:24 ` Amit Kapila <[email protected]>
1 sibling, 0 replies; 50+ messages in thread
From: Amit Kapila @ 2024-07-29 09:24 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: shveta malik <[email protected]>; Nisha Moond <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Ashutosh Bapat <[email protected]>
On Mon, Jul 29, 2024 at 11:44 AM Zhijie Hou (Fujitsu)
<[email protected]> wrote:
>
> On Friday, July 26, 2024 7:34 PM Amit Kapila <[email protected]> wrote:
> >
> > On Thu, Jul 25, 2024 at 4:12 PM Amit Kapila <[email protected]>
> > wrote:
> > > >
> > A few more comments:
>
> Thanks for the comments.
>
> > 1.
> > For duplicate key, the patch reports conflict as following:
> > ERROR: conflict insert_exists detected on relation "public.t1"
> > 2024-07-26 11:06:34.570 IST [27800] DETAIL: Key (c1)=(1) already exists in
> > unique index "t1_pkey", which was modified by origin 1 in transaction 770 at
> > 2024-07-26 09:16:47.79805+05:30.
> > 2024-07-26 11:06:34.570 IST [27800] CONTEXT: processing remote data for
> > replication origin "pg_16387" during message type "INSERT" for replication
> > target relation "public.t1" in transaction 742, finished at 0/151A108
> >
> > In detail, it is better to display the origin name instead of the origin id. This will
> > be similar to what we do in CONTEXT information.
>
>
> Agreed. Before modifying this, I'd like to confirm the message style in the
> cases where origin id may not have a corresponding origin name (e.g., if the
> data was modified locally (id = 0), or if the origin that modified the data has
> been dropped). I thought of two styles:
>
> 1)
> - for local change: "xxx was modified by a different origin \"(local)\" in transaction 123 at 2024.."
> - for dropped origin: "xxx was modified by a different origin \"(unknown)\" in transaction 123 at 2024.."
>
> One issue for this style is that user may create an origin with the same name
> here (e.g. "(local)" and "(unknown)").
>
> 2)
> - for local change: "xxx was modified locally in transaction 123 at 2024.."
>
This sounds good.
> - for dropped origin: "xxx was modified by an unknown different origin 1234 in transaction 123 at 2024.."
>
For this one, how about: "xxx was modified by a non-existent origin in
transaction 123 at 2024.."?
Also, in code please do write comments when each of these two can occur.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Conflict detection and logging in logical replication
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-19 08:36 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-22 09:03 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-25 06:34 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-25 10:42 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
2024-07-26 11:33 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
2024-07-29 06:14 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
@ 2024-07-29 10:59 ` Dilip Kumar <[email protected]>
2024-07-29 12:14 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
1 sibling, 1 reply; 50+ messages in thread
From: Dilip Kumar @ 2024-07-29 10:59 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Nisha Moond <[email protected]>; pgsql-hackers; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Ashutosh Bapat <[email protected]>
On Mon, Jul 29, 2024 at 11:44 AM Zhijie Hou (Fujitsu)
<[email protected]> wrote:
>
I was going through v7-0001, and I have some initial comments.
@@ -536,11 +542,9 @@ ExecCheckIndexConstraints(ResultRelInfo
*resultRelInfo, TupleTableSlot *slot,
ExprContext *econtext;
Datum values[INDEX_MAX_KEYS];
bool isnull[INDEX_MAX_KEYS];
- ItemPointerData invalidItemPtr;
bool checkedIndex = false;
ItemPointerSetInvalid(conflictTid);
- ItemPointerSetInvalid(&invalidItemPtr);
/*
* Get information from the result relation info structure.
@@ -629,7 +633,7 @@ ExecCheckIndexConstraints(ResultRelInfo
*resultRelInfo, TupleTableSlot *slot,
satisfiesConstraint =
check_exclusion_or_unique_constraint(heapRelation, indexRelation,
- indexInfo, &invalidItemPtr,
+ indexInfo, &slot->tts_tid,
values, isnull, estate, false,
CEOUC_WAIT, true,
conflictTid);
What is the purpose of this change? I mean
'check_exclusion_or_unique_constraint' says that 'tupleid'
should be invalidItemPtr if the tuple is not yet inserted and
ExecCheckIndexConstraints is called by ExecInsert
before inserting the tuple. So what is this change? Would this change
ExecInsert's behavior as well?
----
----
+ReCheckConflictIndexes(ResultRelInfo *resultRelInfo, EState *estate,
+ ConflictType type, List *recheckIndexes,
+ TupleTableSlot *slot)
+{
+ /* Re-check all the unique indexes for potential conflicts */
+ foreach_oid(uniqueidx, resultRelInfo->ri_onConflictArbiterIndexes)
+ {
+ TupleTableSlot *conflictslot;
+
+ if (list_member_oid(recheckIndexes, uniqueidx) &&
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(ERROR, type, resultRelInfo->ri_RelationDesc, uniqueidx,
+ xmin, origin, committs, conflictslot);
+ }
+ }
+}
and
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
- slot, estate, false, false,
- NULL, NIL, false);
+ slot, estate, false,
+ conflictindexes ? true : false,
+ &conflict,
+ conflictindexes, false);
+
+ /*
+ * Rechecks the conflict indexes to fetch the conflicting local tuple
+ * and reports the conflict. We perform this check here, instead of
+ * perform an additional index scan before the actual insertion and
+ * reporting the conflict if any conflicting tuples are found. This is
+ * to avoid the overhead of executing the extra scan for each INSERT
+ * operation, even when no conflict arises, which could introduce
+ * significant overhead to replication, particularly in cases where
+ * conflicts are rare.
+ */
+ if (conflict)
+ ReCheckConflictIndexes(resultRelInfo, estate, CT_INSERT_EXISTS,
+ recheckIndexes, slot);
This logic is confusing, first, you are calling
ExecInsertIndexTuples() with no duplicate error for the indexes
present in 'ri_onConflictArbiterIndexes' which means
the indexes returned by the function must be a subset of
'ri_onConflictArbiterIndexes' and later in ReCheckConflictIndexes()
you are again processing all the
indexes of 'ri_onConflictArbiterIndexes' and checking if any of these
is a subset of the indexes that is returned by
ExecInsertIndexTuples().
Why are we doing that, I think we can directly use the
'recheckIndexes' which is returned by ExecInsertIndexTuples(), and
those indexes are guaranteed to be a subset of
ri_onConflictArbiterIndexes. No?
---
---
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 50+ messages in thread
* RE: Conflict detection and logging in logical replication
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-19 08:36 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-22 09:03 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-25 06:34 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-25 10:42 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
2024-07-26 11:33 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
2024-07-29 06:14 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-29 10:59 ` Re: Conflict detection and logging in logical replication Dilip Kumar <[email protected]>
@ 2024-07-29 12:14 ` Zhijie Hou (Fujitsu) <[email protected]>
2024-07-30 08:19 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
0 siblings, 1 reply; 50+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-07-29 12:14 UTC (permalink / raw)
To: Dilip Kumar <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Nisha Moond <[email protected]>; pgsql-hackers; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Ashutosh Bapat <[email protected]>
On Monday, July 29, 2024 6:59 PM Dilip Kumar <[email protected]> wrote:
>
> On Mon, Jul 29, 2024 at 11:44 AM Zhijie Hou (Fujitsu)
> <[email protected]> wrote:
> >
>
> I was going through v7-0001, and I have some initial comments.
Thanks for the comments !
>
> @@ -536,11 +542,9 @@ ExecCheckIndexConstraints(ResultRelInfo
> *resultRelInfo, TupleTableSlot *slot,
> ExprContext *econtext;
> Datum values[INDEX_MAX_KEYS];
> bool isnull[INDEX_MAX_KEYS];
> - ItemPointerData invalidItemPtr;
> bool checkedIndex = false;
>
> ItemPointerSetInvalid(conflictTid);
> - ItemPointerSetInvalid(&invalidItemPtr);
>
> /*
> * Get information from the result relation info structure.
> @@ -629,7 +633,7 @@ ExecCheckIndexConstraints(ResultRelInfo
> *resultRelInfo, TupleTableSlot *slot,
>
> satisfiesConstraint =
> check_exclusion_or_unique_constraint(heapRelation, indexRelation,
> - indexInfo, &invalidItemPtr,
> + indexInfo, &slot->tts_tid,
> values, isnull, estate, false,
> CEOUC_WAIT, true,
> conflictTid);
>
> What is the purpose of this change? I mean
> 'check_exclusion_or_unique_constraint' says that 'tupleid'
> should be invalidItemPtr if the tuple is not yet inserted and
> ExecCheckIndexConstraints is called by ExecInsert before inserting the tuple.
> So what is this change?
Because the function ExecCheckIndexConstraints() is now invoked after inserting
a tuple (in the patch). So, we need to ignore the newly inserted tuple when
checking conflict in check_exclusion_or_unique_constraint().
> Would this change ExecInsert's behavior as well?
Thanks for pointing it out, I will check and reply.
>
> ----
> ----
>
> +ReCheckConflictIndexes(ResultRelInfo *resultRelInfo, EState *estate,
> + ConflictType type, List *recheckIndexes,
> + TupleTableSlot *slot)
> +{
> + /* Re-check all the unique indexes for potential conflicts */
> +foreach_oid(uniqueidx, resultRelInfo->ri_onConflictArbiterIndexes)
> + {
> + TupleTableSlot *conflictslot;
> +
> + if (list_member_oid(recheckIndexes, uniqueidx) &&
> + FindConflictTuple(resultRelInfo, estate, uniqueidx, slot,
> + &conflictslot)) { RepOriginId origin; TimestampTz committs;
> + TransactionId xmin;
> +
> + GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
> +ReportApplyConflict(ERROR, type, resultRelInfo->ri_RelationDesc,
> +uniqueidx, xmin, origin, committs, conflictslot); } } }
> and
>
> + conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
> +
> if (resultRelInfo->ri_NumIndices > 0)
> recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
> - slot, estate, false, false,
> - NULL, NIL, false);
> + slot, estate, false,
> + conflictindexes ? true : false,
> + &conflict,
> + conflictindexes, false);
> +
> + /*
> + * Rechecks the conflict indexes to fetch the conflicting local tuple
> + * and reports the conflict. We perform this check here, instead of
> + * perform an additional index scan before the actual insertion and
> + * reporting the conflict if any conflicting tuples are found. This is
> + * to avoid the overhead of executing the extra scan for each INSERT
> + * operation, even when no conflict arises, which could introduce
> + * significant overhead to replication, particularly in cases where
> + * conflicts are rare.
> + */
> + if (conflict)
> + ReCheckConflictIndexes(resultRelInfo, estate, CT_INSERT_EXISTS,
> + recheckIndexes, slot);
>
>
> This logic is confusing, first, you are calling
> ExecInsertIndexTuples() with no duplicate error for the indexes
> present in 'ri_onConflictArbiterIndexes' which means
> the indexes returned by the function must be a subset of
> 'ri_onConflictArbiterIndexes' and later in ReCheckConflictIndexes()
> you are again processing all the
> indexes of 'ri_onConflictArbiterIndexes' and checking if any of these
> is a subset of the indexes that is returned by
> ExecInsertIndexTuples().
I think that's not always true. The indexes returned by the function *may not*
be a subset of 'ri_onConflictArbiterIndexes'. Based on the comments atop of the
ExecInsertIndexTuples, it returns a list of index OIDs for any unique or
exclusion constraints that are deferred, and in addition to that, it will
include the indexes in 'arbiterIndexes' if noDupErr == true.
>
> Why are we doing that, I think we can directly use the
> 'recheckIndexes' which is returned by ExecInsertIndexTuples(), and
> those indexes are guaranteed to be a subset of
> ri_onConflictArbiterIndexes. No?
Based on above, we need to filter the deferred indexes or exclusion constraints
in the 'ri_onConflictArbiterIndexes'.
Best Regards,
Hou zj
^ permalink raw reply [nested|flat] 50+ messages in thread
* RE: Conflict detection and logging in logical replication
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-19 08:36 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-22 09:03 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-25 06:34 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-25 10:42 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
2024-07-26 11:33 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
2024-07-29 06:14 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-29 10:59 ` Re: Conflict detection and logging in logical replication Dilip Kumar <[email protected]>
2024-07-29 12:14 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
@ 2024-07-30 08:19 ` Zhijie Hou (Fujitsu) <[email protected]>
2024-07-30 08:41 ` Re: Conflict detection and logging in logical replication Dilip Kumar <[email protected]>
0 siblings, 1 reply; 50+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2024-07-30 08:19 UTC (permalink / raw)
To: Dilip Kumar <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Nisha Moond <[email protected]>; pgsql-hackers; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Ashutosh Bapat <[email protected]>
> On Monday, July 29, 2024 6:59 PM Dilip Kumar <[email protected]>
> wrote:
> >
> > On Mon, Jul 29, 2024 at 11:44 AM Zhijie Hou (Fujitsu)
> > <[email protected]> wrote:
> > >
> >
> > I was going through v7-0001, and I have some initial comments.
>
> Thanks for the comments !
>
> >
> > @@ -536,11 +542,9 @@ ExecCheckIndexConstraints(ResultRelInfo
> > *resultRelInfo, TupleTableSlot *slot,
> > ExprContext *econtext;
> > Datum values[INDEX_MAX_KEYS];
> > bool isnull[INDEX_MAX_KEYS];
> > - ItemPointerData invalidItemPtr;
> > bool checkedIndex = false;
> >
> > ItemPointerSetInvalid(conflictTid);
> > - ItemPointerSetInvalid(&invalidItemPtr);
> >
> > /*
> > * Get information from the result relation info structure.
> > @@ -629,7 +633,7 @@ ExecCheckIndexConstraints(ResultRelInfo
> > *resultRelInfo, TupleTableSlot *slot,
> >
> > satisfiesConstraint =
> > check_exclusion_or_unique_constraint(heapRelation, indexRelation,
> > - indexInfo, &invalidItemPtr,
> > + indexInfo, &slot->tts_tid,
> > values, isnull, estate, false,
> > CEOUC_WAIT, true,
> > conflictTid);
> >
> > What is the purpose of this change? I mean
> > 'check_exclusion_or_unique_constraint' says that 'tupleid'
> > should be invalidItemPtr if the tuple is not yet inserted and
> > ExecCheckIndexConstraints is called by ExecInsert before inserting the
> tuple.
> > So what is this change?
>
> Because the function ExecCheckIndexConstraints() is now invoked after
> inserting a tuple (in the patch). So, we need to ignore the newly inserted tuple
> when checking conflict in check_exclusion_or_unique_constraint().
>
> > Would this change ExecInsert's behavior as well?
>
> Thanks for pointing it out, I will check and reply.
After checking, I think it may affect ExecInsert's behavior if the slot passed
to ExecCheckIndexConstraints() comes from other tables (e.g. when executing
INSERT INTO SELECT FROM othertbl), because the slot->tts_tid points to a valid
position from another table in this case, which can cause the
check_exclusion_or_unique_constraint to skip a tuple unexpectedly).
I thought about two ideas to fix this: One is to reset the slot->tts_tid before
calling ExecCheckIndexConstraints() in ExecInsert(), but I feel a bit
uncomfortable to this since it is touching existing logic. So, another idea is to
just add a new parameter 'tupletid' in ExecCheckIndexConstraints(), then pass
tupletid=InvalidOffsetNumber in when invoke the function in ExecInsert() and
pass a valid tupletid in the new code paths in the patch. The new
'tupletid' will be passed to check_exclusion_or_unique_constraint to
skip the target tuple. I feel the second one maybe better.
What do you think ?
Best Regards,
Hou zj
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Conflict detection and logging in logical replication
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-19 08:36 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-22 09:03 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-25 06:34 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-25 10:42 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
2024-07-26 11:33 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
2024-07-29 06:14 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-29 10:59 ` Re: Conflict detection and logging in logical replication Dilip Kumar <[email protected]>
2024-07-29 12:14 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-30 08:19 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
@ 2024-07-30 08:41 ` Dilip Kumar <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: Dilip Kumar @ 2024-07-30 08:41 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Nisha Moond <[email protected]>; pgsql-hackers; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Ashutosh Bapat <[email protected]>
On Tue, Jul 30, 2024 at 1:49 PM Zhijie Hou (Fujitsu)
<[email protected]> wrote:
>
> > On Monday, July 29, 2024 6:59 PM Dilip Kumar <[email protected]>
> > wrote:
> > >
> > > On Mon, Jul 29, 2024 at 11:44 AM Zhijie Hou (Fujitsu)
> > > <[email protected]> wrote:
> > > >
> > >
> > > I was going through v7-0001, and I have some initial comments.
> >
> > Thanks for the comments !
> >
> > >
> > > @@ -536,11 +542,9 @@ ExecCheckIndexConstraints(ResultRelInfo
> > > *resultRelInfo, TupleTableSlot *slot,
> > > ExprContext *econtext;
> > > Datum values[INDEX_MAX_KEYS];
> > > bool isnull[INDEX_MAX_KEYS];
> > > - ItemPointerData invalidItemPtr;
> > > bool checkedIndex = false;
> > >
> > > ItemPointerSetInvalid(conflictTid);
> > > - ItemPointerSetInvalid(&invalidItemPtr);
> > >
> > > /*
> > > * Get information from the result relation info structure.
> > > @@ -629,7 +633,7 @@ ExecCheckIndexConstraints(ResultRelInfo
> > > *resultRelInfo, TupleTableSlot *slot,
> > >
> > > satisfiesConstraint =
> > > check_exclusion_or_unique_constraint(heapRelation, indexRelation,
> > > - indexInfo, &invalidItemPtr,
> > > + indexInfo, &slot->tts_tid,
> > > values, isnull, estate, false,
> > > CEOUC_WAIT, true,
> > > conflictTid);
> > >
> > > What is the purpose of this change? I mean
> > > 'check_exclusion_or_unique_constraint' says that 'tupleid'
> > > should be invalidItemPtr if the tuple is not yet inserted and
> > > ExecCheckIndexConstraints is called by ExecInsert before inserting the
> > tuple.
> > > So what is this change?
> >
> > Because the function ExecCheckIndexConstraints() is now invoked after
> > inserting a tuple (in the patch). So, we need to ignore the newly inserted tuple
> > when checking conflict in check_exclusion_or_unique_constraint().
> >
> > > Would this change ExecInsert's behavior as well?
> >
> > Thanks for pointing it out, I will check and reply.
>
> After checking, I think it may affect ExecInsert's behavior if the slot passed
> to ExecCheckIndexConstraints() comes from other tables (e.g. when executing
> INSERT INTO SELECT FROM othertbl), because the slot->tts_tid points to a valid
> position from another table in this case, which can cause the
> check_exclusion_or_unique_constraint to skip a tuple unexpectedly).
>
> I thought about two ideas to fix this: One is to reset the slot->tts_tid before
> calling ExecCheckIndexConstraints() in ExecInsert(), but I feel a bit
> uncomfortable to this since it is touching existing logic. So, another idea is to
> just add a new parameter 'tupletid' in ExecCheckIndexConstraints(), then pass
> tupletid=InvalidOffsetNumber in when invoke the function in ExecInsert() and
> pass a valid tupletid in the new code paths in the patch. The new
> 'tupletid' will be passed to check_exclusion_or_unique_constraint to
> skip the target tuple. I feel the second one maybe better.
>
> What do you think ?
Yes, the second approach seems good to me.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Conflict detection and logging in logical replication
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-19 08:36 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-22 09:03 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-25 06:34 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
@ 2024-07-26 09:33 ` Nisha Moond <[email protected]>
2024-07-26 10:07 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
1 sibling, 1 reply; 50+ messages in thread
From: Nisha Moond @ 2024-07-26 09:33 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: shveta malik <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Amit Kapila <[email protected]>; Ashutosh Bapat <[email protected]>
On Thu, Jul 25, 2024 at 12:04 PM Zhijie Hou (Fujitsu)
<[email protected]> wrote:
> Here is the V6 patch set which addressed Shveta and Nisha's comments
> in [1][2][3][4].
Thanks for the patch.
I tested the v6-0001 patch with partition table scenarios. Please
review the following scenario where Pub updates a tuple, causing it to
move from one partition to another on Sub.
Setup:
Pub:
create table tab (a int not null, b int not null);
alter table tab add constraint tab_pk primary key (a,b);
Sub:
create table tab (a int not null, b int not null) partition by range (b);
alter table tab add constraint tab_pk primary key (a,b);
create table tab_1 partition of tab FOR values from (MINVALUE) TO (100);
create table tab_2 partition of tab FOR values from (101) TO (MAXVALUE);
Test:
Pub: insert into tab values (1,1);
Sub: update tab set a=1 where a=1; > just to make it Sub's origin
Sub: insert into tab values (1,101);
Pub: update b=101 where b=1; --> Both 'update_differ' and
'insert_exists' are detected.
For non-partitioned tables, a similar update results in
'update_differ' and 'update_exists' conflicts. After detecting
'update_differ', the apply worker proceeds to apply the remote update
and if a tuple with the updated key already exists, it raises
'update_exists'.
This same behavior is expected for partitioned tables too.
Thanks,
Nisha
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Conflict detection and logging in logical replication
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-19 08:36 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-22 09:03 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-25 06:34 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-26 09:33 ` Re: Conflict detection and logging in logical replication Nisha Moond <[email protected]>
@ 2024-07-26 10:07 ` shveta malik <[email protected]>
2024-07-26 10:25 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
0 siblings, 1 reply; 50+ messages in thread
From: shveta malik @ 2024-07-26 10:07 UTC (permalink / raw)
To: Nisha Moond <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Amit Kapila <[email protected]>; Ashutosh Bapat <[email protected]>; shveta malik <[email protected]>
On Fri, Jul 26, 2024 at 3:03 PM Nisha Moond <[email protected]> wrote:
>
> On Thu, Jul 25, 2024 at 12:04 PM Zhijie Hou (Fujitsu)
> <[email protected]> wrote:
> > Here is the V6 patch set which addressed Shveta and Nisha's comments
> > in [1][2][3][4].
>
> Thanks for the patch.
> I tested the v6-0001 patch with partition table scenarios. Please
> review the following scenario where Pub updates a tuple, causing it to
> move from one partition to another on Sub.
>
> Setup:
> Pub:
> create table tab (a int not null, b int not null);
> alter table tab add constraint tab_pk primary key (a,b);
> Sub:
> create table tab (a int not null, b int not null) partition by range (b);
> alter table tab add constraint tab_pk primary key (a,b);
> create table tab_1 partition of tab FOR values from (MINVALUE) TO (100);
> create table tab_2 partition of tab FOR values from (101) TO (MAXVALUE);
>
> Test:
> Pub: insert into tab values (1,1);
> Sub: update tab set a=1 where a=1; > just to make it Sub's origin
> Sub: insert into tab values (1,101);
> Pub: update b=101 where b=1; --> Both 'update_differ' and
> 'insert_exists' are detected.
>
> For non-partitioned tables, a similar update results in
> 'update_differ' and 'update_exists' conflicts. After detecting
> 'update_differ', the apply worker proceeds to apply the remote update
> and if a tuple with the updated key already exists, it raises
> 'update_exists'.
> This same behavior is expected for partitioned tables too.
Good catch. Yes, from the user's perspective, an update_* conflict
should be raised when performing an update operation. But internally
since we are deleting from one partition and inserting to another, we
are hitting insert_exist. To convert this insert_exist to udpate_exist
conflict, perhaps we need to change insert-operation to
update-operation as the default resolver is 'always apply update' in
case of update_differ. But not sure how much complexity it will add to
the code. If it makes the code too complex, I think we can retain the
existing behaviour but document this multi-partition case. And in the
resolver patch, we can handle the resolution of insert_exists by
converting it to update. Thoughts?
thanks
Shveta
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Conflict detection and logging in logical replication
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-19 08:36 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-22 09:03 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-25 06:34 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-26 09:33 ` Re: Conflict detection and logging in logical replication Nisha Moond <[email protected]>
2024-07-26 10:07 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
@ 2024-07-26 10:25 ` Amit Kapila <[email protected]>
2024-07-26 10:57 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
0 siblings, 1 reply; 50+ messages in thread
From: Amit Kapila @ 2024-07-26 10:25 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Nisha Moond <[email protected]>; Zhijie Hou (Fujitsu) <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Ashutosh Bapat <[email protected]>
On Fri, Jul 26, 2024 at 3:37 PM shveta malik <[email protected]> wrote:
>
> On Fri, Jul 26, 2024 at 3:03 PM Nisha Moond <[email protected]> wrote:
> >
> > On Thu, Jul 25, 2024 at 12:04 PM Zhijie Hou (Fujitsu)
> > <[email protected]> wrote:
> > > Here is the V6 patch set which addressed Shveta and Nisha's comments
> > > in [1][2][3][4].
> >
> > Thanks for the patch.
> > I tested the v6-0001 patch with partition table scenarios. Please
> > review the following scenario where Pub updates a tuple, causing it to
> > move from one partition to another on Sub.
> >
> > Setup:
> > Pub:
> > create table tab (a int not null, b int not null);
> > alter table tab add constraint tab_pk primary key (a,b);
> > Sub:
> > create table tab (a int not null, b int not null) partition by range (b);
> > alter table tab add constraint tab_pk primary key (a,b);
> > create table tab_1 partition of tab FOR values from (MINVALUE) TO (100);
> > create table tab_2 partition of tab FOR values from (101) TO (MAXVALUE);
> >
> > Test:
> > Pub: insert into tab values (1,1);
> > Sub: update tab set a=1 where a=1; > just to make it Sub's origin
> > Sub: insert into tab values (1,101);
> > Pub: update b=101 where b=1; --> Both 'update_differ' and
> > 'insert_exists' are detected.
> >
> > For non-partitioned tables, a similar update results in
> > 'update_differ' and 'update_exists' conflicts. After detecting
> > 'update_differ', the apply worker proceeds to apply the remote update
> > and if a tuple with the updated key already exists, it raises
> > 'update_exists'.
> > This same behavior is expected for partitioned tables too.
>
> Good catch. Yes, from the user's perspective, an update_* conflict
> should be raised when performing an update operation. But internally
> since we are deleting from one partition and inserting to another, we
> are hitting insert_exist. To convert this insert_exist to udpate_exist
> conflict, perhaps we need to change insert-operation to
> update-operation as the default resolver is 'always apply update' in
> case of update_differ.
>
But we already document that behind the scenes such an update is a
DELETE+INSERT operation [1]. Also, all the privilege checks or before
row triggers are of type insert, so, I think it is okay to consider
this as insert_exists conflict and document it. Later, resolver should
also fire for insert_exists conflict.
One more thing we need to consider is whether we should LOG or ERROR
for update/delete_differ conflicts. If we LOG as the patch is doing
then we are intentionally overwriting the row when the user may not
expect it. OTOH, without a patch anyway we are overwriting, so there
is an argument that logging by default is what the user will expect.
What do you think?
[1] - https://www.postgresql.org/docs/devel/sql-update.html (See ...
Behind the scenes, the row movement is actually a DELETE and INSERT
operation.)
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Conflict detection and logging in logical replication
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-19 08:36 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-22 09:03 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-25 06:34 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-26 09:33 ` Re: Conflict detection and logging in logical replication Nisha Moond <[email protected]>
2024-07-26 10:07 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-26 10:25 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
@ 2024-07-26 10:57 ` shveta malik <[email protected]>
2024-07-29 04:01 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
0 siblings, 1 reply; 50+ messages in thread
From: shveta malik @ 2024-07-26 10:57 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Nisha Moond <[email protected]>; Zhijie Hou (Fujitsu) <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Ashutosh Bapat <[email protected]>; shveta malik <[email protected]>
On Fri, Jul 26, 2024 at 3:56 PM Amit Kapila <[email protected]> wrote:
>
> On Fri, Jul 26, 2024 at 3:37 PM shveta malik <[email protected]> wrote:
> >
> > On Fri, Jul 26, 2024 at 3:03 PM Nisha Moond <[email protected]> wrote:
> > >
> > > On Thu, Jul 25, 2024 at 12:04 PM Zhijie Hou (Fujitsu)
> > > <[email protected]> wrote:
> > > > Here is the V6 patch set which addressed Shveta and Nisha's comments
> > > > in [1][2][3][4].
> > >
> > > Thanks for the patch.
> > > I tested the v6-0001 patch with partition table scenarios. Please
> > > review the following scenario where Pub updates a tuple, causing it to
> > > move from one partition to another on Sub.
> > >
> > > Setup:
> > > Pub:
> > > create table tab (a int not null, b int not null);
> > > alter table tab add constraint tab_pk primary key (a,b);
> > > Sub:
> > > create table tab (a int not null, b int not null) partition by range (b);
> > > alter table tab add constraint tab_pk primary key (a,b);
> > > create table tab_1 partition of tab FOR values from (MINVALUE) TO (100);
> > > create table tab_2 partition of tab FOR values from (101) TO (MAXVALUE);
> > >
> > > Test:
> > > Pub: insert into tab values (1,1);
> > > Sub: update tab set a=1 where a=1; > just to make it Sub's origin
> > > Sub: insert into tab values (1,101);
> > > Pub: update b=101 where b=1; --> Both 'update_differ' and
> > > 'insert_exists' are detected.
> > >
> > > For non-partitioned tables, a similar update results in
> > > 'update_differ' and 'update_exists' conflicts. After detecting
> > > 'update_differ', the apply worker proceeds to apply the remote update
> > > and if a tuple with the updated key already exists, it raises
> > > 'update_exists'.
> > > This same behavior is expected for partitioned tables too.
> >
> > Good catch. Yes, from the user's perspective, an update_* conflict
> > should be raised when performing an update operation. But internally
> > since we are deleting from one partition and inserting to another, we
> > are hitting insert_exist. To convert this insert_exist to udpate_exist
> > conflict, perhaps we need to change insert-operation to
> > update-operation as the default resolver is 'always apply update' in
> > case of update_differ.
> >
>
> But we already document that behind the scenes such an update is a
> DELETE+INSERT operation [1]. Also, all the privilege checks or before
> row triggers are of type insert, so, I think it is okay to consider
> this as insert_exists conflict and document it. Later, resolver should
> also fire for insert_exists conflict.
Thanks for the link. +1 on existing behaviour of insert_exists conflict.
> One more thing we need to consider is whether we should LOG or ERROR
> for update/delete_differ conflicts. If we LOG as the patch is doing
> then we are intentionally overwriting the row when the user may not
> expect it. OTOH, without a patch anyway we are overwriting, so there
> is an argument that logging by default is what the user will expect.
> What do you think?
I was under the impression that in this patch we do not intend to
change behaviour of HEAD and thus only LOG the conflict wherever
possible. And in the next patch of resolver, based on the user's input
of error/skip/or resolve, we take the action. I still think it is
better to stick to the said behaviour. Only if we commit the resolver
patch in the same version where we commit the detection patch, then we
can take the risk of changing this default behaviour to 'always
error'. Otherwise users will be left with conflicts arising but no
automatic way to resolve those. But for users who really want their
application to error out, we can provide an additional GUC in this
patch itself which changes the behaviour to 'always ERROR on
conflict'. Thoughts?
thanks
Shveta
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Conflict detection and logging in logical replication
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-19 08:36 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-22 09:03 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-25 06:34 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-26 09:33 ` Re: Conflict detection and logging in logical replication Nisha Moond <[email protected]>
2024-07-26 10:07 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-26 10:25 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
2024-07-26 10:57 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
@ 2024-07-29 04:01 ` Amit Kapila <[email protected]>
2024-07-29 11:55 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
0 siblings, 1 reply; 50+ messages in thread
From: Amit Kapila @ 2024-07-29 04:01 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Nisha Moond <[email protected]>; Zhijie Hou (Fujitsu) <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Ashutosh Bapat <[email protected]>
On Fri, Jul 26, 2024 at 4:28 PM shveta malik <[email protected]> wrote:
>
> On Fri, Jul 26, 2024 at 3:56 PM Amit Kapila <[email protected]> wrote:
> >
>
> > One more thing we need to consider is whether we should LOG or ERROR
> > for update/delete_differ conflicts. If we LOG as the patch is doing
> > then we are intentionally overwriting the row when the user may not
> > expect it. OTOH, without a patch anyway we are overwriting, so there
> > is an argument that logging by default is what the user will expect.
> > What do you think?
>
> I was under the impression that in this patch we do not intend to
> change behaviour of HEAD and thus only LOG the conflict wherever
> possible.
>
Earlier, I thought it was good to keep LOGGING the conflict where
there is no chance of wrong data update but for cases where we can do
something wrong, it is better to ERROR out. For example, for an
update_differ case where the apply worker can overwrite the data from
a different origin, it is better to ERROR out. I thought this case was
comparable to an existing ERROR case like a unique constraint
violation. But I see your point as well that one might expect the
existing behavior where we are silently overwriting the different
origin data. The one idea to address this concern is to suggest users
set the detect_conflict subscription option as off but I guess that
would make this feature unusable for users who don't want to ERROR out
for different origin update cases.
> And in the next patch of resolver, based on the user's input
> of error/skip/or resolve, we take the action. I still think it is
> better to stick to the said behaviour. Only if we commit the resolver
> patch in the same version where we commit the detection patch, then we
> can take the risk of changing this default behaviour to 'always
> error'. Otherwise users will be left with conflicts arising but no
> automatic way to resolve those. But for users who really want their
> application to error out, we can provide an additional GUC in this
> patch itself which changes the behaviour to 'always ERROR on
> conflict'.
>
I don't see a need of GUC here, even if we want we can have a
subscription option such conflict_log_level. But users may want to
either LOG or ERROR based on conflict type. For example, there won't
be any data inconsistency in two node replication for delete_missing
case as one is trying to delete already deleted data, so LOGGING such
a case should be sufficient whereas update_differ could lead to
different data on two nodes, so the user may want to ERROR out in such
a case.
We can keep the current behavior as default for the purpose of
conflict detection but can have a separate patch to decide whether to
LOG/ERROR based on conflict_type.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Conflict detection and logging in logical replication
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-19 08:36 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-22 09:03 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-25 06:34 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-26 09:33 ` Re: Conflict detection and logging in logical replication Nisha Moond <[email protected]>
2024-07-26 10:07 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-26 10:25 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
2024-07-26 10:57 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-29 04:01 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
@ 2024-07-29 11:55 ` shveta malik <[email protected]>
0 siblings, 0 replies; 50+ messages in thread
From: shveta malik @ 2024-07-29 11:55 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Nisha Moond <[email protected]>; Zhijie Hou (Fujitsu) <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Ashutosh Bapat <[email protected]>; shveta malik <[email protected]>
On Mon, Jul 29, 2024 at 9:31 AM Amit Kapila <[email protected]> wrote:
>
> On Fri, Jul 26, 2024 at 4:28 PM shveta malik <[email protected]> wrote:
> >
> > On Fri, Jul 26, 2024 at 3:56 PM Amit Kapila <[email protected]> wrote:
> > >
> >
> > > One more thing we need to consider is whether we should LOG or ERROR
> > > for update/delete_differ conflicts. If we LOG as the patch is doing
> > > then we are intentionally overwriting the row when the user may not
> > > expect it. OTOH, without a patch anyway we are overwriting, so there
> > > is an argument that logging by default is what the user will expect.
> > > What do you think?
> >
> > I was under the impression that in this patch we do not intend to
> > change behaviour of HEAD and thus only LOG the conflict wherever
> > possible.
> >
>
> Earlier, I thought it was good to keep LOGGING the conflict where
> there is no chance of wrong data update but for cases where we can do
> something wrong, it is better to ERROR out. For example, for an
> update_differ case where the apply worker can overwrite the data from
> a different origin, it is better to ERROR out. I thought this case was
> comparable to an existing ERROR case like a unique constraint
> violation. But I see your point as well that one might expect the
> existing behavior where we are silently overwriting the different
> origin data. The one idea to address this concern is to suggest users
> set the detect_conflict subscription option as off but I guess that
> would make this feature unusable for users who don't want to ERROR out
> for different origin update cases.
>
> > And in the next patch of resolver, based on the user's input
> > of error/skip/or resolve, we take the action. I still think it is
> > better to stick to the said behaviour. Only if we commit the resolver
> > patch in the same version where we commit the detection patch, then we
> > can take the risk of changing this default behaviour to 'always
> > error'. Otherwise users will be left with conflicts arising but no
> > automatic way to resolve those. But for users who really want their
> > application to error out, we can provide an additional GUC in this
> > patch itself which changes the behaviour to 'always ERROR on
> > conflict'.
> >
>
> I don't see a need of GUC here, even if we want we can have a
> subscription option such conflict_log_level. But users may want to
> either LOG or ERROR based on conflict type. For example, there won't
> be any data inconsistency in two node replication for delete_missing
> case as one is trying to delete already deleted data, so LOGGING such
> a case should be sufficient whereas update_differ could lead to
> different data on two nodes, so the user may want to ERROR out in such
> a case.
>
> We can keep the current behavior as default for the purpose of
> conflict detection but can have a separate patch to decide whether to
> LOG/ERROR based on conflict_type.
+1 on the idea of giving an option to the user to choose either ERROR
or LOG for each conflict type separately.
thanks
Shveta
^ permalink raw reply [nested|flat] 50+ messages in thread
* Re: Conflict detection and logging in logical replication
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
@ 2024-07-24 04:50 ` Nisha Moond <[email protected]>
2 siblings, 0 replies; 50+ messages in thread
From: Nisha Moond @ 2024-07-24 04:50 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: shveta malik <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>; Jan Wieck <[email protected]>; Tomas Vondra <[email protected]>; Amit Kapila <[email protected]>; Ashutosh Bapat <[email protected]>
On Thu, Jul 18, 2024 at 7:52 AM Zhijie Hou (Fujitsu)
<[email protected]> wrote:
>
> Attach the V5 patch set which changed the following:
>
Tested v5-0001 patch, and it fails to detect the update_exists
conflict for a setup where Pub has a non-partitioned table and Sub has
the same table partitioned.
Below is a testcase showcasing the issue:
Setup:
Pub:
create table tab (a int not null, b int not null);
alter table tab add constraint tab_pk primary key (a,b);
Sub:
create table tab (a int not null, b int not null) partition by range (b);
alter table tab add constraint tab_pk primary key (a,b);
CREATE TABLE tab_1 PARTITION OF tab FOR VALUES FROM (MINVALUE) TO (100);
CREATE TABLE tab_2 PARTITION OF tab FOR VALUES FROM (101) TO (MAXVALUE);
Test:
Pub: insert into tab values (1,1);
Sub: insert into tab values (2,1);
Pub: update tab set a=2 where a=1; --> After this update on Pub,
'update_exists' is expected on Sub, but it fails to detect the
conflict and logs the key violation error -
ERROR: duplicate key value violates unique constraint "tab_1_pkey"
DETAIL: Key (a, b)=(2, 1) already exists.
Thanks,
Nisha
^ permalink raw reply [nested|flat] 50+ messages in thread
end of thread, other threads:[~2024-07-30 08:41 UTC | newest]
Thread overview: 50+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-10-26 11:32 Comment fix and question about dshash.c Antonin Houska <[email protected]>
2018-10-26 21:03 ` Thomas Munro <[email protected]>
2018-10-26 21:38 ` Thomas Munro <[email protected]>
2018-10-27 11:22 ` Antonin Houska <[email protected]>
2018-10-27 11:51 ` Antonin Houska <[email protected]>
2018-10-27 12:22 ` Tomas Vondra <[email protected]>
2018-10-28 02:27 ` Thomas Munro <[email protected]>
2022-12-09 12:49 Re: Add 64-bit XIDs into PostgreSQL 15 Aleksander Alekseev <[email protected]>
2022-12-09 13:54 ` Re: Add 64-bit XIDs into PostgreSQL 15 adherent postgres <[email protected]>
2022-12-09 14:13 ` Re: Add 64-bit XIDs into PostgreSQL 15 Pavel Borisov <[email protected]>
2022-12-09 14:29 ` Re: Add 64-bit XIDs into PostgreSQL 15 Maxim Orlov <[email protected]>
2024-02-27 01:34 [PATCH v11 06/17] table_scan_bitmap_next_block() returns lossy or exact Melanie Plageman <[email protected]>
2024-02-27 01:34 [PATCH v7 06/13] table_scan_bitmap_next_block() returns lossy or exact Melanie Plageman <[email protected]>
2024-02-27 01:34 [PATCH v6 07/14] table_scan_bitmap_next_block() returns lossy or exact Melanie Plageman <[email protected]>
2024-02-27 01:34 [PATCH v10 06/17] table_scan_bitmap_next_block() returns lossy or exact Melanie Plageman <[email protected]>
2024-02-27 01:34 [PATCH v9 06/17] table_scan_bitmap_next_block() returns lossy or exact Melanie Plageman <[email protected]>
2024-02-27 01:34 [PATCH v9 06/17] table_scan_bitmap_next_block() returns lossy or exact Melanie Plageman <[email protected]>
2024-02-27 01:34 [PATCH v10 06/17] table_scan_bitmap_next_block() returns lossy or exact Melanie Plageman <[email protected]>
2024-02-27 01:34 [PATCH v12 06/17] table_scan_bitmap_next_block() returns lossy or exact Melanie Plageman <[email protected]>
2024-02-27 01:34 [PATCH v10 06/17] table_scan_bitmap_next_block() returns lossy or exact Melanie Plageman <[email protected]>
2024-02-27 01:34 [PATCH v6 07/14] table_scan_bitmap_next_block() returns lossy or exact Melanie Plageman <[email protected]>
2024-02-27 01:34 [PATCH v6 07/14] table_scan_bitmap_next_block() returns lossy or exact Melanie Plageman <[email protected]>
2024-02-27 01:34 [PATCH v8 06/17] table_scan_bitmap_next_block() returns lossy or exact Melanie Plageman <[email protected]>
2024-02-27 01:34 [PATCH v7 06/13] table_scan_bitmap_next_block() returns lossy or exact Melanie Plageman <[email protected]>
2024-02-27 01:34 [PATCH v9 06/17] table_scan_bitmap_next_block() returns lossy or exact Melanie Plageman <[email protected]>
2024-02-27 01:34 [PATCH v7 06/13] table_scan_bitmap_next_block() returns lossy or exact Melanie Plageman <[email protected]>
2024-02-27 01:34 [PATCH v8 06/17] table_scan_bitmap_next_block() returns lossy or exact Melanie Plageman <[email protected]>
2024-02-27 01:34 [PATCH v11 06/17] table_scan_bitmap_next_block() returns lossy or exact Melanie Plageman <[email protected]>
2024-02-27 01:34 [PATCH v8 06/17] table_scan_bitmap_next_block() returns lossy or exact Melanie Plageman <[email protected]>
2024-07-11 05:03 Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-18 02:22 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-18 05:50 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-19 08:36 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-22 09:03 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-25 06:34 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-25 10:42 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
2024-07-26 11:33 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
2024-07-29 06:14 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-29 09:24 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
2024-07-29 10:59 ` Re: Conflict detection and logging in logical replication Dilip Kumar <[email protected]>
2024-07-29 12:14 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-30 08:19 ` RE: Conflict detection and logging in logical replication Zhijie Hou (Fujitsu) <[email protected]>
2024-07-30 08:41 ` Re: Conflict detection and logging in logical replication Dilip Kumar <[email protected]>
2024-07-26 09:33 ` Re: Conflict detection and logging in logical replication Nisha Moond <[email protected]>
2024-07-26 10:07 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-26 10:25 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
2024-07-26 10:57 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-29 04:01 ` Re: Conflict detection and logging in logical replication Amit Kapila <[email protected]>
2024-07-29 11:55 ` Re: Conflict detection and logging in logical replication shveta malik <[email protected]>
2024-07-24 04:50 ` Re: Conflict detection and logging in logical replication Nisha Moond <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox