pg.ddx.io pgsql-committers@postgresql.org mailing list archive
help / color / mirror / Atom feedpgsql: Fix skip-empty-xacts with sequences in test_decoding
6+ messages / 3 participants
[nested] [flat]
Commits from this thread:
2022-03-23 dfdb2f3bf5 test_decoding: Add comments about delaying BEGIN/PREPARE
* pgsql: Fix skip-empty-xacts with sequences in test_decoding
@ 2022-02-12 23:00 Tomas Vondra <tomas.vondra@postgresql.org>
2022-02-13 15:26 ` Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding Alvaro Herrera <alvherre@alvh.no-ip.org>
0 siblings, 1 reply; 6+ messages in thread
From: Tomas Vondra @ 2022-02-12 23:00 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
Fix skip-empty-xacts with sequences in test_decoding
Regression tests need to use skip-empty-xacts = false, because there
might be accidental concurrent activity (like autovacuum), particularly
on slow machines. The tests added by 80901b3291 failed to do that in a
couple places, triggering occasional failures on buildfarm.
Fixing the tests however uncovered a bug in the code, because sequence
callbacks did not handle skip-empty-xacts properly. For trasactional
increments we need to check/update the xact_wrote_changes flag, and emit
the BEGIN if it's the first change in the transaction.
Reported-by: Andres Freund
Discussion: https://postgr.es/m/20220212220413.b25amklo7t4xb7ni%40alap3.anarazel.de
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/b779d7d8fdae088d70da5ed9fcd8205035676df3
Modified Files
--------------
contrib/test_decoding/expected/sequence.out | 12 +++++-------
contrib/test_decoding/sql/sequence.sql | 8 ++++----
contrib/test_decoding/test_decoding.c | 22 ++++++++++++++++++++++
3 files changed, 31 insertions(+), 11 deletions(-)
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding
2022-02-12 23:00 pgsql: Fix skip-empty-xacts with sequences in test_decoding Tomas Vondra <tomas.vondra@postgresql.org>
@ 2022-02-13 15:26 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-02-13 17:08 ` Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding Tomas Vondra <tomas.vondra@enterprisedb.com>
0 siblings, 1 reply; 6+ messages in thread
From: Alvaro Herrera @ 2022-02-13 15:26 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@postgresql.org>; +Cc: pgsql-committers@lists.postgresql.org
On 2022-Feb-12, Tomas Vondra wrote:
> Fix skip-empty-xacts with sequences in test_decoding
>
> Regression tests need to use skip-empty-xacts = false, because there
> might be accidental concurrent activity (like autovacuum), particularly
> on slow machines. The tests added by 80901b3291 failed to do that in a
> couple places, triggering occasional failures on buildfarm.
I think you meant "... need to use skip-empty-xacts = true", since you
changed the value from 0 to 1, and the point is precisely to *skip*
those transactions. (The hidden double negative "skip=false" seems
quite confusing BTW.)
> Fixing the tests however uncovered a bug in the code, because sequence
> callbacks did not handle skip-empty-xacts properly. For trasactional
> increments we need to check/update the xact_wrote_changes flag, and emit
> the BEGIN if it's the first change in the transaction.
Hmm. Perhaps there should be a separate test script that runs various
things under skip-empty-xacts=0 and somehow protected against ancillary
activities (maybe a TAP test using autovacuum=0), just so that this new
code is covered ...
--
Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/
"Nadie está tan esclavizado como el que se cree libre no siéndolo" (Goethe)
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding
2022-02-12 23:00 pgsql: Fix skip-empty-xacts with sequences in test_decoding Tomas Vondra <tomas.vondra@postgresql.org>
2022-02-13 15:26 ` Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2022-02-13 17:08 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-02-14 13:36 ` Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding Alvaro Herrera <alvherre@alvh.no-ip.org>
0 siblings, 1 reply; 6+ messages in thread
From: Tomas Vondra @ 2022-02-13 17:08 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; Tomas Vondra <tomas.vondra@postgresql.org>; +Cc: pgsql-committers@lists.postgresql.org
On 2/13/22 16:26, Alvaro Herrera wrote:
> On 2022-Feb-12, Tomas Vondra wrote:
>
>> Fix skip-empty-xacts with sequences in test_decoding
>>
>> Regression tests need to use skip-empty-xacts = false, because there
>> might be accidental concurrent activity (like autovacuum), particularly
>> on slow machines. The tests added by 80901b3291 failed to do that in a
>> couple places, triggering occasional failures on buildfarm.
>
> I think you meant "... need to use skip-empty-xacts = true", since you
> changed the value from 0 to 1, and the point is precisely to *skip*
> those transactions. (The hidden double negative "skip=false" seems
> quite confusing BTW.)
>
Yeah, that's what I meant :-(
>> Fixing the tests however uncovered a bug in the code, because sequence
>> callbacks did not handle skip-empty-xacts properly. For trasactional
>> increments we need to check/update the xact_wrote_changes flag, and emit
>> the BEGIN if it's the first change in the transaction.
>
> Hmm. Perhaps there should be a separate test script that runs various
> things under skip-empty-xacts=0 and somehow protected against ancillary
> activities (maybe a TAP test using autovacuum=0), just so that this new
> code is covered ...
>
I'm not sure what exactly would be the benefit? I mean, it'd be testing
exactly the same thing as now - actually less than now, because instead
of ignoring empty transactions in the last step (output plugin), we'd
make sure not to have any empty transactions.
For the record, the new code in sequence decoding is tested already, at
least for transactional increments.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding
2022-02-12 23:00 pgsql: Fix skip-empty-xacts with sequences in test_decoding Tomas Vondra <tomas.vondra@postgresql.org>
2022-02-13 15:26 ` Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-02-13 17:08 ` Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding Tomas Vondra <tomas.vondra@enterprisedb.com>
@ 2022-02-14 13:36 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-02-14 14:05 ` Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding Tomas Vondra <tomas.vondra@enterprisedb.com>
0 siblings, 1 reply; 6+ messages in thread
From: Alvaro Herrera @ 2022-02-14 13:36 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@enterprisedb.com>; +Cc: Tomas Vondra <tomas.vondra@postgresql.org>; pgsql-committers@lists.postgresql.org
On 2022-Feb-13, Tomas Vondra wrote:
> >> Fixing the tests however uncovered a bug in the code, because sequence
> >> callbacks did not handle skip-empty-xacts properly. For trasactional
> >> increments we need to check/update the xact_wrote_changes flag, and emit
> >> the BEGIN if it's the first change in the transaction.
> >
> > Hmm. Perhaps there should be a separate test script that runs various
> > things under skip-empty-xacts=0 and somehow protected against ancillary
> > activities (maybe a TAP test using autovacuum=0), just so that this new
> > code is covered ...
>
> I'm not sure what exactly would be the benefit?
Yeah, it's pointless.
I propose the attached comment additions instead.
--
Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/
Attachments:
comments.patch (text/x-diff, 1.4K) download
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index ea22649e41..c7a87f5fe5 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -322,6 +322,10 @@ pg_decode_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
txndata->xact_wrote_changes = false;
txn->output_plugin_private = txndata;
+ /*
+ * If asked to skip empty transactions, we'll emit BEGIN at the point where
+ * the first operation is received for this transaction.
+ */
if (data->skip_empty_xacts)
return;
@@ -378,6 +382,10 @@ pg_decode_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn)
txndata->xact_wrote_changes = false;
txn->output_plugin_private = txndata;
+ /*
+ * If asked to skip empty transactions, we'll emit BEGIN at the point where
+ * the first operation is received for this transaction.
+ */
if (data->skip_empty_xacts)
return;
@@ -392,6 +400,10 @@ pg_decode_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
TestDecodingData *data = ctx->output_plugin_private;
TestDecodingTxnData *txndata = txn->output_plugin_private;
+ /*
+ * If asked to skip empty transactions, we'll emit PREPARE at the point
+ * where the first operation is received for this transaction.
+ */
if (data->skip_empty_xacts && !txndata->xact_wrote_changes)
return;
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding
2022-02-12 23:00 pgsql: Fix skip-empty-xacts with sequences in test_decoding Tomas Vondra <tomas.vondra@postgresql.org>
2022-02-13 15:26 ` Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-02-13 17:08 ` Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-02-14 13:36 ` Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2022-02-14 14:05 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-03-23 15:48 ` Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding Alvaro Herrera <alvherre@alvh.no-ip.org>
0 siblings, 1 reply; 6+ messages in thread
From: Tomas Vondra @ 2022-02-14 14:05 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Tomas Vondra <tomas.vondra@postgresql.org>; pgsql-committers@lists.postgresql.org
On 2/14/22 14:36, Alvaro Herrera wrote:
> On 2022-Feb-13, Tomas Vondra wrote:
>
>>>> Fixing the tests however uncovered a bug in the code, because sequence
>>>> callbacks did not handle skip-empty-xacts properly. For trasactional
>>>> increments we need to check/update the xact_wrote_changes flag, and emit
>>>> the BEGIN if it's the first change in the transaction.
>>>
>>> Hmm. Perhaps there should be a separate test script that runs various
>>> things under skip-empty-xacts=0 and somehow protected against ancillary
>>> activities (maybe a TAP test using autovacuum=0), just so that this new
>>> code is covered ...
>>
>> I'm not sure what exactly would be the benefit?
>
> Yeah, it's pointless.
>
> I propose the attached comment additions instead.
>
The comments seem fine to me (although not really related to these
commits about sequence decoding).
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding
2022-02-12 23:00 pgsql: Fix skip-empty-xacts with sequences in test_decoding Tomas Vondra <tomas.vondra@postgresql.org>
2022-02-13 15:26 ` Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-02-13 17:08 ` Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-02-14 13:36 ` Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-02-14 14:05 ` Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding Tomas Vondra <tomas.vondra@enterprisedb.com>
@ 2022-03-23 15:48 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
0 siblings, 0 replies; 6+ messages in thread
From: Alvaro Herrera @ 2022-03-23 15:48 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@enterprisedb.com>; +Cc: Tomas Vondra <tomas.vondra@postgresql.org>; pgsql-committers@lists.postgresql.org
On 2022-Feb-14, Tomas Vondra wrote:
> On 2/14/22 14:36, Alvaro Herrera wrote:
> > I propose the attached comment additions instead.
>
> The comments seem fine to me (although not really related to these commits
> about sequence decoding).
Thanks, pushed.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"Now I have my system running, not a byte was off the shelf;
It rarely breaks and when it does I fix the code myself.
It's stable, clean and elegant, and lightning fast as well,
And it doesn't cost a nickel, so Bill Gates can go to hell."
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2022-03-23 15:48 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-02-12 23:00 pgsql: Fix skip-empty-xacts with sequences in test_decoding Tomas Vondra <tomas.vondra@postgresql.org>
2022-02-13 15:26 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-02-13 17:08 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-02-14 13:36 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-02-14 14:05 ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-03-23 15:48 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox