pg.ddx.io  pgsql-committers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
pgsql: Fix skip-empty-xacts with sequences in test_decoding
6+ messages / 3 participants
[nested] [flat]

Commits from this thread:
  2022-03-23 dfdb2f3bf5  test_decoding: Add comments about delaying BEGIN/PREPARE
* pgsql: Fix skip-empty-xacts with sequences in test_decoding
@ 2022-02-12 23:00  Tomas Vondra <tomas.vondra@postgresql.org>
  0 siblings, 1 reply; 6+ messages in thread

From: Tomas Vondra @ 2022-02-12 23:00 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix skip-empty-xacts with sequences in test_decoding

Regression tests need to use skip-empty-xacts = false, because there
might be accidental concurrent activity (like autovacuum), particularly
on slow machines. The tests added by 80901b3291 failed to do that in a
couple places, triggering occasional failures on buildfarm.

Fixing the tests however uncovered a bug in the code, because sequence
callbacks did not handle skip-empty-xacts properly. For trasactional
increments we need to check/update the xact_wrote_changes flag, and emit
the BEGIN if it's the first change in the transaction.

Reported-by: Andres Freund
Discussion: https://postgr.es/m/20220212220413.b25amklo7t4xb7ni%40alap3.anarazel.de

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/b779d7d8fdae088d70da5ed9fcd8205035676df3

Modified Files
--------------
contrib/test_decoding/expected/sequence.out | 12 +++++-------
contrib/test_decoding/sql/sequence.sql      |  8 ++++----
contrib/test_decoding/test_decoding.c       | 22 ++++++++++++++++++++++
3 files changed, 31 insertions(+), 11 deletions(-)



^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding
@ 2022-02-13 15:26  Alvaro Herrera <alvherre@alvh.no-ip.org>
  parent: Tomas Vondra <tomas.vondra@postgresql.org>
  0 siblings, 1 reply; 6+ messages in thread

From: Alvaro Herrera @ 2022-02-13 15:26 UTC (permalink / raw)
  To: Tomas Vondra <tomas.vondra@postgresql.org>; +Cc: pgsql-committers@lists.postgresql.org

On 2022-Feb-12, Tomas Vondra wrote:

> Fix skip-empty-xacts with sequences in test_decoding
> 
> Regression tests need to use skip-empty-xacts = false, because there
> might be accidental concurrent activity (like autovacuum), particularly
> on slow machines. The tests added by 80901b3291 failed to do that in a
> couple places, triggering occasional failures on buildfarm.

I think you meant "... need to use skip-empty-xacts = true", since you
changed the value from 0 to 1, and the point is precisely to *skip*
those transactions.  (The hidden double negative "skip=false" seems
quite confusing BTW.)

> Fixing the tests however uncovered a bug in the code, because sequence
> callbacks did not handle skip-empty-xacts properly. For trasactional
> increments we need to check/update the xact_wrote_changes flag, and emit
> the BEGIN if it's the first change in the transaction.

Hmm.  Perhaps there should be a separate test script that runs various
things under skip-empty-xacts=0 and somehow protected against ancillary
activities (maybe a TAP test using autovacuum=0), just so that this new
code is covered ...

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/
"Nadie está tan esclavizado como el que se cree libre no siéndolo" (Goethe)





^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding
@ 2022-02-13 17:08  Tomas Vondra <tomas.vondra@enterprisedb.com>
  parent: Alvaro Herrera <alvherre@alvh.no-ip.org>
  0 siblings, 1 reply; 6+ messages in thread

From: Tomas Vondra @ 2022-02-13 17:08 UTC (permalink / raw)
  To: Alvaro Herrera <alvherre@alvh.no-ip.org>; Tomas Vondra <tomas.vondra@postgresql.org>; +Cc: pgsql-committers@lists.postgresql.org



On 2/13/22 16:26, Alvaro Herrera wrote:
> On 2022-Feb-12, Tomas Vondra wrote:
> 
>> Fix skip-empty-xacts with sequences in test_decoding
>>
>> Regression tests need to use skip-empty-xacts = false, because there
>> might be accidental concurrent activity (like autovacuum), particularly
>> on slow machines. The tests added by 80901b3291 failed to do that in a
>> couple places, triggering occasional failures on buildfarm.
> 
> I think you meant "... need to use skip-empty-xacts = true", since you
> changed the value from 0 to 1, and the point is precisely to *skip*
> those transactions.  (The hidden double negative "skip=false" seems
> quite confusing BTW.)
> 

Yeah, that's what I meant :-(

>> Fixing the tests however uncovered a bug in the code, because sequence
>> callbacks did not handle skip-empty-xacts properly. For trasactional
>> increments we need to check/update the xact_wrote_changes flag, and emit
>> the BEGIN if it's the first change in the transaction.
> 
> Hmm.  Perhaps there should be a separate test script that runs various
> things under skip-empty-xacts=0 and somehow protected against ancillary
> activities (maybe a TAP test using autovacuum=0), just so that this new
> code is covered ...
> 

I'm not sure what exactly would be the benefit? I mean, it'd be testing
exactly the same thing as now - actually less than now, because instead
of ignoring empty transactions in the last step (output plugin), we'd
make sure not to have any empty transactions.

For the record, the new code in sequence decoding is tested already, at
least for transactional increments.

regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company





^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding
@ 2022-02-14 13:36  Alvaro Herrera <alvherre@alvh.no-ip.org>
  parent: Tomas Vondra <tomas.vondra@enterprisedb.com>
  0 siblings, 1 reply; 6+ messages in thread

From: Alvaro Herrera @ 2022-02-14 13:36 UTC (permalink / raw)
  To: Tomas Vondra <tomas.vondra@enterprisedb.com>; +Cc: Tomas Vondra <tomas.vondra@postgresql.org>; pgsql-committers@lists.postgresql.org

On 2022-Feb-13, Tomas Vondra wrote:

> >> Fixing the tests however uncovered a bug in the code, because sequence
> >> callbacks did not handle skip-empty-xacts properly. For trasactional
> >> increments we need to check/update the xact_wrote_changes flag, and emit
> >> the BEGIN if it's the first change in the transaction.
> > 
> > Hmm.  Perhaps there should be a separate test script that runs various
> > things under skip-empty-xacts=0 and somehow protected against ancillary
> > activities (maybe a TAP test using autovacuum=0), just so that this new
> > code is covered ...
> 
> I'm not sure what exactly would be the benefit?

Yeah, it's pointless.

I propose the attached comment additions instead.

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/

Attachments:

comments.patch (text/x-diff, 1.4K) download diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c index ea22649e41..c7a87f5fe5 100644 --- a/contrib/test_decoding/test_decoding.c +++ b/contrib/test_decoding/test_decoding.c @@ -322,6 +322,10 @@ pg_decode_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn) txndata->xact_wrote_changes = false; txn->output_plugin_private = txndata; + /* + * If asked to skip empty transactions, we'll emit BEGIN at the point where + * the first operation is received for this transaction. + */ if (data->skip_empty_xacts) return; @@ -378,6 +382,10 @@ pg_decode_begin_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn) txndata->xact_wrote_changes = false; txn->output_plugin_private = txndata; + /* + * If asked to skip empty transactions, we'll emit BEGIN at the point where + * the first operation is received for this transaction. + */ if (data->skip_empty_xacts) return; @@ -392,6 +400,10 @@ pg_decode_prepare_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, TestDecodingData *data = ctx->output_plugin_private; TestDecodingTxnData *txndata = txn->output_plugin_private; + /* + * If asked to skip empty transactions, we'll emit PREPARE at the point + * where the first operation is received for this transaction. + */ if (data->skip_empty_xacts && !txndata->xact_wrote_changes) return;
^ permalink raw reply [nested|flat] 6+ messages in thread

* Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding
@ 2022-02-14 14:05  Tomas Vondra <tomas.vondra@enterprisedb.com>
  parent: Alvaro Herrera <alvherre@alvh.no-ip.org>
  0 siblings, 1 reply; 6+ messages in thread

From: Tomas Vondra @ 2022-02-14 14:05 UTC (permalink / raw)
  To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Tomas Vondra <tomas.vondra@postgresql.org>; pgsql-committers@lists.postgresql.org



On 2/14/22 14:36, Alvaro Herrera wrote:
> On 2022-Feb-13, Tomas Vondra wrote:
> 
>>>> Fixing the tests however uncovered a bug in the code, because sequence
>>>> callbacks did not handle skip-empty-xacts properly. For trasactional
>>>> increments we need to check/update the xact_wrote_changes flag, and emit
>>>> the BEGIN if it's the first change in the transaction.
>>>
>>> Hmm.  Perhaps there should be a separate test script that runs various
>>> things under skip-empty-xacts=0 and somehow protected against ancillary
>>> activities (maybe a TAP test using autovacuum=0), just so that this new
>>> code is covered ...
>>
>> I'm not sure what exactly would be the benefit?
> 
> Yeah, it's pointless.
> 
> I propose the attached comment additions instead.
> 

The comments seem fine to me (although not really related to these 
commits about sequence decoding).


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company





^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: pgsql: Fix skip-empty-xacts with sequences in test_decoding
@ 2022-03-23 15:48  Alvaro Herrera <alvherre@alvh.no-ip.org>
  parent: Tomas Vondra <tomas.vondra@enterprisedb.com>
  0 siblings, 0 replies; 6+ messages in thread

From: Alvaro Herrera @ 2022-03-23 15:48 UTC (permalink / raw)
  To: Tomas Vondra <tomas.vondra@enterprisedb.com>; +Cc: Tomas Vondra <tomas.vondra@postgresql.org>; pgsql-committers@lists.postgresql.org

On 2022-Feb-14, Tomas Vondra wrote:

> On 2/14/22 14:36, Alvaro Herrera wrote:

> > I propose the attached comment additions instead.
> 
> The comments seem fine to me (although not really related to these commits
> about sequence decoding).

Thanks, pushed.

-- 
Álvaro Herrera        Breisgau, Deutschland  —  https://www.EnterpriseDB.com/
"Now I have my system running, not a byte was off the shelf;
It rarely breaks and when it does I fix the code myself.
It's stable, clean and elegant, and lightning fast as well,
And it doesn't cost a nickel, so Bill Gates can go to hell."





^ permalink  raw  reply  [nested|flat] 6+ messages in thread


end of thread, other threads:[~2022-03-23 15:48 UTC | newest]

Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-02-12 23:00 pgsql: Fix skip-empty-xacts with sequences in test_decoding Tomas Vondra <tomas.vondra@postgresql.org>
2022-02-13 15:26 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-02-13 17:08   ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-02-14 13:36     ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-02-14 14:05       ` Tomas Vondra <tomas.vondra@enterprisedb.com>
2022-03-23 15:48         ` Alvaro Herrera <alvherre@alvh.no-ip.org>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox