pg.ddx.io  pgsql-committers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
pgsql: Add more $Test::Builder::Level in the TAP tests
12+ messages / 2 participants
[nested] [flat]

Commits from this thread:
  2021-10-18 384f1abdb9  Fix portability issues in new TAP tests of psql
* pgsql: Add more $Test::Builder::Level in the TAP tests
@ 2021-10-12 02:17  Michael Paquier <michael@paquier.xyz>
  0 siblings, 1 reply; 12+ messages in thread

From: Michael Paquier @ 2021-10-12 02:17 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Add more $Test::Builder::Level in the TAP tests

Incrementing the level of the call stack reported is useful for
debugging purposes as it allows to control which part of the test is
exactly failing, especially if a test is structured with subroutines
that call routines from Test::More.

This adds more incrementations of $Test::Builder::Level where debugging
gets improved (for example it does not make sense for some paths like
pg_rewind where long subroutines are used).

A note is added to src/test/perl/README about that, based on a
suggestion from Andrew Dunstan and a wording coming from both of us.

Usage of Test::Builder::Level has spread in 12, so a backpatch down to
this version is done.

Reviewed-by: Andrew Dunstan, Peter Eisentraut, Daniel Gustafsson
Discussion: https://postgr.es/m/YV1CCFwgM1RV1LeS@paquier.xyz
Backpatch-through: 12

Branch
------
REL_14_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/d834ebcf23208b3ae2109c0cae9af077202a27a4

Modified Files
--------------
contrib/amcheck/t/001_verify_heapam.pl               |  8 ++++++++
contrib/test_decoding/t/001_repl_stats.pl            |  2 ++
src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl |  2 ++
src/bin/pg_verifybackup/t/005_bad_manifest.pl        |  6 ++++++
src/bin/psql/t/010_tab_completion.pl                 |  4 ++++
src/test/kerberos/t/001_auth.pl                      |  2 ++
src/test/perl/README                                 | 10 +++++++++-
src/test/recovery/t/001_stream_rep.pl                |  2 ++
src/test/recovery/t/003_recovery_targets.pl          |  2 ++
src/test/recovery/t/007_sync_rep.pl                  |  2 ++
src/test/recovery/t/009_twophase.pl                  |  2 ++
src/test/recovery/t/018_wal_optimize.pl              |  2 ++
12 files changed, 43 insertions(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* pgsql: Add more $Test::Builder::Level in the TAP tests
@ 2021-10-12 02:17  Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 12+ messages in thread

From: Michael Paquier @ 2021-10-12 02:17 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Add more $Test::Builder::Level in the TAP tests

Incrementing the level of the call stack reported is useful for
debugging purposes as it allows to control which part of the test is
exactly failing, especially if a test is structured with subroutines
that call routines from Test::More.

This adds more incrementations of $Test::Builder::Level where debugging
gets improved (for example it does not make sense for some paths like
pg_rewind where long subroutines are used).

A note is added to src/test/perl/README about that, based on a
suggestion from Andrew Dunstan and a wording coming from both of us.

Usage of Test::Builder::Level has spread in 12, so a backpatch down to
this version is done.

Reviewed-by: Andrew Dunstan, Peter Eisentraut, Daniel Gustafsson
Discussion: https://postgr.es/m/YV1CCFwgM1RV1LeS@paquier.xyz
Backpatch-through: 12

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/f9c4cb686800d46ef9e9e90ed5133493b23962af

Modified Files
--------------
contrib/amcheck/t/001_verify_heapam.pl               |  8 ++++++++
contrib/test_decoding/t/001_repl_stats.pl            |  2 ++
src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl |  2 ++
src/bin/pg_ctl/t/004_logrotate.pl                    |  2 ++
src/bin/pg_verifybackup/t/005_bad_manifest.pl        |  6 ++++++
src/bin/psql/t/010_tab_completion.pl                 |  4 ++++
src/test/kerberos/t/001_auth.pl                      |  2 ++
src/test/perl/README                                 | 10 +++++++++-
src/test/recovery/t/001_stream_rep.pl                |  2 ++
src/test/recovery/t/003_recovery_targets.pl          |  2 ++
src/test/recovery/t/007_sync_rep.pl                  |  2 ++
src/test/recovery/t/009_twophase.pl                  |  2 ++
src/test/recovery/t/018_wal_optimize.pl              |  2 ++
13 files changed, 45 insertions(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* pgsql: Add more $Test::Builder::Level in the TAP tests
@ 2021-10-12 02:17  Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 12+ messages in thread

From: Michael Paquier @ 2021-10-12 02:17 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Add more $Test::Builder::Level in the TAP tests

Incrementing the level of the call stack reported is useful for
debugging purposes as it allows to control which part of the test is
exactly failing, especially if a test is structured with subroutines
that call routines from Test::More.

This adds more incrementations of $Test::Builder::Level where debugging
gets improved (for example it does not make sense for some paths like
pg_rewind where long subroutines are used).

A note is added to src/test/perl/README about that, based on a
suggestion from Andrew Dunstan and a wording coming from both of us.

Usage of Test::Builder::Level has spread in 12, so a backpatch down to
this version is done.

Reviewed-by: Andrew Dunstan, Peter Eisentraut, Daniel Gustafsson
Discussion: https://postgr.es/m/YV1CCFwgM1RV1LeS@paquier.xyz
Backpatch-through: 12

Branch
------
REL_13_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/bab0ff2e44b781ec1b1180c5e96bdb95c385c80b

Modified Files
--------------
src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl |  2 ++
src/bin/pg_verifybackup/t/005_bad_manifest.pl        |  6 ++++++
src/bin/psql/t/010_tab_completion.pl                 |  4 ++++
src/test/kerberos/t/001_auth.pl                      |  2 ++
src/test/perl/README                                 | 10 +++++++++-
src/test/recovery/t/001_stream_rep.pl                |  2 ++
src/test/recovery/t/003_recovery_targets.pl          |  2 ++
src/test/recovery/t/007_sync_rep.pl                  |  2 ++
src/test/recovery/t/009_twophase.pl                  |  2 ++
src/test/recovery/t/018_wal_optimize.pl              |  2 ++
10 files changed, 33 insertions(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* pgsql: Add more $Test::Builder::Level in the TAP tests
@ 2021-10-12 02:17  Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 12+ messages in thread

From: Michael Paquier @ 2021-10-12 02:17 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Add more $Test::Builder::Level in the TAP tests

Incrementing the level of the call stack reported is useful for
debugging purposes as it allows to control which part of the test is
exactly failing, especially if a test is structured with subroutines
that call routines from Test::More.

This adds more incrementations of $Test::Builder::Level where debugging
gets improved (for example it does not make sense for some paths like
pg_rewind where long subroutines are used).

A note is added to src/test/perl/README about that, based on a
suggestion from Andrew Dunstan and a wording coming from both of us.

Usage of Test::Builder::Level has spread in 12, so a backpatch down to
this version is done.

Reviewed-by: Andrew Dunstan, Peter Eisentraut, Daniel Gustafsson
Discussion: https://postgr.es/m/YV1CCFwgM1RV1LeS@paquier.xyz
Backpatch-through: 12

Branch
------
REL_12_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/d5ab331a339d1a5f484e514e8cd231c291261dec

Modified Files
--------------
src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl |  2 ++
src/test/kerberos/t/001_auth.pl                      |  2 ++
src/test/perl/README                                 | 10 +++++++++-
src/test/recovery/t/001_stream_rep.pl                |  2 ++
src/test/recovery/t/003_recovery_targets.pl          |  2 ++
src/test/recovery/t/007_sync_rep.pl                  |  2 ++
src/test/recovery/t/009_twophase.pl                  |  2 ++
7 files changed, 21 insertions(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: pgsql: Add more $Test::Builder::Level in the TAP tests
@ 2021-10-14 03:14  Peter Geoghegan <pg@bowt.ie>
  parent: Michael Paquier <michael@paquier.xyz>
  0 siblings, 1 reply; 12+ messages in thread

From: Peter Geoghegan @ 2021-10-14 03:14 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-committers <pgsql-committers@lists.postgresql.org>

On Mon, Oct 11, 2021 at 7:17 PM Michael Paquier <michael@paquier.xyz> wrote:
> Add more $Test::Builder::Level in the TAP tests

I saw an issue just now that I suspect is linked with this commit. I
ran my parallel "make check-world" recipe, and saw this failure:

#   Failed test 'handling of unexpected PQresultStatus: matches'
#   at t/001_basic.pl line 43.
#                   'psql:/home/pg/.psqlrc:56: ERROR:  syntax error
# psql:/home/pg/.psqlrc:57: ERROR:  syntax error
# LOG:  received replication command: START_REPLICATION 0/0
# unexpected PQresultStatus: 8
# '
#     doesn't match '(?^:^unexpected PQresultStatus: 8$)'
# Looks like you failed 1 test of 25.
t/001_basic.pl ...........
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/25 subtests
t/010_tab_completion.pl .. ok
t/020_cancel.pl .......... ok

-- 
Peter Geoghegan





^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: pgsql: Add more $Test::Builder::Level in the TAP tests
@ 2021-10-14 03:51  Michael Paquier <michael@paquier.xyz>
  parent: Peter Geoghegan <pg@bowt.ie>
  0 siblings, 1 reply; 12+ messages in thread

From: Michael Paquier @ 2021-10-14 03:51 UTC (permalink / raw)
  To: Peter Geoghegan <pg@bowt.ie>; +Cc: pgsql-committers <pgsql-committers@lists.postgresql.org>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>

On Wed, Oct 13, 2021 at 08:14:20PM -0700, Peter Geoghegan wrote:
> On Mon, Oct 11, 2021 at 7:17 PM Michael Paquier <michael@paquier.xyz> wrote:
> > Add more $Test::Builder::Level in the TAP tests
> 
> I saw an issue just now that I suspect is linked with this commit. I
> ran my parallel "make check-world" recipe, and saw this failure:

This commit has not touched 001_basic.pl, nor does it touch any of the
modules we have in src/test/perl/, so that's unrelated.  In the worst
case, what you would get after this commit is a better report by
incrementing the level stack.

The problem is that this test script assumes that it is fine to not
use -X/--no-psqlrc in the commands of psql run, making this test prone
to fail depending on the local setup.  Please note that we pass down
-X through PostgresNode::psql, to keep the tests isolated from the
external world, so this could be fixed either by using -X but I think
that it would be better to switch to psql() with a comparison done to
the outputs of stdout, stderr and the error code returned.  The
failure can be reproduced easily once one adds some junk in their
local psqlrc.

Adding Peter E in CC, as the author of c0280bc & co.

> #   Failed test 'handling of unexpected PQresultStatus: matches'
> #   at t/001_basic.pl line 43.
> #                   'psql:/home/pg/.psqlrc:56: ERROR:  syntax error
> # psql:/home/pg/.psqlrc:57: ERROR:  syntax error
> # LOG:  received replication command: START_REPLICATION 0/0
> # unexpected PQresultStatus: 8
> # '

You may also want to fix your local .psqlrc, while on it.
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: pgsql: Add more $Test::Builder::Level in the TAP tests
@ 2021-10-14 03:54  Peter Geoghegan <pg@bowt.ie>
  parent: Michael Paquier <michael@paquier.xyz>
  0 siblings, 1 reply; 12+ messages in thread

From: Peter Geoghegan @ 2021-10-14 03:54 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-committers <pgsql-committers@lists.postgresql.org>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>

On Wed, Oct 13, 2021 at 8:51 PM Michael Paquier <michael@paquier.xyz> wrote:
> > #   Failed test 'handling of unexpected PQresultStatus: matches'
> > #   at t/001_basic.pl line 43.
> > #                   'psql:/home/pg/.psqlrc:56: ERROR:  syntax error
> > # psql:/home/pg/.psqlrc:57: ERROR:  syntax error
> > # LOG:  received replication command: START_REPLICATION 0/0
> > # unexpected PQresultStatus: 8
> > # '
>
> You may also want to fix your local .psqlrc, while on it.

Why, though? It doesn't do this in any other context.

-- 
Peter Geoghegan





^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: pgsql: Add more $Test::Builder::Level in the TAP tests
@ 2021-10-14 04:00  Michael Paquier <michael@paquier.xyz>
  parent: Peter Geoghegan <pg@bowt.ie>
  0 siblings, 1 reply; 12+ messages in thread

From: Michael Paquier @ 2021-10-14 04:00 UTC (permalink / raw)
  To: Peter Geoghegan <pg@bowt.ie>; +Cc: pgsql-committers <pgsql-committers@lists.postgresql.org>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>

On Wed, Oct 13, 2021 at 08:54:16PM -0700, Peter Geoghegan wrote:
> On Wed, Oct 13, 2021 at 8:51 PM Michael Paquier <michael@paquier.xyz> wrote:
>>> #   Failed test 'handling of unexpected PQresultStatus: matches'
>>> #   at t/001_basic.pl line 43.
>>> #                   'psql:/home/pg/.psqlrc:56: ERROR:  syntax error
>>> # psql:/home/pg/.psqlrc:57: ERROR:  syntax error
>>> # LOG:  received replication command: START_REPLICATION 0/0
>>> # unexpected PQresultStatus: 8
>>> # '
>>
>> You may also want to fix your local .psqlrc, while on it.
> 
> Why, though? It doesn't do this in any other context.

Even for single psql commands in your environment?  That would be
surprising that this does not show the same error about lines 56/57 in
/home/pg/.psqlrc, but I guess that's up to you for this part..

It does not change the fact that the test needs to be fixed.  TAP
tests should be designed to not be influenced by the external world.
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: pgsql: Add more $Test::Builder::Level in the TAP tests
@ 2021-10-14 04:04  Peter Geoghegan <pg@bowt.ie>
  parent: Michael Paquier <michael@paquier.xyz>
  0 siblings, 1 reply; 12+ messages in thread

From: Peter Geoghegan @ 2021-10-14 04:04 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-committers <pgsql-committers@lists.postgresql.org>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>

On Wed, Oct 13, 2021 at 9:00 PM Michael Paquier <michael@paquier.xyz> wrote:
> Even for single psql commands in your environment?  That would be
> surprising that this does not show the same error about lines 56/57 in
> /home/pg/.psqlrc, but I guess that's up to you for this part..

It's doing this:

-------------------------------------------------
-- Source custom variables from dedicated file --
-------------------------------------------------
\set queries `cat $HOME/dotfiles/psql-rc-queries.sql`
\o /dev/null
:queries -- this is line 56
\gset

I've been doing this for many years now. It's nothing new.

> It does not change the fact that the test needs to be fixed.  TAP
> tests should be designed to not be influenced by the external world.

Right.

-- 
Peter Geoghegan





^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: pgsql: Add more $Test::Builder::Level in the TAP tests
@ 2021-10-16 06:22  Michael Paquier <michael@paquier.xyz>
  parent: Peter Geoghegan <pg@bowt.ie>
  0 siblings, 1 reply; 12+ messages in thread

From: Michael Paquier @ 2021-10-16 06:22 UTC (permalink / raw)
  To: Peter Geoghegan <pg@bowt.ie>; +Cc: pgsql-committers <pgsql-committers@lists.postgresql.org>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>

On Wed, Oct 13, 2021 at 09:04:11PM -0700, Peter Geoghegan wrote:
> On Wed, Oct 13, 2021 at 9:00 PM Michael Paquier <michael@paquier.xyz> wrote:
>> It does not change the fact that the test needs to be fixed.  TAP
>> tests should be designed to not be influenced by the external world.
> 
> Right.

Hearing nothing, I have just poked at this problem.  And I'd rather
make all the tests go through PostgresNode::psql from which it is
possible to grab stdout, stderr and the result code, as we could miss
local changes in this test if we need a centralized fix, for problems
like the one with SIGPIPE where we don't use -c in a psql command.
Peter, does it take care of your problem?
--
Michael

Attachments:

psql-tap-fix.patch (text/x-diff, 2.6K) download diff --git a/src/bin/psql/t/001_basic.pl b/src/bin/psql/t/001_basic.pl index 685744744a..eb9041d1f3 100644 --- a/src/bin/psql/t/001_basic.pl +++ b/src/bin/psql/t/001_basic.pl @@ -15,10 +15,46 @@ program_options_handling_ok('psql'); my ($stdout, $stderr); my $result; +# Execute a psql command and check its result patterns. +sub psql_like +{ + local $Test::Builder::Level = $Test::Builder::Level + 1; + + my $node = shift; + my $test_name = shift; + my $query = shift; + my $expected_stdout = shift; + my $expected_stderr = shift; + + die "cannot specify both expected stdout and stderr here" + if (defined($expected_stdout) && defined($expected_stderr)); + + # Use the context of a WAL sender, some of the tests rely on that. + my ($ret, $stdout, $stderr) = $node->psql( + 'postgres', $query, + on_error_die => 0, + replication => 'database'); + + if (defined($expected_stdout)) + { + is($ret, 0, "$test_name: expected result code"); + is($stderr, '', "$test_name: no stderr"); + like($stdout, $expected_stdout, "$test_name: stdout matches"); + } + if (defined($expected_stderr)) + { + isnt($ret, 0, "$test_name: expected result code"); + like($stderr, $expected_stderr, "$test_name: stderr matches"); + } + + return; +} + # test --help=foo, analogous to program_help_ok() foreach my $arg (qw(commands variables)) { - $result = IPC::Run::run [ 'psql', "--help=$arg" ], '>', \$stdout, '2>', \$stderr; + $result = IPC::Run::run [ 'psql', "--help=$arg" ], '>', \$stdout, '2>', + \$stderr; ok($result, "psql --help=$arg exit code 0"); isnt($stdout, '', "psql --help=$arg goes to stdout"); is($stderr, '', "psql --help=$arg nothing to stderr"); @@ -34,11 +70,13 @@ max_wal_senders = 4 }); $node->start; -$node->command_like([ 'psql', '-c', '\copyright' ], qr/Copyright/, '\copyright'); -$node->command_like([ 'psql', '-c', '\help' ], qr/ALTER/, '\help without arguments'); -$node->command_like([ 'psql', '-c', '\help SELECT' ], qr/SELECT/, '\help'); - +psql_like($node, '\copyright', '\copyright', qr/Copyright/, undef); +psql_like($node, '\help without arguments', '\help', qr/ALTER/, undef); +psql_like($node, '\help with argument', '\help SELECT', qr/SELECT/, undef); # Test clean handling of unsupported replication command responses -$node->command_fails_like([ 'psql', '-d', 'replication=database', '-c', 'START_REPLICATION 0/0' ], - qr/^unexpected PQresultStatus: 8$/, 'handling of unexpected PQresultStatus'); +psql_like( + $node, + 'handling of unexpected PQresultStatus', + 'START_REPLICATION 0/0', + undef, qr/unexpected PQresultStatus: 8$/);
signature.asc (application/pgp-signature, 832B) download ^ permalink raw reply [nested|flat] 12+ messages in thread

* Re: pgsql: Add more $Test::Builder::Level in the TAP tests
@ 2021-10-17 03:08  Peter Geoghegan <pg@bowt.ie>
  parent: Michael Paquier <michael@paquier.xyz>
  0 siblings, 1 reply; 12+ messages in thread

From: Peter Geoghegan @ 2021-10-17 03:08 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-committers <pgsql-committers@lists.postgresql.org>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>

On Fri, Oct 15, 2021 at 11:22 PM Michael Paquier <michael@paquier.xyz> wrote:
> Hearing nothing, I have just poked at this problem.

> Peter, does it take care of your problem?

Yes, it does.

Thank you for taking care of this.

--
Peter Geoghegan





^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: pgsql: Add more $Test::Builder::Level in the TAP tests
@ 2021-10-18 01:04  Michael Paquier <michael@paquier.xyz>
  parent: Peter Geoghegan <pg@bowt.ie>
  0 siblings, 0 replies; 12+ messages in thread

From: Michael Paquier @ 2021-10-18 01:04 UTC (permalink / raw)
  To: Peter Geoghegan <pg@bowt.ie>; +Cc: pgsql-committers <pgsql-committers@lists.postgresql.org>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>

On Sat, Oct 16, 2021 at 08:08:28PM -0700, Peter Geoghegan wrote:
> Thank you for taking care of this.

Okay.  Done, then.
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 12+ messages in thread


end of thread, other threads:[~2021-10-18 01:04 UTC | newest]

Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-10-12 02:17 pgsql: Add more $Test::Builder::Level in the TAP tests Michael Paquier <michael@paquier.xyz>
2021-10-12 02:17 pgsql: Add more $Test::Builder::Level in the TAP tests Michael Paquier <michael@paquier.xyz>
2021-10-14 03:14 ` Peter Geoghegan <pg@bowt.ie>
2021-10-14 03:51   ` Michael Paquier <michael@paquier.xyz>
2021-10-14 03:54     ` Peter Geoghegan <pg@bowt.ie>
2021-10-14 04:00       ` Michael Paquier <michael@paquier.xyz>
2021-10-14 04:04         ` Peter Geoghegan <pg@bowt.ie>
2021-10-16 06:22           ` Michael Paquier <michael@paquier.xyz>
2021-10-17 03:08             ` Peter Geoghegan <pg@bowt.ie>
2021-10-18 01:04               ` Michael Paquier <michael@paquier.xyz>
2021-10-12 02:17 pgsql: Add more $Test::Builder::Level in the TAP tests Michael Paquier <michael@paquier.xyz>
2021-10-12 02:17 pgsql: Add more $Test::Builder::Level in the TAP tests Michael Paquier <michael@paquier.xyz>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox