pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedA test for replay of regression tests
82+ messages / 11 participants
[nested] [flat]
Commits from this thread:
2022-04-04 cbf4177f2c Disable synchronize_seqscans in 027_stream_regress.pl.
2022-01-27 7340aceed7 Specify --host in 027_stream_regress.pl's pg_regress invocation.
2022-01-21 cfe7bd17e4 Add new simple TAP test for tablespaces, attempt II.
* A test for replay of regression tests
@ 2021-04-23 05:37 Thomas Munro <thomas.munro@gmail.com>
0 siblings, 3 replies; 82+ messages in thread
From: Thomas Munro @ 2021-04-23 05:37 UTC (permalink / raw)
To: pgsql-hackers
Hi,
We have automated tests for many specific replication and recovery
scenarios, but nothing that tests replay of a wide range of records.
People working on recovery code often use installcheck (presumably
along with other custom tests) to exercise it, sometimes with
wal_consistency_check enabled. So, why don't we automate that? Aside
from exercising the WAL decoding machinery (which brought me here),
that'd hopefully provide some decent improvements in coverage of the
various redo routines, many of which are not currently exercised at
all.
I'm not quite sure where it belongs, though. The attached initial
sketch patch puts it under rc/test/recovery near other similar things,
but I'm not sure if it's really OK to invoke make -C ../regress from
here. I copied pg_update/test.sh's trick of using a different
outputdir to avoid clobbering a concurrent run under src/test/regress,
and I also needed to invent a way to stop it from running the cursed
tablespace test (deferring startup of the standby also works but eats
way too much space, which I learned after blowing out a smallish
development VM's disk). Alternatively, we could put it under
src/test/regress, which makes some kind of logical sense, but it would
make a quick "make check" take more than twice as long. Perhaps it
could use a different target, one that check-world somehow reaches but
not check, and that also doesn't seem great. Ideas on how to
structure this or improve the perl welcome.
Attachments:
0001-Test-replay-of-regression-tests.patch (text/x-patch, 6.3K) download
From f7d199cefdec98bcdc5f580e5175ce09cb1299b2 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Fri, 23 Apr 2021 15:37:09 +1200
Subject: [PATCH] Test replay of regression tests.
Add a new TAP test under src/test/recovery that runs the regression
tests with wal_consistency_checking=all.
---
src/test/recovery/Makefile | 3 +
src/test/recovery/t/025_stream_rep_regress.pl | 67 +++++++++++++++++++
src/test/regress/pg_regress.c | 27 ++++++++
3 files changed, 97 insertions(+)
create mode 100644 src/test/recovery/t/025_stream_rep_regress.pl
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
index 96442ceb4e..7dfcbae4b5 100644
--- a/src/test/recovery/Makefile
+++ b/src/test/recovery/Makefile
@@ -18,6 +18,9 @@ include $(top_builddir)/src/Makefile.global
check:
$(prove_check)
+# Tell the TAP tests how to invoke make.
+export MAKE
+
installcheck:
$(prove_installcheck)
diff --git a/src/test/recovery/t/025_stream_rep_regress.pl b/src/test/recovery/t/025_stream_rep_regress.pl
new file mode 100644
index 0000000000..a7dbe54322
--- /dev/null
+++ b/src/test/recovery/t/025_stream_rep_regress.pl
@@ -0,0 +1,67 @@
+# Run the standard regression tests with streaming replication
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+# Initialize primary node
+my $node_primary = get_new_node('primary');
+# A specific role is created to perform some tests related to replication,
+# and it needs proper authentication configuration.
+$node_primary->init(
+ allows_streaming => 1,
+ auth_extra => [ '--create-role', 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', 'wal_consistency_checking = all');
+$node_primary->start;
+is( $node_primary->psql(
+ 'postgres',
+ qq[SELECT pg_create_physical_replication_slot('standby_1');]),
+ 0,
+ 'physical slot created on primary');
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_primary->backup($backup_name);
+
+# Create streaming standby linking to primary
+my $node_standby_1 = get_new_node('standby_1');
+$node_standby_1->init_from_backup($node_primary, $backup_name,
+ has_streaming => 1);
+$node_standby_1->append_conf('postgresql.conf',
+ "primary_slot_name = standby_1");
+$node_standby_1->start;
+
+# Create some content on primary and check its presence in standby 1
+$node_primary->safe_psql('postgres',
+ "CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a");
+
+# Wait for standby to catch up
+$node_primary->wait_for_catchup($node_standby_1, 'replay',
+ $node_primary->lsn('insert'));
+
+my $result =
+ $node_standby_1->safe_psql('postgres', "SELECT count(*) FROM tab_int");
+print "standby 1: $result\n";
+is($result, qq(1002), 'check streamed content on standby 1');
+
+# XXX The tablespace tests don't currently work when the standby shares a
+# filesystem with the primary due to colliding absolute paths. We'll skip
+# that for now.
+
+# Run the regression tests against the primary.
+system_or_bail($ENV{MAKE}, '-C', '../regress', 'installcheck',
+ 'PGPORT=' . $node_primary->port,
+ 'EXTRA_REGRESS_OPTS=--skip-tests=tablespace --outputdir=' . TestLib::tempdir);
+
+# Wait for standby to catch up and do a sanity check
+$node_primary->safe_psql('postgres', "INSERT INTO tab_int VALUES (42)");
+$node_primary->wait_for_catchup($node_standby_1, 'replay',
+ $node_primary->lsn('insert'));
+$result =
+ $node_standby_1->safe_psql('postgres', "SELECT count(*) FROM tab_int");
+print "standby 1: $result\n";
+is($result, qq(1003), 'check streamed content on standby 1');
+
+$node_standby_1->stop;
+$node_primary->stop;
diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index b7d80bd9bb..b75eccfa7c 100644
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -83,6 +83,7 @@ static int max_connections = 0;
static int max_concurrent_tests = 0;
static char *encoding = NULL;
static _stringlist *schedulelist = NULL;
+static _stringlist *skip_tests = NULL;
static _stringlist *extra_tests = NULL;
static char *temp_instance = NULL;
static _stringlist *temp_configs = NULL;
@@ -203,6 +204,22 @@ split_to_stringlist(const char *s, const char *delim, _stringlist **listhead)
free(sc);
}
+/*
+ * Test if a string is in a list.
+ */
+static bool
+string_in_stringlist(const char *s, _stringlist *list)
+{
+ while (list)
+ {
+ if (strcmp(list->str, s) == 0)
+ return true;
+ list = list->next;
+ }
+
+ return false;
+}
+
/*
* Print a progress banner on stdout.
*/
@@ -1662,7 +1679,11 @@ run_schedule(const char *schedule, test_start_function startfunc,
if (scbuf[0] == '\0' || scbuf[0] == '#')
continue;
if (strncmp(scbuf, "test: ", 6) == 0)
+ {
test = scbuf + 6;
+ if (string_in_stringlist(test, skip_tests))
+ continue;
+ }
else if (strncmp(scbuf, "ignore: ", 8) == 0)
{
c = scbuf + 8;
@@ -1860,6 +1881,7 @@ run_schedule(const char *schedule, test_start_function startfunc,
}
free_stringlist(&ignorelist);
+ free_stringlist(&skip_tests);
fclose(scf);
}
@@ -2065,6 +2087,7 @@ help(void)
printf(_(" --outputdir=DIR place output files in DIR (default \".\")\n"));
printf(_(" --schedule=FILE use test ordering schedule from FILE\n"));
printf(_(" (can be used multiple times to concatenate)\n"));
+ printf(_(" --skip-tests=LIST comma-separated list of tests to skip\n"));
printf(_(" --temp-instance=DIR create a temporary instance in DIR\n"));
printf(_(" --use-existing use an existing installation\n"));
printf(_(" -V, --version output version information, then exit\n"));
@@ -2116,6 +2139,7 @@ regression_main(int argc, char *argv[],
{"load-extension", required_argument, NULL, 22},
{"config-auth", required_argument, NULL, 24},
{"max-concurrent-tests", required_argument, NULL, 25},
+ {"skip-tests", required_argument, NULL, 26},
{NULL, 0, NULL, 0}
};
@@ -2245,6 +2269,9 @@ regression_main(int argc, char *argv[],
case 25:
max_concurrent_tests = atoi(optarg);
break;
+ case 26:
+ split_to_stringlist(optarg, ",", &skip_tests);
+ break;
default:
/* getopt_long already emitted a complaint */
fprintf(stderr, _("\nTry \"%s -h\" for more information.\n"),
--
2.30.2
* RE: A test for replay of regression tests
@ 2021-04-23 06:27 tsunakawa.takay@fujitsu.com <tsunakawa.takay@fujitsu.com>
parent: Thomas Munro <thomas.munro@gmail.com>
2 siblings, 1 reply; 82+ messages in thread
From: tsunakawa.takay@fujitsu.com @ 2021-04-23 06:27 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: pgsql-hackers
From: Thomas Munro <thomas.munro@gmail.com>
> We have automated tests for many specific replication and recovery scenarios,
> but nothing that tests replay of a wide range of records.
> People working on recovery code often use installcheck (presumably along
> with other custom tests) to exercise it, sometimes with
> wal_consistency_check enabled. So, why don't we automate that? Aside
> from exercising the WAL decoding machinery (which brought me here), that'd
> hopefully provide some decent improvements in coverage of the various redo
> routines, many of which are not currently exercised at all.
>
> I'm not quite sure where it belongs, though. The attached initial sketch patch
I think that's a good attempt. It'd be better to confirm that the database contents are identical on the primary and standby. But... I remember when I ran make installcheck with a standby connected, then ran pg_dumpall on both the primary and standby and compare the two output files, about 40 lines of difference were observed. Those differences were all about the sequence values. I didn't think about whether I should/can remove the differences.
+# XXX The tablespace tests don't currently work when the standby shares a
+# filesystem with the primary due to colliding absolute paths. We'll skip
+# that for now.
Maybe we had better have a hidden feature that creates tablespace contents in
/tablespace_location/PG_..._<some_name>/
<some_name> is the value of cluster_name or application_name.
Or, we may provide a visible feature that allows users to store tablespace contents in a specified directory regardless of the LOCATION value in CREATE TABLESPACE. For instance, add a GUC like
table_space_directory = '/some_dir'
Then, the tablespace contents are stored in /some_dir/<tablespace_name>/. This may be useful if a DBaaS provider wants to offer some tablespace-based feature, say tablespace transparent data encryption, but doesn't want users to specify filesystem directories.
Regards
Takayuki Tsunakawa
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-04-23 07:51 Thomas Munro <thomas.munro@gmail.com>
parent: tsunakawa.takay@fujitsu.com <tsunakawa.takay@fujitsu.com>
0 siblings, 0 replies; 82+ messages in thread
From: Thomas Munro @ 2021-04-23 07:51 UTC (permalink / raw)
To: tsunakawa.takay@fujitsu.com <tsunakawa.takay@fujitsu.com>; +Cc: pgsql-hackers
On Fri, Apr 23, 2021 at 6:27 PM tsunakawa.takay@fujitsu.com
<tsunakawa.takay@fujitsu.com> wrote:
> From: Thomas Munro <thomas.munro@gmail.com>
> > I'm not quite sure where it belongs, though. The attached initial sketch patch
>
> I think that's a good attempt. It'd be better to confirm that the database contents are identical on the primary and standby. But... I remember when I ran make installcheck with a standby connected, then ran pg_dumpall on both the primary and standby and compare the two output files, about 40 lines of difference were observed. Those differences were all about the sequence values. I didn't think about whether I should/can remove the differences.
Interesting idea. I hadn't noticed the thing with sequences before.
> +# XXX The tablespace tests don't currently work when the standby shares a
> +# filesystem with the primary due to colliding absolute paths. We'll skip
> +# that for now.
>
> Maybe we had better have a hidden feature that creates tablespace contents in
>
> /tablespace_location/PG_..._<some_name>/
>
> <some_name> is the value of cluster_name or application_name.
>
> Or, we may provide a visible feature that allows users to store tablespace contents in a specified directory regardless of the LOCATION value in CREATE TABLESPACE. For instance, add a GUC like
>
> table_space_directory = '/some_dir'
>
> Then, the tablespace contents are stored in /some_dir/<tablespace_name>/. This may be useful if a DBaaS provider wants to offer some tablespace-based feature, say tablespace transparent data encryption, but doesn't want users to specify filesystem directories.
Yeah, a few similar ideas have been put forward before, for example in
this thread:
https://www.postgresql.org/message-id/flat/CALfoeisEF92F5nJ-aAcuWTvF_Aogxq_1bHLem_kVfM_tHc2mfg%40mai...
... but also others. I would definitely like to fix that problem too
(having worked on several things that interact with tablespaces, I
definitely want to be able to extend those tests in future patches,
and get coverage on the build farm and CI), but with --skip-tests that
could be done independently.
Apparently the invocation of make failed for some reason on CI (not
sure why, works for me), but in any case, that feels a bit clunky and
might not ever work on Windows, so perhaps we should figure out how to
invoke the pg_regress[.exe] program directly.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-04-23 15:20 Andres Freund <andres@anarazel.de>
parent: Thomas Munro <thomas.munro@gmail.com>
2 siblings, 1 reply; 82+ messages in thread
From: Andres Freund @ 2021-04-23 15:20 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: pgsql-hackers
Hi,
On 2021-04-23 17:37:48 +1200, Thomas Munro wrote:
> We have automated tests for many specific replication and recovery
> scenarios, but nothing that tests replay of a wide range of records.
> People working on recovery code often use installcheck (presumably
> along with other custom tests) to exercise it, sometimes with
> wal_consistency_check enabled. So, why don't we automate that? Aside
> from exercising the WAL decoding machinery (which brought me here),
> that'd hopefully provide some decent improvements in coverage of the
> various redo routines, many of which are not currently exercised at
> all.
Yay.
> I'm not quite sure where it belongs, though. The attached initial
> sketch patch puts it under rc/test/recovery near other similar things,
> but I'm not sure if it's really OK to invoke make -C ../regress from
> here.
I'd say it's not ok, and we should just invoke pg_regress without make.
> Add a new TAP test under src/test/recovery that runs the regression
> tests with wal_consistency_checking=all.
Hm. I wonder if running with wal_consistency_checking=all doesn't also
reduce coverage of some aspects of recovery, by increasing record sizes
etc.
> I copied pg_update/test.sh's trick of using a different
> outputdir to avoid clobbering a concurrent run under src/test/regress,
> and I also needed to invent a way to stop it from running the cursed
> tablespace test (deferring startup of the standby also works but eats
> way too much space, which I learned after blowing out a smallish
> development VM's disk).
That's because you are using wal_consistency_checking=all, right?
Because IIRC we don't generate that much WAL otherwise?
> +# Create some content on primary and check its presence in standby 1
> +$node_primary->safe_psql('postgres',
> + "CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a");
> +
> +# Wait for standby to catch up
> +$node_primary->wait_for_catchup($node_standby_1, 'replay',
> + $node_primary->lsn('insert'));
> +my $result =
> + $node_standby_1->safe_psql('postgres', "SELECT count(*) FROM tab_int");
> +print "standby 1: $result\n";
> +is($result, qq(1002), 'check streamed content on standby 1');
Why is this needed?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-04-23 15:38 Andrew Dunstan <andrew@dunslane.net>
parent: Thomas Munro <thomas.munro@gmail.com>
2 siblings, 0 replies; 82+ messages in thread
From: Andrew Dunstan @ 2021-04-23 15:38 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers
On 4/23/21 1:37 AM, Thomas Munro wrote:
> Hi,
>
> We have automated tests for many specific replication and recovery
> scenarios, but nothing that tests replay of a wide range of records.
> People working on recovery code often use installcheck (presumably
> along with other custom tests) to exercise it, sometimes with
> wal_consistency_check enabled. So, why don't we automate that? Aside
> from exercising the WAL decoding machinery (which brought me here),
> that'd hopefully provide some decent improvements in coverage of the
> various redo routines, many of which are not currently exercised at
> all.
>
> I'm not quite sure where it belongs, though. The attached initial
> sketch patch puts it under rc/test/recovery near other similar things,
> but I'm not sure if it's really OK to invoke make -C ../regress from
> here. I copied pg_update/test.sh's trick of using a different
> outputdir to avoid clobbering a concurrent run under src/test/regress,
> and I also needed to invent a way to stop it from running the cursed
> tablespace test (deferring startup of the standby also works but eats
> way too much space, which I learned after blowing out a smallish
> development VM's disk). Alternatively, we could put it under
> src/test/regress, which makes some kind of logical sense, but it would
> make a quick "make check" take more than twice as long. Perhaps it
> could use a different target, one that check-world somehow reaches but
> not check, and that also doesn't seem great. Ideas on how to
> structure this or improve the perl welcome.
Nice, I like adding a skip-tests option to pg_regress.
The perl looks ok - I assume those
print "standby 1: $result\n";
lines are there for debugging. Normally you would just process $result
using the Test::More functions
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-04-23 15:53 Tom Lane <tgl@sss.pgh.pa.us>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 82+ messages in thread
From: Tom Lane @ 2021-04-23 15:53 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers
Andres Freund <andres@anarazel.de> writes:
> On 2021-04-23 17:37:48 +1200, Thomas Munro wrote:
>> We have automated tests for many specific replication and recovery
>> scenarios, but nothing that tests replay of a wide range of records.
> Yay.
+1
>> Add a new TAP test under src/test/recovery that runs the regression
>> tests with wal_consistency_checking=all.
> Hm. I wonder if running with wal_consistency_checking=all doesn't also
> reduce coverage of some aspects of recovery, by increasing record sizes
> etc.
Yeah. I found out earlier that wal_consistency_checking=all is an
absolute PIG. Running the regression tests that way requires tens of
gigabytes of disk space, and a significant amount of time if your
disk is not very speedy. If we put this into the buildfarm at all,
it would have to be opt-in, not run-by-default, because a lot of BF
animals simply don't have the horsepower. I think I'd vote against
adding it to check-world, too; the cost/benefit ratio is not good
unless you are specifically working on replay functions.
And as you say, it alters the behavior enough to make it a little
questionable whether we're exercising the "normal" code paths.
The two things I'd say about this are (1) Whether to use
wal_consistency_checking, and with what value, needs to be
easily adjustable. (2) People will want to run other test suites
than the core regression tests, eg contrib modules.
regards, tom lane
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-04-23 17:04 Andres Freund <andres@anarazel.de>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 82+ messages in thread
From: Andres Freund @ 2021-04-23 17:04 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers
Hi,
On 2021-04-23 11:53:35 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > Hm. I wonder if running with wal_consistency_checking=all doesn't also
> > reduce coverage of some aspects of recovery, by increasing record sizes
> > etc.
>
> Yeah. I found out earlier that wal_consistency_checking=all is an
> absolute PIG. Running the regression tests that way requires tens of
> gigabytes of disk space, and a significant amount of time if your
> disk is not very speedy. If we put this into the buildfarm at all,
> it would have to be opt-in, not run-by-default, because a lot of BF
> animals simply don't have the horsepower. I think I'd vote against
> adding it to check-world, too; the cost/benefit ratio is not good
> unless you are specifically working on replay functions.
I think it'd be a huge improvement to test recovery during check-world
by default - it's a huge swath of crucial code that practically has no
test coverage. I agree that testing by default with
wal_consistency_checking=all isn't feasible due to the time & space
overhead, so I think we should not do that.
> The two things I'd say about this are (1) Whether to use
> wal_consistency_checking, and with what value, needs to be
> easily adjustable. (2) People will want to run other test suites
> than the core regression tests, eg contrib modules.
I'm not really excited about tackling 2) initially. I think it's a
significant issue that we don't have test coverage for most redo
routines and that we should change that with some urgency - even if we
back out these WAL prefetch related changes, there've been sufficiently
many changes affecting WAL that I think it's worth doing the minimal
thing soon.
I don't think there's actually that much need to test contrib modules
through recovery - most of them don't seem like they'd add meaningful
coverage? The exception is contrib/bloom, but perhaps that'd be better
tackled with a dedicated test?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-04-23 17:13 Tom Lane <tgl@sss.pgh.pa.us>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 82+ messages in thread
From: Tom Lane @ 2021-04-23 17:13 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers
Andres Freund <andres@anarazel.de> writes:
> On 2021-04-23 11:53:35 -0400, Tom Lane wrote:
>> Yeah. I found out earlier that wal_consistency_checking=all is an
>> absolute PIG. Running the regression tests that way requires tens of
>> gigabytes of disk space, and a significant amount of time if your
>> disk is not very speedy. If we put this into the buildfarm at all,
>> it would have to be opt-in, not run-by-default, because a lot of BF
>> animals simply don't have the horsepower. I think I'd vote against
>> adding it to check-world, too; the cost/benefit ratio is not good
>> unless you are specifically working on replay functions.
> I think it'd be a huge improvement to test recovery during check-world
> by default - it's a huge swath of crucial code that practically has no
> test coverage. I agree that testing by default with
> wal_consistency_checking=all isn't feasible due to the time & space
> overhead, so I think we should not do that.
I was mainly objecting to enabling wal_consistency_checking by default.
I agree it's bad that we have so little routine test coverage on WAL
replay code.
>> The two things I'd say about this are (1) Whether to use
>> wal_consistency_checking, and with what value, needs to be
>> easily adjustable. (2) People will want to run other test suites
>> than the core regression tests, eg contrib modules.
> I don't think there's actually that much need to test contrib modules
> through recovery - most of them don't seem like they'd add meaningful
> coverage? The exception is contrib/bloom, but perhaps that'd be better
> tackled with a dedicated test?
contrib/bloom is indeed the only(?) case within contrib, but in my mind
that's a proxy for what people will be needing to test out-of-core AMs.
It might not be a test to run by default, but there needs to be a way
to do it.
Also, I suspect that there are bits of GIST/GIN/SPGIST that are not
well exercised if you don't run the contrib modules that add opclasses
for those.
regards, tom lane
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-04-23 17:22 Andres Freund <andres@anarazel.de>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 82+ messages in thread
From: Andres Freund @ 2021-04-23 17:22 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers
Hi,
On 2021-04-23 13:13:15 -0400, Tom Lane wrote:
> contrib/bloom is indeed the only(?) case within contrib, but in my mind
> that's a proxy for what people will be needing to test out-of-core AMs.
> It might not be a test to run by default, but there needs to be a way
> to do it.
Hm. My experience in the past was that the best way to test an external
AM is to run the core regression tests with a different
default_table_access_method. That does require some work of ensuring the
AM is installed and the relevant extension created, which in turn
requires a different test schedule, or hacking template1. So likely a
different test script anyway?
> Also, I suspect that there are bits of GIST/GIN/SPGIST that are not
> well exercised if you don't run the contrib modules that add opclasses
> for those.
Possible - still think it'd be best to get the minimal thing in asap,
and then try to extend further afterwards... Perfect being the enemy of
good and all that.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-05-04 11:12 Thomas Munro <thomas.munro@gmail.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2021-05-04 11:12 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers; Andrew Dunstan <andrew@dunslane.net>
Ok, here's a new version incorporating feedback so far.
1. Invoke pg_regress directly (no make).
2. Use PG_TEST_EXTRA="wal_consistency_checking" as a way to opt in to
the more expensive test.
3. Use parallel schedule rather than serial. It's faster but also
the non-determinism might discover more things. This required
changing the TAP test max_connections setting from 10 to 25.
4. Remove some extraneous print statements and
check-if-data-is-replicated-using-SELECT tests that are technically
not needed (I had copied those from 001_stream_rep.pl).
Attachments:
v2-0001-Test-replay-of-regression-tests.patch (text/x-patch, 7.2K) download
From c14c839be91b4b7b534c899370a5f450f118fed3 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Fri, 23 Apr 2021 15:37:09 +1200
Subject: [PATCH v2] Test replay of regression tests.
Add a new TAP test under src/test/recovery to run the standard
regression tests with a streaming replica replaying the WAL. This
provides a basic workout for WAL decoding and redo code.
Optionally, enable (expensive) wal_consistency_checking if configured in
PG_TEST_EXTRA.
For now, skip the tablespace tests with a new option to pg_regress,
until we have way to run them with a replica on the same file system.
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
doc/src/sgml/regress.sgml | 11 ++++
src/test/perl/PostgresNode.pm | 2 +-
src/test/recovery/t/025_stream_rep_regress.pl | 62 +++++++++++++++++++
src/test/regress/pg_regress.c | 27 ++++++++
4 files changed, 101 insertions(+), 1 deletion(-)
create mode 100644 src/test/recovery/t/025_stream_rep_regress.pl
diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml
index cb401a45b3..7a10c83d8a 100644
--- a/doc/src/sgml/regress.sgml
+++ b/doc/src/sgml/regress.sgml
@@ -289,6 +289,17 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl'
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>wal_consistency_checking</literal></term>
+ <listitem>
+ <para>
+ Uses <literal>wal_consistency_checking=all</literal> while running
+ some of the tests under <filename>src/test/recovery</filename>. Not
+ enabled by default because it is resource intensive.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
Tests for features that are not supported by the current build
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 0eb8df5fbf..9fdf1eb060 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -482,7 +482,7 @@ sub init
print $conf "hot_standby = on\n";
# conservative settings to ensure we can run multiple postmasters:
print $conf "shared_buffers = 1MB\n";
- print $conf "max_connections = 10\n";
+ print $conf "max_connections = 25\n";
# limit disk space consumption, too:
print $conf "max_wal_size = 128MB\n";
}
diff --git a/src/test/recovery/t/025_stream_rep_regress.pl b/src/test/recovery/t/025_stream_rep_regress.pl
new file mode 100644
index 0000000000..8b125a1d67
--- /dev/null
+++ b/src/test/recovery/t/025_stream_rep_regress.pl
@@ -0,0 +1,62 @@
+# Run the standard regression tests with streaming replication
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+# Initialize primary node
+my $node_primary = get_new_node('primary');
+# A specific role is created to perform some tests related to replication,
+# and it needs proper authentication configuration.
+$node_primary->init(
+ allows_streaming => 1,
+ auth_extra => [ '--create-role', 'repl_role' ]);
+
+# WAL consistency checking is resource intensive so require opt-in with the
+# PG_TEST_EXTRA environment variable.
+if ($ENV{PG_TEST_EXTRA} &&
+ $ENV{PG_TEST_EXTRA} =~ m/\bwal_consistency_checking\b/) {
+ $node_primary->append_conf('postgresql.conf',
+ 'wal_consistency_checking = all');
+}
+
+$node_primary->start;
+is( $node_primary->psql(
+ 'postgres',
+ qq[SELECT pg_create_physical_replication_slot('standby_1');]),
+ 0,
+ 'physical slot created on primary');
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_primary->backup($backup_name);
+
+# Create streaming standby linking to primary
+my $node_standby_1 = get_new_node('standby_1');
+$node_standby_1->init_from_backup($node_primary, $backup_name,
+ has_streaming => 1);
+$node_standby_1->append_conf('postgresql.conf',
+ "primary_slot_name = standby_1");
+$node_standby_1->start;
+
+# XXX The tablespace tests don't currently work when the standby shares a
+# filesystem with the primary, due to colliding absolute paths. We'll skip
+# that for now.
+
+# Run the regression tests against the primary.
+system_or_bail("../regress/pg_regress",
+ "--port=" . $node_primary->port,
+ "--schedule=../regress/parallel_schedule",
+ "--dlpath=../regress",
+ "--inputdir=../regress",
+ "--skip-tests=tablespace");
+
+# Wait for standby to catch up
+$node_primary->wait_for_catchup($node_standby_1, 'replay',
+ $node_primary->lsn('insert'));
+
+ok(1, "caught up");
+
+$node_standby_1->stop;
+$node_primary->stop;
diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index b7d80bd9bb..b75eccfa7c 100644
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -83,6 +83,7 @@ static int max_connections = 0;
static int max_concurrent_tests = 0;
static char *encoding = NULL;
static _stringlist *schedulelist = NULL;
+static _stringlist *skip_tests = NULL;
static _stringlist *extra_tests = NULL;
static char *temp_instance = NULL;
static _stringlist *temp_configs = NULL;
@@ -203,6 +204,22 @@ split_to_stringlist(const char *s, const char *delim, _stringlist **listhead)
free(sc);
}
+/*
+ * Test if a string is in a list.
+ */
+static bool
+string_in_stringlist(const char *s, _stringlist *list)
+{
+ while (list)
+ {
+ if (strcmp(list->str, s) == 0)
+ return true;
+ list = list->next;
+ }
+
+ return false;
+}
+
/*
* Print a progress banner on stdout.
*/
@@ -1662,7 +1679,11 @@ run_schedule(const char *schedule, test_start_function startfunc,
if (scbuf[0] == '\0' || scbuf[0] == '#')
continue;
if (strncmp(scbuf, "test: ", 6) == 0)
+ {
test = scbuf + 6;
+ if (string_in_stringlist(test, skip_tests))
+ continue;
+ }
else if (strncmp(scbuf, "ignore: ", 8) == 0)
{
c = scbuf + 8;
@@ -1860,6 +1881,7 @@ run_schedule(const char *schedule, test_start_function startfunc,
}
free_stringlist(&ignorelist);
+ free_stringlist(&skip_tests);
fclose(scf);
}
@@ -2065,6 +2087,7 @@ help(void)
printf(_(" --outputdir=DIR place output files in DIR (default \".\")\n"));
printf(_(" --schedule=FILE use test ordering schedule from FILE\n"));
printf(_(" (can be used multiple times to concatenate)\n"));
+ printf(_(" --skip-tests=LIST comma-separated list of tests to skip\n"));
printf(_(" --temp-instance=DIR create a temporary instance in DIR\n"));
printf(_(" --use-existing use an existing installation\n"));
printf(_(" -V, --version output version information, then exit\n"));
@@ -2116,6 +2139,7 @@ regression_main(int argc, char *argv[],
{"load-extension", required_argument, NULL, 22},
{"config-auth", required_argument, NULL, 24},
{"max-concurrent-tests", required_argument, NULL, 25},
+ {"skip-tests", required_argument, NULL, 26},
{NULL, 0, NULL, 0}
};
@@ -2245,6 +2269,9 @@ regression_main(int argc, char *argv[],
case 25:
max_concurrent_tests = atoi(optarg);
break;
+ case 26:
+ split_to_stringlist(optarg, ",", &skip_tests);
+ break;
default:
/* getopt_long already emitted a complaint */
fprintf(stderr, _("\nTry \"%s -h\" for more information.\n"),
--
2.30.2
* Re: A test for replay of regression tests
@ 2021-06-07 23:44 Anastasia Lubennikova <lubennikovaav@gmail.com>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Anastasia Lubennikova @ 2021-06-07 23:44 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers; Andrew Dunstan <andrew@dunslane.net>
вт, 8 июн. 2021 г. в 02:25, Thomas Munro <thomas.munro@gmail.com>:
> Ok, here's a new version incorporating feedback so far.
>
> 1. Invoke pg_regress directly (no make).
>
> 2. Use PG_TEST_EXTRA="wal_consistency_checking" as a way to opt in to
> the more expensive test.
>
> 3. Use parallel schedule rather than serial. It's faster but also
> the non-determinism might discover more things. This required
> changing the TAP test max_connections setting from 10 to 25.
>
> 4. Remove some extraneous print statements and
> check-if-data-is-replicated-using-SELECT tests that are technically
> not needed (I had copied those from 001_stream_rep.pl).
>
Thank you for working on this test set!
I was especially glad to see the skip-tests option for pg_regress. I think
it will become a very handy tool for hackers.
To try the patch I had to resolve a few merge conflicts, see a rebased
version in attachments.
> auth_extra => [ '--create-role', 'repl_role' ]);
This line and the comment above it look like some copy-paste artifacts. Did
I get it right? If so, I suggest removing them.
Other than that, the patch looks good to me.
--
Best regards,
Lubennikova Anastasia
Attachments:
v3-0001-Test-replay-of-regression-tests.patch (text/x-patch, 6.3K) download
commit 2eeaf5802c060612831b3fdeb33401a07c230f83
Author: anastasia <lubennikovaav@gmail.com>
Date: Tue Jun 8 02:01:35 2021 +0300
v3-0001-Test-replay-of-regression-tests.patch
diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml
index cb401a45b3..7a10c83d8a 100644
--- a/doc/src/sgml/regress.sgml
+++ b/doc/src/sgml/regress.sgml
@@ -289,6 +289,17 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl'
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>wal_consistency_checking</literal></term>
+ <listitem>
+ <para>
+ Uses <literal>wal_consistency_checking=all</literal> while running
+ some of the tests under <filename>src/test/recovery</filename>. Not
+ enabled by default because it is resource intensive.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
Tests for features that are not supported by the current build
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 45d1636128..7b0af9fd78 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -484,7 +484,7 @@ sub init
print $conf "hot_standby = on\n";
# conservative settings to ensure we can run multiple postmasters:
print $conf "shared_buffers = 1MB\n";
- print $conf "max_connections = 10\n";
+ print $conf "max_connections = 25\n";
# limit disk space consumption, too:
print $conf "max_wal_size = 128MB\n";
}
diff --git a/src/test/recovery/t/025_stream_rep_regress.pl b/src/test/recovery/t/025_stream_rep_regress.pl
new file mode 100644
index 0000000000..8b125a1d67
--- /dev/null
+++ b/src/test/recovery/t/025_stream_rep_regress.pl
@@ -0,0 +1,62 @@
+# Run the standard regression tests with streaming replication
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+# Initialize primary node
+my $node_primary = get_new_node('primary');
+# A specific role is created to perform some tests related to replication,
+# and it needs proper authentication configuration.
+$node_primary->init(
+ allows_streaming => 1,
+ auth_extra => [ '--create-role', 'repl_role' ]);
+
+# WAL consistency checking is resource intensive so require opt-in with the
+# PG_TEST_EXTRA environment variable.
+if ($ENV{PG_TEST_EXTRA} &&
+ $ENV{PG_TEST_EXTRA} =~ m/\bwal_consistency_checking\b/) {
+ $node_primary->append_conf('postgresql.conf',
+ 'wal_consistency_checking = all');
+}
+
+$node_primary->start;
+is( $node_primary->psql(
+ 'postgres',
+ qq[SELECT pg_create_physical_replication_slot('standby_1');]),
+ 0,
+ 'physical slot created on primary');
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_primary->backup($backup_name);
+
+# Create streaming standby linking to primary
+my $node_standby_1 = get_new_node('standby_1');
+$node_standby_1->init_from_backup($node_primary, $backup_name,
+ has_streaming => 1);
+$node_standby_1->append_conf('postgresql.conf',
+ "primary_slot_name = standby_1");
+$node_standby_1->start;
+
+# XXX The tablespace tests don't currently work when the standby shares a
+# filesystem with the primary, due to colliding absolute paths. We'll skip
+# that for now.
+
+# Run the regression tests against the primary.
+system_or_bail("../regress/pg_regress",
+ "--port=" . $node_primary->port,
+ "--schedule=../regress/parallel_schedule",
+ "--dlpath=../regress",
+ "--inputdir=../regress",
+ "--skip-tests=tablespace");
+
+# Wait for standby to catch up
+$node_primary->wait_for_catchup($node_standby_1, 'replay',
+ $node_primary->lsn('insert'));
+
+ok(1, "caught up");
+
+$node_standby_1->stop;
+$node_primary->stop;
diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index e04d365258..510afaadbf 100644
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -83,6 +83,7 @@ static int max_connections = 0;
static int max_concurrent_tests = 0;
static char *encoding = NULL;
static _stringlist *schedulelist = NULL;
+static _stringlist *skip_tests = NULL;
static _stringlist *extra_tests = NULL;
static char *temp_instance = NULL;
static _stringlist *temp_configs = NULL;
@@ -203,6 +204,22 @@ split_to_stringlist(const char *s, const char *delim, _stringlist **listhead)
free(sc);
}
+/*
+ * Test if a string is in a list.
+ */
+static bool
+string_in_stringlist(const char *s, _stringlist *list)
+{
+ while (list)
+ {
+ if (strcmp(list->str, s) == 0)
+ return true;
+ list = list->next;
+ }
+
+ return false;
+}
+
/*
* Print a progress banner on stdout.
*/
@@ -1672,7 +1689,11 @@ run_schedule(const char *schedule, test_start_function startfunc,
if (scbuf[0] == '\0' || scbuf[0] == '#')
continue;
if (strncmp(scbuf, "test: ", 6) == 0)
+ {
test = scbuf + 6;
+ if (string_in_stringlist(test, skip_tests))
+ continue;
+ }
else if (strncmp(scbuf, "ignore: ", 8) == 0)
{
c = scbuf + 8;
@@ -1870,6 +1891,7 @@ run_schedule(const char *schedule, test_start_function startfunc,
}
free_stringlist(&ignorelist);
+ free_stringlist(&skip_tests);
fclose(scf);
}
@@ -2076,6 +2098,7 @@ help(void)
printf(_(" --outputdir=DIR place output files in DIR (default \".\")\n"));
printf(_(" --schedule=FILE use test ordering schedule from FILE\n"));
printf(_(" (can be used multiple times to concatenate)\n"));
+ printf(_(" --skip-tests=LIST comma-separated list of tests to skip\n"));
printf(_(" --temp-instance=DIR create a temporary instance in DIR\n"));
printf(_(" --use-existing use an existing installation\n"));
printf(_(" -V, --version output version information, then exit\n"));
@@ -2128,6 +2151,7 @@ regression_main(int argc, char *argv[],
{"config-auth", required_argument, NULL, 24},
{"max-concurrent-tests", required_argument, NULL, 25},
{"make-testtablespace-dir", no_argument, NULL, 26},
+ {"skip-tests", required_argument, NULL, 27},
{NULL, 0, NULL, 0}
};
@@ -2261,6 +2285,9 @@ regression_main(int argc, char *argv[],
case 26:
make_testtablespace_dir = true;
break;
+ case 27:
+ split_to_stringlist(optarg, ",", &skip_tests);
+ break;
default:
/* getopt_long already emitted a complaint */
fprintf(stderr, _("\nTry \"%s -h\" for more information.\n"),
* Re: A test for replay of regression tests
@ 2021-06-10 07:37 Anastasia Lubennikova <lubennikovaav@gmail.com>
parent: Anastasia Lubennikova <lubennikovaav@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Anastasia Lubennikova @ 2021-06-10 07:37 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers; Andrew Dunstan <andrew@dunslane.net>
вт, 8 июн. 2021 г. в 02:44, Anastasia Lubennikova <lubennikovaav@gmail.com>:
>
> вт, 8 июн. 2021 г. в 02:25, Thomas Munro <thomas.munro@gmail.com>:
>
>> Ok, here's a new version incorporating feedback so far.
>>
>> 1. Invoke pg_regress directly (no make).
>>
>> 2. Use PG_TEST_EXTRA="wal_consistency_checking" as a way to opt in to
>> the more expensive test.
>>
>> 3. Use parallel schedule rather than serial. It's faster but also
>> the non-determinism might discover more things. This required
>> changing the TAP test max_connections setting from 10 to 25.
>>
>> 4. Remove some extraneous print statements and
>> check-if-data-is-replicated-using-SELECT tests that are technically
>> not needed (I had copied those from 001_stream_rep.pl).
>>
>
> Thank you for working on this test set!
> I was especially glad to see the skip-tests option for pg_regress. I think
> it will become a very handy tool for hackers.
>
> To try the patch I had to resolve a few merge conflicts, see a rebased
> version in attachments.
>
> > auth_extra => [ '--create-role', 'repl_role' ]);
> This line and the comment above it look like some copy-paste artifacts.
> Did I get it right? If so, I suggest removing them.
> Other than that, the patch looks good to me.
>
For some reason, it failed on cfbot, so I'll switch it back to 'Waiting on
author'.
BTW, do I get it right, that cfbot CI will need some adjustments to print
regression.out for this test?
See one more revision of the patch attached. It contains the following
changes:
- rebase to recent main
- removed 'auth_extra' piece, that I mentioned above.
- added lacking make clean and gitignore changes.
--
Best regards,
Lubennikova Anastasia
Attachments:
v4-0001-Test-replay-of-regression-tests.patch (text/x-patch, 7.0K) download
commit 9630f6367d2b5079976bef0a27bb925d00cb798d
Author: anastasia <lubennikovaav@gmail.com>
Date: Tue Jun 8 02:01:35 2021 +0300
v4-0001-Test-replay-of-regression-tests.patch
diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml
index cb401a45b3..7a10c83d8a 100644
--- a/doc/src/sgml/regress.sgml
+++ b/doc/src/sgml/regress.sgml
@@ -289,6 +289,17 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl'
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>wal_consistency_checking</literal></term>
+ <listitem>
+ <para>
+ Uses <literal>wal_consistency_checking=all</literal> while running
+ some of the tests under <filename>src/test/recovery</filename>. Not
+ enabled by default because it is resource intensive.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
Tests for features that are not supported by the current build
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 45d1636128..7b0af9fd78 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -484,7 +484,7 @@ sub init
print $conf "hot_standby = on\n";
# conservative settings to ensure we can run multiple postmasters:
print $conf "shared_buffers = 1MB\n";
- print $conf "max_connections = 10\n";
+ print $conf "max_connections = 25\n";
# limit disk space consumption, too:
print $conf "max_wal_size = 128MB\n";
}
diff --git a/src/test/recovery/.gitignore b/src/test/recovery/.gitignore
index 871e943d50..03410b70a3 100644
--- a/src/test/recovery/.gitignore
+++ b/src/test/recovery/.gitignore
@@ -1,2 +1,10 @@
# Generated by test suite
/tmp_check/
+/results/
+/expected/
+/sql/
+/log/
+
+# Note: regression.* are only left behind on a failure; that's why they're not ignored
+#/regression.diffs
+#/regression.out
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
index 96442ceb4e..0b0d6094cc 100644
--- a/src/test/recovery/Makefile
+++ b/src/test/recovery/Makefile
@@ -15,6 +15,10 @@ subdir = src/test/recovery
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
+# These are created by pg_regress.
+# So we need rules to clean them.
+pg_regress_clean_files += sql/ expected/
+
check:
$(prove_check)
@@ -22,4 +26,4 @@ installcheck:
$(prove_installcheck)
clean distclean maintainer-clean:
- rm -rf tmp_check
+ rm -rf $(pg_regress_clean_files)
diff --git a/src/test/recovery/t/026_stream_rep_regress.pl b/src/test/recovery/t/026_stream_rep_regress.pl
new file mode 100644
index 0000000000..0043405739
--- /dev/null
+++ b/src/test/recovery/t/026_stream_rep_regress.pl
@@ -0,0 +1,59 @@
+# Run the standard regression tests with streaming replication
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+# Initialize primary node
+my $node_primary = get_new_node('primary');
+$node_primary->init(
+ allows_streaming => 1);
+
+# WAL consistency checking is resource intensive so require opt-in with the
+# PG_TEST_EXTRA environment variable.
+if ($ENV{PG_TEST_EXTRA} &&
+ $ENV{PG_TEST_EXTRA} =~ m/\bwal_consistency_checking\b/) {
+ $node_primary->append_conf('postgresql.conf',
+ 'wal_consistency_checking = all');
+}
+
+$node_primary->start;
+is( $node_primary->psql(
+ 'postgres',
+ qq[SELECT pg_create_physical_replication_slot('standby_1');]),
+ 0,
+ 'physical slot created on primary');
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_primary->backup($backup_name);
+
+# Create streaming standby linking to primary
+my $node_standby_1 = get_new_node('standby_1');
+$node_standby_1->init_from_backup($node_primary, $backup_name,
+ has_streaming => 1);
+$node_standby_1->append_conf('postgresql.conf',
+ "primary_slot_name = standby_1");
+$node_standby_1->start;
+
+# XXX The tablespace tests don't currently work when the standby shares a
+# filesystem with the primary, due to colliding absolute paths. We'll skip
+# that for now.
+
+# Run the regression tests against the primary.
+system_or_bail("../regress/pg_regress",
+ "--port=" . $node_primary->port,
+ "--schedule=../regress/parallel_schedule",
+ "--dlpath=../regress",
+ "--inputdir=../regress",
+ "--skip-tests=tablespace");
+
+# Wait for standby to catch up
+$node_primary->wait_for_catchup($node_standby_1, 'replay',
+ $node_primary->lsn('insert'));
+
+ok(1, "caught up");
+
+$node_standby_1->stop;
+$node_primary->stop;
diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index e04d365258..510afaadbf 100644
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -83,6 +83,7 @@ static int max_connections = 0;
static int max_concurrent_tests = 0;
static char *encoding = NULL;
static _stringlist *schedulelist = NULL;
+static _stringlist *skip_tests = NULL;
static _stringlist *extra_tests = NULL;
static char *temp_instance = NULL;
static _stringlist *temp_configs = NULL;
@@ -203,6 +204,22 @@ split_to_stringlist(const char *s, const char *delim, _stringlist **listhead)
free(sc);
}
+/*
+ * Test if a string is in a list.
+ */
+static bool
+string_in_stringlist(const char *s, _stringlist *list)
+{
+ while (list)
+ {
+ if (strcmp(list->str, s) == 0)
+ return true;
+ list = list->next;
+ }
+
+ return false;
+}
+
/*
* Print a progress banner on stdout.
*/
@@ -1672,7 +1689,11 @@ run_schedule(const char *schedule, test_start_function startfunc,
if (scbuf[0] == '\0' || scbuf[0] == '#')
continue;
if (strncmp(scbuf, "test: ", 6) == 0)
+ {
test = scbuf + 6;
+ if (string_in_stringlist(test, skip_tests))
+ continue;
+ }
else if (strncmp(scbuf, "ignore: ", 8) == 0)
{
c = scbuf + 8;
@@ -1870,6 +1891,7 @@ run_schedule(const char *schedule, test_start_function startfunc,
}
free_stringlist(&ignorelist);
+ free_stringlist(&skip_tests);
fclose(scf);
}
@@ -2076,6 +2098,7 @@ help(void)
printf(_(" --outputdir=DIR place output files in DIR (default \".\")\n"));
printf(_(" --schedule=FILE use test ordering schedule from FILE\n"));
printf(_(" (can be used multiple times to concatenate)\n"));
+ printf(_(" --skip-tests=LIST comma-separated list of tests to skip\n"));
printf(_(" --temp-instance=DIR create a temporary instance in DIR\n"));
printf(_(" --use-existing use an existing installation\n"));
printf(_(" -V, --version output version information, then exit\n"));
@@ -2128,6 +2151,7 @@ regression_main(int argc, char *argv[],
{"config-auth", required_argument, NULL, 24},
{"max-concurrent-tests", required_argument, NULL, 25},
{"make-testtablespace-dir", no_argument, NULL, 26},
+ {"skip-tests", required_argument, NULL, 27},
{NULL, 0, NULL, 0}
};
@@ -2261,6 +2285,9 @@ regression_main(int argc, char *argv[],
case 26:
make_testtablespace_dir = true;
break;
+ case 27:
+ split_to_stringlist(optarg, ",", &skip_tests);
+ break;
default:
/* getopt_long already emitted a complaint */
fprintf(stderr, _("\nTry \"%s -h\" for more information.\n"),
* Re: A test for replay of regression tests
@ 2021-06-10 07:47 Thomas Munro <thomas.munro@gmail.com>
parent: Anastasia Lubennikova <lubennikovaav@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2021-06-10 07:47 UTC (permalink / raw)
To: Anastasia Lubennikova <lubennikovaav@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers; Andrew Dunstan <andrew@dunslane.net>
On Thu, Jun 10, 2021 at 7:37 PM Anastasia Lubennikova
<lubennikovaav@gmail.com> wrote:
> вт, 8 июн. 2021 г. в 02:44, Anastasia Lubennikova <lubennikovaav@gmail.com>:
>> Thank you for working on this test set!
>> I was especially glad to see the skip-tests option for pg_regress. I think it will become a very handy tool for hackers.
>>
>> To try the patch I had to resolve a few merge conflicts, see a rebased version in attachments.
>>
>> > auth_extra => [ '--create-role', 'repl_role' ]);
>> This line and the comment above it look like some copy-paste artifacts. Did I get it right? If so, I suggest removing them.
>> Other than that, the patch looks good to me.
>
> For some reason, it failed on cfbot, so I'll switch it back to 'Waiting on author'.
> BTW, do I get it right, that cfbot CI will need some adjustments to print regression.out for this test?
>
> See one more revision of the patch attached. It contains the following changes:
> - rebase to recent main
> - removed 'auth_extra' piece, that I mentioned above.
> - added lacking make clean and gitignore changes.
Thanks! Yeah, there does seem to be a mysterious CI failure there,
not reproducible locally for me. You're right that it's not dumping
enough information to diagnose the problem... I will look into it
tomorrow.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-10-06 06:10 Thomas Munro <thomas.munro@gmail.com>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2021-10-06 06:10 UTC (permalink / raw)
To: Anastasia Lubennikova <lubennikovaav@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers; Andrew Dunstan <andrew@dunslane.net>
On Thu, Jun 10, 2021 at 7:47 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Thu, Jun 10, 2021 at 7:37 PM Anastasia Lubennikova
> <lubennikovaav@gmail.com> wrote:
> > For some reason, it failed on cfbot, so I'll switch it back to 'Waiting on
Sorry for the delay. I got stuck in a CI loop trying to make a new
improved version work on Windows, so far without success. If anyone
can tell me what's wrong on that OS I'd be very grateful to hear it.
I don't know if it's something I haven't understood about reparse
points, or something to do with "restricted tokens" and accounts
privileges. The symptoms on Cirrus are:
DROP TABLESPACE regress_tblspacewith;
+WARNING: could not open directory
"pg_tblspc/16386/PG_15_202109101": No such file or directory
+WARNING: could not stat file "pg_tblspc/16386": No such file or directory
The short explanation of this new version is that the tablespace tests
now work on a pair of local nodes because you can do this sort of
thing:
postgres=# create tablespace xxx location 'pg_user_files/xxx';
ERROR: directory "pg_user_files/xxx" does not exist
postgres=# create tablespace xxx new location 'pg_user_files/xxx';
CREATE TABLESPACE
Patches:
0001: Allow restricted relative tablespace paths.
Rationale: I really want to be able to run the tablespace test with a
local replica, instead of just skipping it (including but not only
from this new TAP test). After re-reading a bunch of threads about
how to achieve that that never went anywhere and considering various
ideas already presented, I wondered if we could agree on allowing
relative paths under one specific directory "pg_user_files" (a
directory that PostgreSQL itself will completely ignore). Better
names welcome.
I wonder if for Windows we'd want to switch to real symlinks, since,
as far as I know from some light reading, reparse points are converted
to absolute paths on creation, so the pgdata directory would be less
portable than it would be on POSIX systems.
0002: CREATE TABLESPACE ... NEW LOCATION.
The new syntax "NEW" says that it's OK if the directory doesn't exist
yet, we'll just create it.
Rationale: With relative paths, it's tricky for pg_regress to find
the data directory of the primary server + any streaming replicas that
may be downstream from it (and possibly remote) to create the
directory, but the server can do it easily. Better syntax welcome.
(I originally wanted to use WITH (<something>) but that syntax is
tangled up with persistent relopts.)
0003: Use relative paths for tablespace regression test.
Remove the pg_regress logic for creating the directory, and switch to
relative paths using the above.
0004: Test replay of regression tests.
Same as before, this adds a replicated run of the regression tests in
src/test/recovery/t/027_stream_regress.pl, with an optional expensive
mode that you can enable with
PG_TEST_EXTRA="wal_consistency_checking".
I removed the useless --create-role as pointed out by Anastasia.
I added a step to compare the contents of the primary and replica
servers with pg_dump, as suggested by Tsunakawa-san.
I think the way I pass in the psql source directory to --bindir is not
good, but I've reached my daily limit of Perl; how should I be
specifying the tmp_install bin directory here? This is so pg_regress
can find psql.
system_or_bail("../regress/pg_regress",
"--bindir=../../bin/psql",
"--port=" . $node_primary->port,
"--schedule=../regress/parallel_schedule",
"--dlpath=../regress",
"--inputdir=../regress");
Attachments: download all (4 files, .tar.gz)
v5-0001-Allow-restricted-relative-tablespace-paths.patch (text/x-patch, 4.3K) download
From 0a8c431683976eeb9ca52a25ac19b7dd7dfe8107 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 25 May 2021 20:42:17 +1200
Subject: [PATCH v5 1/6] Allow restricted relative tablespace paths.
Traditionally, tablespace paths provided to LOCATION had to be absolute,
because relative paths risked colliding with PostgreSQL-managed files
and generally blurring the boundary between system-managed and
user-managed data.
This commit adds a very small exception: it allows relative paths to be
used, but only under a new directory "pg_user_files". This is intended
to allow automated testing of replication in later patches.
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
src/backend/commands/tablespace.c | 23 ++++++++++++++++++++---
src/bin/initdb/initdb.c | 1 +
src/common/string.c | 9 +++++++++
src/include/common/string.h | 1 +
4 files changed, 31 insertions(+), 3 deletions(-)
diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c
index 4b96eec9df..adfbde6b29 100644
--- a/src/backend/commands/tablespace.c
+++ b/src/backend/commands/tablespace.c
@@ -70,6 +70,7 @@
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
#include "common/file_perm.h"
+#include "common/string.h"
#include "miscadmin.h"
#include "postmaster/bgwriter.h"
#include "storage/fd.h"
@@ -267,14 +268,16 @@ CreateTableSpace(CreateTableSpaceStmt *stmt)
errmsg("tablespace location cannot contain single quotes")));
/*
- * Allowing relative paths seems risky
+ * Allowing relative paths seems risky in general, but we'll allow it under
+ * pg_user_files.
*
* this also helps us ensure that location is not empty or whitespace
*/
- if (!is_absolute_path(location))
+ if (!is_absolute_path(location) &&
+ !pg_str_startswith(location, "pg_user_files/"))
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
- errmsg("tablespace location must be an absolute path")));
+ errmsg("tablespace location must be an absolute path or a relative path under pg_user_files/")));
/*
* Check that location isn't too long. Remember that we're going to append
@@ -587,6 +590,7 @@ DropTableSpace(DropTableSpaceStmt *stmt)
static void
create_tablespace_directories(const char *location, const Oid tablespaceoid)
{
+ char buffer[MAXPGPATH];
char *linkloc;
char *location_with_version_dir;
struct stat st;
@@ -651,6 +655,19 @@ create_tablespace_directories(const char *location, const Oid tablespaceoid)
if (InRecovery)
remove_tablespace_symlink(linkloc);
+#ifndef WIN32
+ /*
+ * If location is relative to pgdata, prefix it with ".." so that we have a
+ * path relative to the symlink. (Not needed for Windows, because our
+ * symlink() wrapper will interpret it as relative to pgdata instead).
+ */
+ if (!is_absolute_path(location))
+ {
+ snprintf(buffer, sizeof(buffer), "../%s", location);
+ location = buffer;
+ }
+#endif
+
/*
* Create the symlink under PGDATA
*/
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 1ed4808d53..896a385e7e 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -224,6 +224,7 @@ static const char *const subdirs[] = {
"pg_tblspc",
"pg_stat",
"pg_stat_tmp",
+ "pg_user_files",
"pg_xact",
"pg_logical",
"pg_logical/snapshots",
diff --git a/src/common/string.c b/src/common/string.c
index 3aa378c051..7e7e34f1f5 100644
--- a/src/common/string.c
+++ b/src/common/string.c
@@ -24,6 +24,15 @@
#include "common/string.h"
+/*
+ * Returns whether the string `str' has the prefix `start'.
+ */
+bool
+pg_str_startswith(const char *str, const char *start)
+{
+ return strncmp(str, start, strlen(start)) == 0;
+}
+
/*
* Returns whether the string `str' has the postfix `end'.
*/
diff --git a/src/include/common/string.h b/src/include/common/string.h
index 686c158efe..f1140b81fa 100644
--- a/src/include/common/string.h
+++ b/src/include/common/string.h
@@ -13,6 +13,7 @@
struct StringInfoData; /* avoid including stringinfo.h here */
/* functions in src/common/string.c */
+extern bool pg_str_startswith(const char *str, const char *start);
extern bool pg_str_endswith(const char *str, const char *end);
extern int strtoint(const char *pg_restrict str, char **pg_restrict endptr,
int base);
--
2.30.2
v5-0002-CREATE-TABLESPACE-.-NEW-LOCATION.patch (text/x-patch, 7.6K) download
From 51dfdf276e1c8bdef1ea6710b6e7887650d2567a Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Wed, 4 Aug 2021 17:20:48 +1200
Subject: [PATCH v5 2/6] CREATE TABLESPACE ... NEW LOCATION.
Allow tablespace directories to be created on demand. This allows the
regression test suite to be run without knowing the path of, or having
access to, the server's data directory.
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
doc/src/sgml/ref/create_tablespace.sgml | 14 ++++++----
src/backend/commands/tablespace.c | 36 ++++++++++++++++++-------
src/backend/parser/gram.y | 13 ++++++---
src/include/commands/tablespace.h | 1 +
src/include/nodes/parsenodes.h | 1 +
5 files changed, 47 insertions(+), 18 deletions(-)
diff --git a/doc/src/sgml/ref/create_tablespace.sgml b/doc/src/sgml/ref/create_tablespace.sgml
index 84fa7ee5e2..497b2568fb 100644
--- a/doc/src/sgml/ref/create_tablespace.sgml
+++ b/doc/src/sgml/ref/create_tablespace.sgml
@@ -23,7 +23,7 @@ PostgreSQL documentation
<synopsis>
CREATE TABLESPACE <replaceable class="parameter">tablespace_name</replaceable>
[ OWNER { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER } ]
- LOCATION '<replaceable class="parameter">directory</replaceable>'
+ [ NEW ] LOCATION '<replaceable class="parameter">directory</replaceable>'
[ WITH ( <replaceable class="parameter">tablespace_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] ) ]
</synopsis>
</refsynopsisdiv>
@@ -92,10 +92,14 @@ CREATE TABLESPACE <replaceable class="parameter">tablespace_name</replaceable>
<listitem>
<para>
The directory that will be used for the tablespace. The directory
- must exist (<command>CREATE TABLESPACE</command> will not create it),
- should be empty, and must be owned by the
- <productname>PostgreSQL</productname> system user. The directory must be
- specified by an absolute path name.
+ must exist, be empty, and be owned by the
+ <productname>PostgreSQL</productname> system user unless
+ <command>NEW</command> is specified. If
+ <command>NEW</command> is specified it will be created, and the parent
+ directory must exist.
+ The directory must be specified by an absolute path name, or a path
+ name that is relative to the data directory and begins with
+ <literal>pg_user_files/</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c
index adfbde6b29..b2a1a12fbc 100644
--- a/src/backend/commands/tablespace.c
+++ b/src/backend/commands/tablespace.c
@@ -91,7 +91,8 @@ char *temp_tablespaces = NULL;
static void create_tablespace_directories(const char *location,
- const Oid tablespaceoid);
+ const Oid tablespaceoid,
+ bool missing_ok);
static bool destroy_tablespace_directories(Oid tablespaceoid, bool redo);
@@ -369,13 +370,14 @@ CreateTableSpace(CreateTableSpaceStmt *stmt)
/* Post creation hook for new tablespace */
InvokeObjectPostCreateHook(TableSpaceRelationId, tablespaceoid, 0);
- create_tablespace_directories(location, tablespaceoid);
+ create_tablespace_directories(location, tablespaceoid, stmt->missing_ok);
/* Record the filesystem change in XLOG */
{
xl_tblspc_create_rec xlrec;
xlrec.ts_id = tablespaceoid;
+ xlrec.ts_missing_ok = stmt->missing_ok;
XLogBeginInsert();
XLogRegisterData((char *) &xlrec,
@@ -588,7 +590,8 @@ DropTableSpace(DropTableSpaceStmt *stmt)
* to the specified directory
*/
static void
-create_tablespace_directories(const char *location, const Oid tablespaceoid)
+create_tablespace_directories(const char *location, const Oid tablespaceoid,
+ bool missing_ok)
{
char buffer[MAXPGPATH];
char *linkloc;
@@ -601,16 +604,28 @@ create_tablespace_directories(const char *location, const Oid tablespaceoid)
/*
* Attempt to coerce target directory to safe permissions. If this fails,
- * it doesn't exist or has the wrong owner.
+ * it doesn't exist or has the wrong owner. If NEW LOCATION, we'll create
+ * the directory.
*/
if (chmod(location, pg_dir_create_mode) != 0)
{
if (errno == ENOENT)
- ereport(ERROR,
- (errcode(ERRCODE_UNDEFINED_FILE),
- errmsg("directory \"%s\" does not exist", location),
- InRecovery ? errhint("Create this directory for the tablespace before "
- "restarting the server.") : 0));
+ {
+ if (missing_ok)
+ {
+ if (mkdir(location, pg_dir_create_mode) != 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_FILE),
+ errmsg("could not create directory \"%s\"",
+ location)));
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_FILE),
+ errmsg("directory \"%s\" does not exist", location),
+ InRecovery ? errhint("Create this directory for the tablespace before "
+ "restarting the server.") : 0));
+ }
else
ereport(ERROR,
(errcode_for_file_access(),
@@ -1537,8 +1552,9 @@ tblspc_redo(XLogReaderState *record)
{
xl_tblspc_create_rec *xlrec = (xl_tblspc_create_rec *) XLogRecGetData(record);
char *location = xlrec->ts_path;
+ bool missing_ok = xlrec->ts_missing_ok;
- create_tablespace_directories(location, xlrec->ts_id);
+ create_tablespace_directories(location, xlrec->ts_id, missing_ok);
}
else if (info == XLOG_TBLSPC_DROP)
{
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 08f1bf1031..3a30d299f8 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -571,6 +571,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <boolean> constraints_set_mode
%type <str> OptTableSpace OptConsTableSpace
%type <rolespec> OptTableSpaceOwner
+%type <boolean> OptTableSpaceNew
%type <ival> opt_check_option
%type <str> opt_provider security_label
@@ -4552,13 +4553,14 @@ opt_procedural:
*
*****************************************************************************/
-CreateTableSpaceStmt: CREATE TABLESPACE name OptTableSpaceOwner LOCATION Sconst opt_reloptions
+CreateTableSpaceStmt: CREATE TABLESPACE name OptTableSpaceOwner OptTableSpaceNew LOCATION Sconst opt_reloptions
{
CreateTableSpaceStmt *n = makeNode(CreateTableSpaceStmt);
n->tablespacename = $3;
n->owner = $4;
- n->location = $6;
- n->options = $7;
+ n->missing_ok = $5;
+ n->location = $7;
+ n->options = $8;
$$ = (Node *) n;
}
;
@@ -4567,6 +4569,11 @@ OptTableSpaceOwner: OWNER RoleSpec { $$ = $2; }
| /*EMPTY */ { $$ = NULL; }
;
+OptTableSpaceNew:
+ NEW { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
/*****************************************************************************
*
* QUERY :
diff --git a/src/include/commands/tablespace.h b/src/include/commands/tablespace.h
index 49cfc8479a..7a53c886c0 100644
--- a/src/include/commands/tablespace.h
+++ b/src/include/commands/tablespace.h
@@ -26,6 +26,7 @@
typedef struct xl_tblspc_create_rec
{
Oid ts_id;
+ bool ts_missing_ok;
char ts_path[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string */
} xl_tblspc_create_rec;
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3138877553..6ab32e0786 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -2321,6 +2321,7 @@ typedef struct CreateTableSpaceStmt
RoleSpec *owner;
char *location;
List *options;
+ bool missing_ok;
} CreateTableSpaceStmt;
typedef struct DropTableSpaceStmt
--
2.30.2
v5-0003-Use-relative-paths-for-tablespace-regression-test.patch (text/x-patch, 9.1K) download
From 5f54546db0e06add415728dd86a5b36fa23c28f0 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Wed, 4 Aug 2021 22:17:54 +1200
Subject: [PATCH v5 3/6] Use relative paths for tablespace regression test.
Remove the machinery from pg_regress that manages the testtablespace
directory. Instead, use a relative path.
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
src/test/regress/GNUmakefile | 3 +-
.../tablespace.out} | 6 +--
src/test/regress/pg_regress.c | 40 -------------------
.../tablespace.source => sql/tablespace.sql} | 6 +--
src/tools/msvc/vcregress.pl | 2 -
5 files changed, 7 insertions(+), 50 deletions(-)
rename src/test/regress/{output/tablespace.source => expected/tablespace.out} (99%)
rename src/test/regress/{input/tablespace.source => sql/tablespace.sql} (98%)
diff --git a/src/test/regress/GNUmakefile b/src/test/regress/GNUmakefile
index fe6e0c98aa..d014b4a35d 100644
--- a/src/test/regress/GNUmakefile
+++ b/src/test/regress/GNUmakefile
@@ -119,7 +119,7 @@ submake-contrib-spi: | submake-libpgport submake-generated-headers
## Run tests
##
-REGRESS_OPTS = --dlpath=. --max-concurrent-tests=20 --make-testtablespace-dir \
+REGRESS_OPTS = --dlpath=. --max-concurrent-tests=20 \
$(EXTRA_REGRESS_OPTS)
check: all
@@ -163,5 +163,4 @@ clean distclean maintainer-clean: clean-lib
rm -f pg_regress_main.o pg_regress.o pg_regress$(X)
# things created by various check targets
rm -f $(output_files) $(input_files)
- rm -rf testtablespace
rm -rf $(pg_regress_clean_files)
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/expected/tablespace.out
similarity index 99%
rename from src/test/regress/output/tablespace.source
rename to src/test/regress/expected/tablespace.out
index e7629d470e..c815901219 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/expected/tablespace.out
@@ -1,7 +1,7 @@
-- create a tablespace using WITH clause
-CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (some_nonexistent_parameter = true); -- fail
+CREATE TABLESPACE regress_tblspacewith LOCATION 'pg_user_files/testtablespace' WITH (some_nonexistent_parameter = true); -- fail
ERROR: unrecognized parameter "some_nonexistent_parameter"
-CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (random_page_cost = 3.0); -- ok
+CREATE TABLESPACE regress_tblspacewith NEW LOCATION 'pg_user_files/testtablespace' WITH (random_page_cost = 3.0); -- ok
-- check to see the parameter was used
SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
spcoptions
@@ -12,7 +12,7 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
-- drop the tablespace so we can re-use the location
DROP TABLESPACE regress_tblspacewith;
-- create a tablespace we can use
-CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
+CREATE TABLESPACE regress_tblspace LOCATION 'pg_user_files/testtablespace';
-- try setting and resetting some properties for the new tablespace
ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true); -- fail
diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index 05296f7ee1..57a6231767 100644
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -475,7 +475,6 @@ replace_string(StringInfo string, const char *replace, const char *replacement)
static void
convert_sourcefiles_in(const char *source_subdir, const char *dest_dir, const char *dest_subdir, const char *suffix)
{
- char testtablespace[MAXPGPATH];
char indir[MAXPGPATH];
char outdir_sub[MAXPGPATH];
char **name;
@@ -504,9 +503,6 @@ convert_sourcefiles_in(const char *source_subdir, const char *dest_dir, const ch
if (!directory_exists(outdir_sub))
make_directory(outdir_sub);
- /* We might need to replace @testtablespace@ */
- snprintf(testtablespace, MAXPGPATH, "%s/testtablespace", outputdir);
-
/* finally loop on each file and do the replacement */
for (name = names; *name; name++)
{
@@ -552,7 +548,6 @@ convert_sourcefiles_in(const char *source_subdir, const char *dest_dir, const ch
{
replace_string(&line, "@abs_srcdir@", inputdir);
replace_string(&line, "@abs_builddir@", outputdir);
- replace_string(&line, "@testtablespace@", testtablespace);
replace_string(&line, "@libdir@", dlpath);
replace_string(&line, "@DLSUFFIX@", DLSUFFIX);
fputs(line.data, outfile);
@@ -585,32 +580,6 @@ convert_sourcefiles(void)
convert_sourcefiles_in("output", outputdir, "expected", "out");
}
-/*
- * Clean out the test tablespace dir, or create it if it doesn't exist.
- *
- * On Windows, doing this cleanup here makes it possible to run the
- * regression tests under a Windows administrative user account with the
- * restricted token obtained when starting pg_regress.
- */
-static void
-prepare_testtablespace_dir(void)
-{
- char testtablespace[MAXPGPATH];
-
- snprintf(testtablespace, MAXPGPATH, "%s/testtablespace", outputdir);
-
- if (directory_exists(testtablespace))
- {
- if (!rmtree(testtablespace, true))
- {
- fprintf(stderr, _("\n%s: could not remove test tablespace \"%s\"\n"),
- progname, testtablespace);
- exit(2);
- }
- }
- make_directory(testtablespace);
-}
-
/*
* Scan resultmap file to find which platform-specific expected files to use.
*
@@ -2092,7 +2061,6 @@ help(void)
printf(_(" --launcher=CMD use CMD as launcher of psql\n"));
printf(_(" --load-extension=EXT load the named extension before running the\n"));
printf(_(" tests; can appear multiple times\n"));
- printf(_(" --make-testtablespace-dir create testtablespace directory\n"));
printf(_(" --max-connections=N maximum number of concurrent connections\n"));
printf(_(" (default is 0, meaning unlimited)\n"));
printf(_(" --max-concurrent-tests=N maximum number of concurrent tests in schedule\n"));
@@ -2151,12 +2119,10 @@ regression_main(int argc, char *argv[],
{"load-extension", required_argument, NULL, 22},
{"config-auth", required_argument, NULL, 24},
{"max-concurrent-tests", required_argument, NULL, 25},
- {"make-testtablespace-dir", no_argument, NULL, 26},
{NULL, 0, NULL, 0}
};
bool use_unix_sockets;
- bool make_testtablespace_dir = false;
_stringlist *sl;
int c;
int i;
@@ -2282,9 +2248,6 @@ regression_main(int argc, char *argv[],
case 25:
max_concurrent_tests = atoi(optarg);
break;
- case 26:
- make_testtablespace_dir = true;
- break;
default:
/* getopt_long already emitted a complaint */
fprintf(stderr, _("\nTry \"%s -h\" for more information.\n"),
@@ -2337,9 +2300,6 @@ regression_main(int argc, char *argv[],
unlimit_core_size();
#endif
- if (make_testtablespace_dir)
- prepare_testtablespace_dir();
-
if (temp_instance)
{
FILE *pg_conf;
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/sql/tablespace.sql
similarity index 98%
rename from src/test/regress/input/tablespace.source
rename to src/test/regress/sql/tablespace.sql
index cb9774ecc8..a4a4067c42 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/sql/tablespace.sql
@@ -1,6 +1,6 @@
-- create a tablespace using WITH clause
-CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (some_nonexistent_parameter = true); -- fail
-CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (random_page_cost = 3.0); -- ok
+CREATE TABLESPACE regress_tblspacewith LOCATION 'pg_user_files/testtablespace' WITH (some_nonexistent_parameter = true); -- fail
+CREATE TABLESPACE regress_tblspacewith NEW LOCATION 'pg_user_files/testtablespace' WITH (random_page_cost = 3.0); -- ok
-- check to see the parameter was used
SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
@@ -9,7 +9,7 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
DROP TABLESPACE regress_tblspacewith;
-- create a tablespace we can use
-CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
+CREATE TABLESPACE regress_tblspace LOCATION 'pg_user_files/testtablespace';
-- try setting and resetting some properties for the new tablespace
ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index 35e8f67f01..1852c34109 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -118,7 +118,6 @@ sub installcheck_internal
"--bindir=../../../$Config/psql",
"--schedule=${schedule}_schedule",
"--max-concurrent-tests=20",
- "--make-testtablespace-dir",
"--encoding=SQL_ASCII",
"--no-locale");
push(@args, $maxconn) if $maxconn;
@@ -153,7 +152,6 @@ sub check
"--bindir=",
"--schedule=${schedule}_schedule",
"--max-concurrent-tests=20",
- "--make-testtablespace-dir",
"--encoding=SQL_ASCII",
"--no-locale",
"--temp-instance=./tmp_check");
--
2.30.2
v5-0004-Test-replay-of-regression-tests.patch (text/x-patch, 5.0K) download
From e5f50c2bac431963ec6e10b85dbf25c0b3b5e3ca Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 5 Oct 2021 21:01:02 +1300
Subject: [PATCH v5 4/6] Test replay of regression tests.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Add a new TAP test under src/test/recovery to run the standard
regression tests while a streaming replica replays the WAL. This
provides a basic workout for WAL decoding and redo code, and compares
the replicated result.
Optionally, enable (expensive) wal_consistency_checking if listed in
the env variable PG_TEST_EXTRA.
Reviewed-by: 綱川 貴之 (Takayuki Tsunakawa) <tsunakawa.takay@fujitsu.com>
Reviewed-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Anastasia Lubennikova <lubennikovaav@gmail.com>
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
doc/src/sgml/regress.sgml | 11 ++++
src/test/perl/PostgresNode.pm | 2 +-
src/test/recovery/t/027_stream_regress.pl | 68 +++++++++++++++++++++++
3 files changed, 80 insertions(+), 1 deletion(-)
create mode 100644 src/test/recovery/t/027_stream_regress.pl
diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml
index 724ef566e7..64fc81776d 100644
--- a/doc/src/sgml/regress.sgml
+++ b/doc/src/sgml/regress.sgml
@@ -289,6 +289,17 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl'
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>wal_consistency_checking</literal></term>
+ <listitem>
+ <para>
+ Uses <literal>wal_consistency_checking=all</literal> while running
+ some of the tests under <filename>src/test/recovery</filename>. Not
+ enabled by default because it is resource intensive.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
Tests for features that are not supported by the current build
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index c59da758c7..48997fbcc9 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -460,7 +460,7 @@ sub init
print $conf "hot_standby = on\n";
# conservative settings to ensure we can run multiple postmasters:
print $conf "shared_buffers = 1MB\n";
- print $conf "max_connections = 10\n";
+ print $conf "max_connections = 25\n";
# limit disk space consumption, too:
print $conf "max_wal_size = 128MB\n";
}
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
new file mode 100644
index 0000000000..403d7d5366
--- /dev/null
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -0,0 +1,68 @@
+# Run the standard regression tests with streaming replication
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 4;
+
+# Initialize primary node
+my $node_primary = PostgresNode->new('primary');
+$node_primary->init(allows_streaming => 1);
+
+# WAL consistency checking is resource intensive so require opt-in with the
+# PG_TEST_EXTRA environment variable.
+if ($ENV{PG_TEST_EXTRA} &&
+ $ENV{PG_TEST_EXTRA} =~ m/\bwal_consistency_checking\b/) {
+ $node_primary->append_conf('postgresql.conf',
+ 'wal_consistency_checking = all');
+}
+
+$node_primary->start;
+is( $node_primary->psql(
+ 'postgres',
+ qq[SELECT pg_create_physical_replication_slot('standby_1');]),
+ 0,
+ 'physical slot created on primary');
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_primary->backup($backup_name);
+
+# Create streaming standby linking to primary
+my $node_standby_1 = PostgresNode->new('standby_1');
+$node_standby_1->init_from_backup($node_primary, $backup_name,
+ has_streaming => 1);
+$node_standby_1->append_conf('postgresql.conf',
+ "primary_slot_name = standby_1");
+$node_standby_1->start;
+
+# Run the regression tests against the primary.
+system_or_bail("../regress/pg_regress",
+ "--bindir=../../bin/psql",
+ "--port=" . $node_primary->port,
+ "--schedule=../regress/parallel_schedule",
+ "--dlpath=../regress",
+ "--inputdir=../regress");
+
+# Clobber all sequences with their next value, so that we don't have
+# differences between nodes due to caching.
+$node_primary->psql('regression',
+ "select setval(seqrelid, nextval(seqrelid)) from pg_sequence");
+
+# Wait for standby to catch up
+$node_primary->wait_for_catchup($node_standby_1, 'replay',
+ $node_primary->lsn('insert'));
+
+# Perform a logical dump of primary and standby, and check that they match
+command_ok(
+ [ "pg_dump", '-f', 'primary.dump', '--no-sync', '-p', $node_primary->port, 'regression' ],
+ "dump primary server");
+command_ok(
+ [ "pg_dump", '-f', 'standby.dump', '--no-sync', '-p', $node_standby_1->port, 'regression' ],
+ "dump standby server");
+command_ok(
+ [ "diff", 'primary.dump', 'standby.dump' ],
+ "compare primary and standby dumps");
+
+$node_standby_1->stop;
+$node_primary->stop;
--
2.30.2
* Re: A test for replay of regression tests
@ 2021-11-23 09:07 Thomas Munro <thomas.munro@gmail.com>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2021-11-23 09:07 UTC (permalink / raw)
To: Anastasia Lubennikova <lubennikovaav@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers; Andrew Dunstan <andrew@dunslane.net>
On Wed, Oct 6, 2021 at 7:10 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> I wonder if for Windows we'd want to switch to real symlinks, since,
> as far as I know from some light reading, reparse points are converted
> to absolute paths on creation, so the pgdata directory would be less
> portable than it would be on POSIX systems.
I looked into that a bit, and it seems that the newer "real" symlinks
can't be created without admin privileges, unlike junction points, so
that wouldn't help. I still need to figure out what this
junction-based patch set is doing wrong on Windows though trial-by-CI.
In the meantime, here is a rebase over recent changes to the Perl
testing APIs.
Attachments: download all (4 files, .tar.gz)
v6-0001-Allow-restricted-relative-tablespace-paths.patch (text/x-patch, 4.3K) download
From 0a76e6570b8f6eeac146a726a82c7e0e2315cdff Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 25 May 2021 20:42:17 +1200
Subject: [PATCH v6 1/4] Allow restricted relative tablespace paths.
Traditionally, tablespace paths provided to LOCATION had to be absolute,
because relative paths risked colliding with PostgreSQL-managed files
and generally blurring the boundary between system-managed and
user-managed data.
This commit adds a very small exception: it allows relative paths to be
used, but only under a new directory "pg_user_files". This is intended
to allow automated testing of replication in later patches.
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
src/backend/commands/tablespace.c | 23 ++++++++++++++++++++---
src/bin/initdb/initdb.c | 1 +
src/common/string.c | 9 +++++++++
src/include/common/string.h | 1 +
4 files changed, 31 insertions(+), 3 deletions(-)
diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c
index 4b96eec9df..adfbde6b29 100644
--- a/src/backend/commands/tablespace.c
+++ b/src/backend/commands/tablespace.c
@@ -70,6 +70,7 @@
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
#include "common/file_perm.h"
+#include "common/string.h"
#include "miscadmin.h"
#include "postmaster/bgwriter.h"
#include "storage/fd.h"
@@ -267,14 +268,16 @@ CreateTableSpace(CreateTableSpaceStmt *stmt)
errmsg("tablespace location cannot contain single quotes")));
/*
- * Allowing relative paths seems risky
+ * Allowing relative paths seems risky in general, but we'll allow it under
+ * pg_user_files.
*
* this also helps us ensure that location is not empty or whitespace
*/
- if (!is_absolute_path(location))
+ if (!is_absolute_path(location) &&
+ !pg_str_startswith(location, "pg_user_files/"))
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
- errmsg("tablespace location must be an absolute path")));
+ errmsg("tablespace location must be an absolute path or a relative path under pg_user_files/")));
/*
* Check that location isn't too long. Remember that we're going to append
@@ -587,6 +590,7 @@ DropTableSpace(DropTableSpaceStmt *stmt)
static void
create_tablespace_directories(const char *location, const Oid tablespaceoid)
{
+ char buffer[MAXPGPATH];
char *linkloc;
char *location_with_version_dir;
struct stat st;
@@ -651,6 +655,19 @@ create_tablespace_directories(const char *location, const Oid tablespaceoid)
if (InRecovery)
remove_tablespace_symlink(linkloc);
+#ifndef WIN32
+ /*
+ * If location is relative to pgdata, prefix it with ".." so that we have a
+ * path relative to the symlink. (Not needed for Windows, because our
+ * symlink() wrapper will interpret it as relative to pgdata instead).
+ */
+ if (!is_absolute_path(location))
+ {
+ snprintf(buffer, sizeof(buffer), "../%s", location);
+ location = buffer;
+ }
+#endif
+
/*
* Create the symlink under PGDATA
*/
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 3c61c789e4..5ce1c28174 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -224,6 +224,7 @@ static const char *const subdirs[] = {
"pg_tblspc",
"pg_stat",
"pg_stat_tmp",
+ "pg_user_files",
"pg_xact",
"pg_logical",
"pg_logical/snapshots",
diff --git a/src/common/string.c b/src/common/string.c
index 3aa378c051..7e7e34f1f5 100644
--- a/src/common/string.c
+++ b/src/common/string.c
@@ -24,6 +24,15 @@
#include "common/string.h"
+/*
+ * Returns whether the string `str' has the prefix `start'.
+ */
+bool
+pg_str_startswith(const char *str, const char *start)
+{
+ return strncmp(str, start, strlen(start)) == 0;
+}
+
/*
* Returns whether the string `str' has the postfix `end'.
*/
diff --git a/src/include/common/string.h b/src/include/common/string.h
index 8eb5271ec8..ddd384f6f9 100644
--- a/src/include/common/string.h
+++ b/src/include/common/string.h
@@ -21,6 +21,7 @@ typedef struct PromptInterruptContext
} PromptInterruptContext;
/* functions in src/common/string.c */
+extern bool pg_str_startswith(const char *str, const char *start);
extern bool pg_str_endswith(const char *str, const char *end);
extern int strtoint(const char *pg_restrict str, char **pg_restrict endptr,
int base);
--
2.33.1
v6-0002-CREATE-TABLESPACE-.-NEW-LOCATION.patch (text/x-patch, 7.6K) download
From 4b802e6760d31a477be285b3af71dbabb417469a Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Wed, 4 Aug 2021 17:20:48 +1200
Subject: [PATCH v6 2/4] CREATE TABLESPACE ... NEW LOCATION.
Allow tablespace directories to be created on demand. This allows the
regression test suite to be run without knowing the path of, or having
access to, the server's data directory.
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
doc/src/sgml/ref/create_tablespace.sgml | 14 ++++++----
src/backend/commands/tablespace.c | 36 ++++++++++++++++++-------
src/backend/parser/gram.y | 13 ++++++---
src/include/commands/tablespace.h | 1 +
src/include/nodes/parsenodes.h | 1 +
5 files changed, 47 insertions(+), 18 deletions(-)
diff --git a/doc/src/sgml/ref/create_tablespace.sgml b/doc/src/sgml/ref/create_tablespace.sgml
index 84fa7ee5e2..497b2568fb 100644
--- a/doc/src/sgml/ref/create_tablespace.sgml
+++ b/doc/src/sgml/ref/create_tablespace.sgml
@@ -23,7 +23,7 @@ PostgreSQL documentation
<synopsis>
CREATE TABLESPACE <replaceable class="parameter">tablespace_name</replaceable>
[ OWNER { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER } ]
- LOCATION '<replaceable class="parameter">directory</replaceable>'
+ [ NEW ] LOCATION '<replaceable class="parameter">directory</replaceable>'
[ WITH ( <replaceable class="parameter">tablespace_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] ) ]
</synopsis>
</refsynopsisdiv>
@@ -92,10 +92,14 @@ CREATE TABLESPACE <replaceable class="parameter">tablespace_name</replaceable>
<listitem>
<para>
The directory that will be used for the tablespace. The directory
- must exist (<command>CREATE TABLESPACE</command> will not create it),
- should be empty, and must be owned by the
- <productname>PostgreSQL</productname> system user. The directory must be
- specified by an absolute path name.
+ must exist, be empty, and be owned by the
+ <productname>PostgreSQL</productname> system user unless
+ <command>NEW</command> is specified. If
+ <command>NEW</command> is specified it will be created, and the parent
+ directory must exist.
+ The directory must be specified by an absolute path name, or a path
+ name that is relative to the data directory and begins with
+ <literal>pg_user_files/</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c
index adfbde6b29..b2a1a12fbc 100644
--- a/src/backend/commands/tablespace.c
+++ b/src/backend/commands/tablespace.c
@@ -91,7 +91,8 @@ char *temp_tablespaces = NULL;
static void create_tablespace_directories(const char *location,
- const Oid tablespaceoid);
+ const Oid tablespaceoid,
+ bool missing_ok);
static bool destroy_tablespace_directories(Oid tablespaceoid, bool redo);
@@ -369,13 +370,14 @@ CreateTableSpace(CreateTableSpaceStmt *stmt)
/* Post creation hook for new tablespace */
InvokeObjectPostCreateHook(TableSpaceRelationId, tablespaceoid, 0);
- create_tablespace_directories(location, tablespaceoid);
+ create_tablespace_directories(location, tablespaceoid, stmt->missing_ok);
/* Record the filesystem change in XLOG */
{
xl_tblspc_create_rec xlrec;
xlrec.ts_id = tablespaceoid;
+ xlrec.ts_missing_ok = stmt->missing_ok;
XLogBeginInsert();
XLogRegisterData((char *) &xlrec,
@@ -588,7 +590,8 @@ DropTableSpace(DropTableSpaceStmt *stmt)
* to the specified directory
*/
static void
-create_tablespace_directories(const char *location, const Oid tablespaceoid)
+create_tablespace_directories(const char *location, const Oid tablespaceoid,
+ bool missing_ok)
{
char buffer[MAXPGPATH];
char *linkloc;
@@ -601,16 +604,28 @@ create_tablespace_directories(const char *location, const Oid tablespaceoid)
/*
* Attempt to coerce target directory to safe permissions. If this fails,
- * it doesn't exist or has the wrong owner.
+ * it doesn't exist or has the wrong owner. If NEW LOCATION, we'll create
+ * the directory.
*/
if (chmod(location, pg_dir_create_mode) != 0)
{
if (errno == ENOENT)
- ereport(ERROR,
- (errcode(ERRCODE_UNDEFINED_FILE),
- errmsg("directory \"%s\" does not exist", location),
- InRecovery ? errhint("Create this directory for the tablespace before "
- "restarting the server.") : 0));
+ {
+ if (missing_ok)
+ {
+ if (mkdir(location, pg_dir_create_mode) != 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_FILE),
+ errmsg("could not create directory \"%s\"",
+ location)));
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_FILE),
+ errmsg("directory \"%s\" does not exist", location),
+ InRecovery ? errhint("Create this directory for the tablespace before "
+ "restarting the server.") : 0));
+ }
else
ereport(ERROR,
(errcode_for_file_access(),
@@ -1537,8 +1552,9 @@ tblspc_redo(XLogReaderState *record)
{
xl_tblspc_create_rec *xlrec = (xl_tblspc_create_rec *) XLogRecGetData(record);
char *location = xlrec->ts_path;
+ bool missing_ok = xlrec->ts_missing_ok;
- create_tablespace_directories(location, xlrec->ts_id);
+ create_tablespace_directories(location, xlrec->ts_id, missing_ok);
}
else if (info == XLOG_TBLSPC_DROP)
{
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index a6d0cefa6b..591b6f437f 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -578,6 +578,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <boolean> constraints_set_mode
%type <str> OptTableSpace OptConsTableSpace
%type <rolespec> OptTableSpaceOwner
+%type <boolean> OptTableSpaceNew
%type <ival> opt_check_option
%type <str> opt_provider security_label
@@ -4559,13 +4560,14 @@ opt_procedural:
*
*****************************************************************************/
-CreateTableSpaceStmt: CREATE TABLESPACE name OptTableSpaceOwner LOCATION Sconst opt_reloptions
+CreateTableSpaceStmt: CREATE TABLESPACE name OptTableSpaceOwner OptTableSpaceNew LOCATION Sconst opt_reloptions
{
CreateTableSpaceStmt *n = makeNode(CreateTableSpaceStmt);
n->tablespacename = $3;
n->owner = $4;
- n->location = $6;
- n->options = $7;
+ n->missing_ok = $5;
+ n->location = $7;
+ n->options = $8;
$$ = (Node *) n;
}
;
@@ -4574,6 +4576,11 @@ OptTableSpaceOwner: OWNER RoleSpec { $$ = $2; }
| /*EMPTY */ { $$ = NULL; }
;
+OptTableSpaceNew:
+ NEW { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
/*****************************************************************************
*
* QUERY :
diff --git a/src/include/commands/tablespace.h b/src/include/commands/tablespace.h
index 49cfc8479a..7a53c886c0 100644
--- a/src/include/commands/tablespace.h
+++ b/src/include/commands/tablespace.h
@@ -26,6 +26,7 @@
typedef struct xl_tblspc_create_rec
{
Oid ts_id;
+ bool ts_missing_ok;
char ts_path[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string */
} xl_tblspc_create_rec;
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 067138e6b5..fc3893f965 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -2322,6 +2322,7 @@ typedef struct CreateTableSpaceStmt
RoleSpec *owner;
char *location;
List *options;
+ bool missing_ok;
} CreateTableSpaceStmt;
typedef struct DropTableSpaceStmt
--
2.33.1
v6-0003-Use-relative-paths-for-tablespace-regression-test.patch (text/x-patch, 9.1K) download
From b165768ecec89d648159d71b79c0a0ccfd2c3b06 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Wed, 4 Aug 2021 22:17:54 +1200
Subject: [PATCH v6 3/4] Use relative paths for tablespace regression test.
Remove the machinery from pg_regress that manages the testtablespace
directory. Instead, use a relative path.
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
src/test/regress/GNUmakefile | 3 +-
.../tablespace.out} | 6 +--
src/test/regress/pg_regress.c | 40 -------------------
.../tablespace.source => sql/tablespace.sql} | 6 +--
src/tools/msvc/vcregress.pl | 2 -
5 files changed, 7 insertions(+), 50 deletions(-)
rename src/test/regress/{output/tablespace.source => expected/tablespace.out} (99%)
rename src/test/regress/{input/tablespace.source => sql/tablespace.sql} (98%)
diff --git a/src/test/regress/GNUmakefile b/src/test/regress/GNUmakefile
index fe6e0c98aa..d014b4a35d 100644
--- a/src/test/regress/GNUmakefile
+++ b/src/test/regress/GNUmakefile
@@ -119,7 +119,7 @@ submake-contrib-spi: | submake-libpgport submake-generated-headers
## Run tests
##
-REGRESS_OPTS = --dlpath=. --max-concurrent-tests=20 --make-testtablespace-dir \
+REGRESS_OPTS = --dlpath=. --max-concurrent-tests=20 \
$(EXTRA_REGRESS_OPTS)
check: all
@@ -163,5 +163,4 @@ clean distclean maintainer-clean: clean-lib
rm -f pg_regress_main.o pg_regress.o pg_regress$(X)
# things created by various check targets
rm -f $(output_files) $(input_files)
- rm -rf testtablespace
rm -rf $(pg_regress_clean_files)
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/expected/tablespace.out
similarity index 99%
rename from src/test/regress/output/tablespace.source
rename to src/test/regress/expected/tablespace.out
index e7629d470e..c815901219 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/expected/tablespace.out
@@ -1,7 +1,7 @@
-- create a tablespace using WITH clause
-CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (some_nonexistent_parameter = true); -- fail
+CREATE TABLESPACE regress_tblspacewith LOCATION 'pg_user_files/testtablespace' WITH (some_nonexistent_parameter = true); -- fail
ERROR: unrecognized parameter "some_nonexistent_parameter"
-CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (random_page_cost = 3.0); -- ok
+CREATE TABLESPACE regress_tblspacewith NEW LOCATION 'pg_user_files/testtablespace' WITH (random_page_cost = 3.0); -- ok
-- check to see the parameter was used
SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
spcoptions
@@ -12,7 +12,7 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
-- drop the tablespace so we can re-use the location
DROP TABLESPACE regress_tblspacewith;
-- create a tablespace we can use
-CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
+CREATE TABLESPACE regress_tblspace LOCATION 'pg_user_files/testtablespace';
-- try setting and resetting some properties for the new tablespace
ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true); -- fail
diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index 2c8a600bad..7c1404dac1 100644
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -477,7 +477,6 @@ replace_string(StringInfo string, const char *replace, const char *replacement)
static void
convert_sourcefiles_in(const char *source_subdir, const char *dest_dir, const char *dest_subdir, const char *suffix)
{
- char testtablespace[MAXPGPATH];
char indir[MAXPGPATH];
char outdir_sub[MAXPGPATH];
char **name;
@@ -506,9 +505,6 @@ convert_sourcefiles_in(const char *source_subdir, const char *dest_dir, const ch
if (!directory_exists(outdir_sub))
make_directory(outdir_sub);
- /* We might need to replace @testtablespace@ */
- snprintf(testtablespace, MAXPGPATH, "%s/testtablespace", outputdir);
-
/* finally loop on each file and do the replacement */
for (name = names; *name; name++)
{
@@ -554,7 +550,6 @@ convert_sourcefiles_in(const char *source_subdir, const char *dest_dir, const ch
{
replace_string(&line, "@abs_srcdir@", inputdir);
replace_string(&line, "@abs_builddir@", outputdir);
- replace_string(&line, "@testtablespace@", testtablespace);
replace_string(&line, "@libdir@", dlpath);
replace_string(&line, "@DLSUFFIX@", DLSUFFIX);
fputs(line.data, outfile);
@@ -587,32 +582,6 @@ convert_sourcefiles(void)
convert_sourcefiles_in("output", outputdir, "expected", "out");
}
-/*
- * Clean out the test tablespace dir, or create it if it doesn't exist.
- *
- * On Windows, doing this cleanup here makes it possible to run the
- * regression tests under a Windows administrative user account with the
- * restricted token obtained when starting pg_regress.
- */
-static void
-prepare_testtablespace_dir(void)
-{
- char testtablespace[MAXPGPATH];
-
- snprintf(testtablespace, MAXPGPATH, "%s/testtablespace", outputdir);
-
- if (directory_exists(testtablespace))
- {
- if (!rmtree(testtablespace, true))
- {
- fprintf(stderr, _("\n%s: could not remove test tablespace \"%s\"\n"),
- progname, testtablespace);
- exit(2);
- }
- }
- make_directory(testtablespace);
-}
-
/*
* Scan resultmap file to find which platform-specific expected files to use.
*
@@ -2152,7 +2121,6 @@ help(void)
printf(_(" --launcher=CMD use CMD as launcher of psql\n"));
printf(_(" --load-extension=EXT load the named extension before running the\n"));
printf(_(" tests; can appear multiple times\n"));
- printf(_(" --make-testtablespace-dir create testtablespace directory\n"));
printf(_(" --max-connections=N maximum number of concurrent connections\n"));
printf(_(" (default is 0, meaning unlimited)\n"));
printf(_(" --max-concurrent-tests=N maximum number of concurrent tests in schedule\n"));
@@ -2211,12 +2179,10 @@ regression_main(int argc, char *argv[],
{"load-extension", required_argument, NULL, 22},
{"config-auth", required_argument, NULL, 24},
{"max-concurrent-tests", required_argument, NULL, 25},
- {"make-testtablespace-dir", no_argument, NULL, 26},
{NULL, 0, NULL, 0}
};
bool use_unix_sockets;
- bool make_testtablespace_dir = false;
_stringlist *sl;
int c;
int i;
@@ -2342,9 +2308,6 @@ regression_main(int argc, char *argv[],
case 25:
max_concurrent_tests = atoi(optarg);
break;
- case 26:
- make_testtablespace_dir = true;
- break;
default:
/* getopt_long already emitted a complaint */
fprintf(stderr, _("\nTry \"%s -h\" for more information.\n"),
@@ -2397,9 +2360,6 @@ regression_main(int argc, char *argv[],
unlimit_core_size();
#endif
- if (make_testtablespace_dir)
- prepare_testtablespace_dir();
-
if (temp_instance)
{
FILE *pg_conf;
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/sql/tablespace.sql
similarity index 98%
rename from src/test/regress/input/tablespace.source
rename to src/test/regress/sql/tablespace.sql
index cb9774ecc8..a4a4067c42 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/sql/tablespace.sql
@@ -1,6 +1,6 @@
-- create a tablespace using WITH clause
-CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (some_nonexistent_parameter = true); -- fail
-CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (random_page_cost = 3.0); -- ok
+CREATE TABLESPACE regress_tblspacewith LOCATION 'pg_user_files/testtablespace' WITH (some_nonexistent_parameter = true); -- fail
+CREATE TABLESPACE regress_tblspacewith NEW LOCATION 'pg_user_files/testtablespace' WITH (random_page_cost = 3.0); -- ok
-- check to see the parameter was used
SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
@@ -9,7 +9,7 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
DROP TABLESPACE regress_tblspacewith;
-- create a tablespace we can use
-CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
+CREATE TABLESPACE regress_tblspace LOCATION 'pg_user_files/testtablespace';
-- try setting and resetting some properties for the new tablespace
ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index f3357b3292..92debf9b0f 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -118,7 +118,6 @@ sub installcheck_internal
"--bindir=../../../$Config/psql",
"--schedule=${schedule}_schedule",
"--max-concurrent-tests=20",
- "--make-testtablespace-dir",
"--encoding=SQL_ASCII",
"--no-locale");
push(@args, $maxconn) if $maxconn;
@@ -153,7 +152,6 @@ sub check
"--bindir=",
"--schedule=${schedule}_schedule",
"--max-concurrent-tests=20",
- "--make-testtablespace-dir",
"--encoding=SQL_ASCII",
"--no-locale",
"--temp-instance=./tmp_check");
--
2.33.1
v6-0004-Test-replay-of-regression-tests.patch (text/x-patch, 5.1K) download
From 59c191d072ceced53daf2e1f237564bdbde24425 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 5 Oct 2021 21:01:02 +1300
Subject: [PATCH v6 4/4] Test replay of regression tests.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Add a new TAP test under src/test/recovery to run the standard
regression tests while a streaming replica replays the WAL. This
provides a basic workout for WAL decoding and redo code, and compares
the replicated result.
Optionally, enable (expensive) wal_consistency_checking if listed in
the env variable PG_TEST_EXTRA.
Reviewed-by: 綱川 貴之 (Takayuki Tsunakawa) <tsunakawa.takay@fujitsu.com>
Reviewed-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Anastasia Lubennikova <lubennikovaav@gmail.com>
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
doc/src/sgml/regress.sgml | 11 ++++
src/test/perl/PostgreSQL/Test/Cluster.pm | 2 +-
src/test/recovery/t/027_stream_regress.pl | 68 +++++++++++++++++++++++
3 files changed, 80 insertions(+), 1 deletion(-)
create mode 100644 src/test/recovery/t/027_stream_regress.pl
diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml
index 724ef566e7..64fc81776d 100644
--- a/doc/src/sgml/regress.sgml
+++ b/doc/src/sgml/regress.sgml
@@ -289,6 +289,17 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl'
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>wal_consistency_checking</literal></term>
+ <listitem>
+ <para>
+ Uses <literal>wal_consistency_checking=all</literal> while running
+ some of the tests under <filename>src/test/recovery</filename>. Not
+ enabled by default because it is resource intensive.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
Tests for features that are not supported by the current build
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 9467a199c8..5cfa137cde 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -460,7 +460,7 @@ sub init
print $conf "hot_standby = on\n";
# conservative settings to ensure we can run multiple postmasters:
print $conf "shared_buffers = 1MB\n";
- print $conf "max_connections = 10\n";
+ print $conf "max_connections = 25\n";
# limit disk space consumption, too:
print $conf "max_wal_size = 128MB\n";
}
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
new file mode 100644
index 0000000000..fb37a21bd1
--- /dev/null
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -0,0 +1,68 @@
+# Run the standard regression tests with streaming replication
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 4;
+
+# Initialize primary node
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+$node_primary->init(allows_streaming => 1);
+
+# WAL consistency checking is resource intensive so require opt-in with the
+# PG_TEST_EXTRA environment variable.
+if ($ENV{PG_TEST_EXTRA} &&
+ $ENV{PG_TEST_EXTRA} =~ m/\bwal_consistency_checking\b/) {
+ $node_primary->append_conf('postgresql.conf',
+ 'wal_consistency_checking = all');
+}
+
+$node_primary->start;
+is( $node_primary->psql(
+ 'postgres',
+ qq[SELECT pg_create_physical_replication_slot('standby_1');]),
+ 0,
+ 'physical slot created on primary');
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_primary->backup($backup_name);
+
+# Create streaming standby linking to primary
+my $node_standby_1 = PostgreSQL::Test::Cluster->new('standby_1');
+$node_standby_1->init_from_backup($node_primary, $backup_name,
+ has_streaming => 1);
+$node_standby_1->append_conf('postgresql.conf',
+ "primary_slot_name = standby_1");
+$node_standby_1->start;
+
+# Run the regression tests against the primary.
+system_or_bail("../regress/pg_regress",
+ "--bindir=../../bin/psql",
+ "--port=" . $node_primary->port,
+ "--schedule=../regress/parallel_schedule",
+ "--dlpath=../regress",
+ "--inputdir=../regress");
+
+# Clobber all sequences with their next value, so that we don't have
+# differences between nodes due to caching.
+$node_primary->psql('regression',
+ "select setval(seqrelid, nextval(seqrelid)) from pg_sequence");
+
+# Wait for standby to catch up
+$node_primary->wait_for_catchup($node_standby_1, 'replay',
+ $node_primary->lsn('insert'));
+
+# Perform a logical dump of primary and standby, and check that they match
+command_ok(
+ [ "pg_dump", '-f', 'primary.dump', '--no-sync', '-p', $node_primary->port, 'regression' ],
+ "dump primary server");
+command_ok(
+ [ "pg_dump", '-f', 'standby.dump', '--no-sync', '-p', $node_standby_1->port, 'regression' ],
+ "dump standby server");
+command_ok(
+ [ "diff", 'primary.dump', 'standby.dump' ],
+ "compare primary and standby dumps");
+
+$node_standby_1->stop;
+$node_primary->stop;
--
2.33.1
* Re: A test for replay of regression tests
@ 2021-11-23 15:47 Andrew Dunstan <andrew@dunslane.net>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Andrew Dunstan @ 2021-11-23 15:47 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; Anastasia Lubennikova <lubennikovaav@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On 11/23/21 04:07, Thomas Munro wrote:
> On Wed, Oct 6, 2021 at 7:10 PM Thomas Munro <thomas.munro@gmail.com> wrote:
>> I wonder if for Windows we'd want to switch to real symlinks, since,
>> as far as I know from some light reading, reparse points are converted
>> to absolute paths on creation, so the pgdata directory would be less
>> portable than it would be on POSIX systems.
> I looked into that a bit, and it seems that the newer "real" symlinks
> can't be created without admin privileges, unlike junction points, so
> that wouldn't help. I still need to figure out what this
> junction-based patch set is doing wrong on Windows though trial-by-CI.
> In the meantime, here is a rebase over recent changes to the Perl
> testing APIs.
More exactly you need to "Create Symbolic Links" privilege. see
<https://github.com/git-for-windows/git/wiki/Symbolic-Links;
I haven't yet worked out a way of granting that from the command line,
but if it's working the buildfarm client (as of git tip) will use it on
windows for the workdirs space saving feature.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-11-29 14:14 Andrew Dunstan <andrew@dunslane.net>
parent: Andrew Dunstan <andrew@dunslane.net>
0 siblings, 1 reply; 82+ messages in thread
From: Andrew Dunstan @ 2021-11-29 14:14 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; Anastasia Lubennikova <lubennikovaav@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On 11/23/21 10:47, Andrew Dunstan wrote:
> On 11/23/21 04:07, Thomas Munro wrote:
>> On Wed, Oct 6, 2021 at 7:10 PM Thomas Munro <thomas.munro@gmail.com> wrote:
>>> I wonder if for Windows we'd want to switch to real symlinks, since,
>>> as far as I know from some light reading, reparse points are converted
>>> to absolute paths on creation, so the pgdata directory would be less
>>> portable than it would be on POSIX systems.
>> I looked into that a bit, and it seems that the newer "real" symlinks
>> can't be created without admin privileges, unlike junction points, so
>> that wouldn't help. I still need to figure out what this
>> junction-based patch set is doing wrong on Windows though trial-by-CI.
>> In the meantime, here is a rebase over recent changes to the Perl
>> testing APIs.
>
> More exactly you need to "Create Symbolic Links" privilege. see
> <https://github.com/git-for-windows/git/wiki/Symbolic-Links;
>
>
> I haven't yet worked out a way of granting that from the command line,
> but if it's working the buildfarm client (as of git tip) will use it on
> windows for the workdirs space saving feature.
Update:
There is a PowerShell module called Carbon which provides this and a
whole lot more. It can be installed in numerous ways, including via
Chocolatey. Here's what I am using:
choco install -y Carbon
Import-Module Carbon
Grant-CPrivilege -Identity myuser -Privilege SeCreateSymbolicLinkPrivilege
See <https://get-carbon.org/Grant-Privilege.html; The command name I
used above is now the preferred spelling, although that's not reflected
on the manual page.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-12-04 04:21 Thomas Munro <thomas.munro@gmail.com>
parent: Andrew Dunstan <andrew@dunslane.net>
0 siblings, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2021-12-04 04:21 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Anastasia Lubennikova <lubennikovaav@gmail.com>; Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Tue, Nov 30, 2021 at 3:14 AM Andrew Dunstan <andrew@dunslane.net> wrote:
> choco install -y Carbon
> Import-Module Carbon
> Grant-CPrivilege -Identity myuser -Privilege SeCreateSymbolicLinkPrivilege
Interesting. Well, I found the problem with my last patch (to wit:
junction points must be absolute, unlike real symlinks, which I'd
considered already but I missed that tmp_check's DataDir had a stray
internal \.\), and now I'm wondering whether these newer real symlinks
could help. The constraints are pretty hard to work with... I thought
about a couple of options:
1. We could try to use real symlinks, and fall back to junction
points if that fails. That means that these new tests I'm proposing
would fail unless you grant that privilege or run in developer mode as
you were saying. It bothers me a bit that developers and the BF would
be testing a different code path than production databases run...
unless you're thinking we should switch to symlinks with no fallback,
and require that privilege to be granted by end users to production
servers at least if they want to use tablespaces, and also drop
pre-Win10 support in the same release? That's bigger than I was
thinking.
2. We could convert relative paths to absolute paths at junction
point creation time, which I tried, and "check" now passes. Problems:
(1) now you can't move pgdata around, (2) the is-the-path-too-long
check performed on a primary is not sufficient to check if the
transformed absolute path will be too long on a replica.
The most conservative simple idea I have so far is: go with option 2,
but make this whole thing an undocumented developer-only mode, and
turn it on in the regression tests. Here's a patch set like that.
Thoughts?
Another option would be to stop using operating system symlinks, and
build the target paths ourselves; I didn't investigate that as it
seemed like a bigger change than this warrants.
Next problem: The below is clearly not the right way to find the
pg_regress binary and bindir, and doesn't work on Windows or VPATH.
Any suggestions for how to do this? I feel like something like
$node->installed_command() or similar logic is needed...
# Run the regression tests against the primary.
# XXX How should we find the pg_regress binary and bindir?
system_or_bail("../regress/pg_regress",
"--bindir=../../bin/psql",
"--port=" . $node_primary->port,
"--schedule=../regress/parallel_schedule",
"--dlpath=../regress",
"--inputdir=../regress");
BTW 0002 is one of those renaming patches from git that GNU patch
doesn't seem to apply correctly, sorry cfbot...
Attachments: download all (3 files, .tar.gz)
v7-0001-Allow-restricted-relative-tablespace-paths.patch (text/x-patch, 7.7K) download
From 1a442bfde9b44d28db7344df56e5c0069dbd2364 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 25 May 2021 20:42:17 +1200
Subject: [PATCH v7 1/5] Allow restricted relative tablespace paths.
Traditionally, tablespace paths provided to LOCATION had to be absolute,
because relative paths risked colliding with PostgreSQL-managed files
and generally blurring the boundary between system-managed and
user-managed data.
This commit adds a very small exception: it allows relative paths to be
used, but only under a new directory "pg_user_files", and only if the
developer-only option allow_relative_tablespaces is enabled. This is
intended to be used in automated testing.
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
src/backend/commands/tablespace.c | 55 +++++++++++++++++++++++++++----
src/backend/utils/misc/guc.c | 12 +++++++
src/bin/initdb/initdb.c | 1 +
src/common/string.c | 9 +++++
src/include/commands/tablespace.h | 2 ++
src/include/common/string.h | 1 +
src/port/dirmod.c | 4 +++
7 files changed, 77 insertions(+), 7 deletions(-)
diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c
index 4b96eec9df..7e98ca5b76 100644
--- a/src/backend/commands/tablespace.c
+++ b/src/backend/commands/tablespace.c
@@ -70,6 +70,7 @@
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
#include "common/file_perm.h"
+#include "common/string.h"
#include "miscadmin.h"
#include "postmaster/bgwriter.h"
#include "storage/fd.h"
@@ -87,6 +88,7 @@
/* GUC variables */
char *default_tablespace = NULL;
char *temp_tablespaces = NULL;
+bool allow_relative_tablespaces = false;
static void create_tablespace_directories(const char *location,
@@ -267,11 +269,15 @@ CreateTableSpace(CreateTableSpaceStmt *stmt)
errmsg("tablespace location cannot contain single quotes")));
/*
- * Allowing relative paths seems risky
+ * Allowing relative paths seems risky in general, but we'll allow it under
+ * pg_user_files if a developer-mode option is enabled, for the purpose
+ * of testing.
*
* this also helps us ensure that location is not empty or whitespace
*/
- if (!is_absolute_path(location))
+ if (!is_absolute_path(location) &&
+ (!allow_relative_tablespaces ||
+ !pg_str_startswith(location, "pg_user_files/")))
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
errmsg("tablespace location must be an absolute path")));
@@ -587,6 +593,7 @@ DropTableSpace(DropTableSpaceStmt *stmt)
static void
create_tablespace_directories(const char *location, const Oid tablespaceoid)
{
+ char buffer[MAXPGPATH];
char *linkloc;
char *location_with_version_dir;
struct stat st;
@@ -602,11 +609,27 @@ create_tablespace_directories(const char *location, const Oid tablespaceoid)
if (chmod(location, pg_dir_create_mode) != 0)
{
if (errno == ENOENT)
- ereport(ERROR,
- (errcode(ERRCODE_UNDEFINED_FILE),
- errmsg("directory \"%s\" does not exist", location),
- InRecovery ? errhint("Create this directory for the tablespace before "
- "restarting the server.") : 0));
+ {
+ /*
+ * We're prepared to create directories under pg_user_files on the
+ * fly, for the benefit of regression tests.
+ */
+ if (pg_str_startswith(location, "pg_user_files/"))
+ {
+ Assert(allow_relative_tablespaces || InRecovery);
+ if (mkdir(location, pg_dir_create_mode) != 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_FILE),
+ errmsg("could not create directory \"%s\"",
+ location)));
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_FILE),
+ errmsg("directory \"%s\" does not exist", location),
+ InRecovery ? errhint("Create this directory for the tablespace before "
+ "restarting the server.") : 0));
+ }
else
ereport(ERROR,
(errcode_for_file_access(),
@@ -651,6 +674,24 @@ create_tablespace_directories(const char *location, const Oid tablespaceoid)
if (InRecovery)
remove_tablespace_symlink(linkloc);
+ if (!is_absolute_path(location))
+ {
+#ifdef WIN32
+ /*
+ * Our Windows junction-based symlink() emulation can't handle relative
+ * paths, so convert "pg_user_files" to an absolute path.
+ */
+ snprintf(buffer, sizeof(buffer), "%s/%s", DataDir, location);
+#else
+ /*
+ * Add a ".." prefix so that we have a path relative to the symlink,
+ * rather than DataDir.
+ */
+ snprintf(buffer, sizeof(buffer), "../%s", location);
+#endif
+ location = buffer;
+ }
+
/*
* Create the symlink under PGDATA
*/
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index ee6a838b3a..ab11bee118 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -46,6 +46,7 @@
#include "catalog/storage.h"
#include "commands/async.h"
#include "commands/prepare.h"
+#include "commands/tablespace.h"
#include "commands/trigger.h"
#include "commands/user.h"
#include "commands/vacuum.h"
@@ -1963,6 +1964,17 @@ static struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"allow_relative_tablespaces", PGC_SUSET, DEVELOPER_OPTIONS,
+ gettext_noop("Allows the tablespaces located under pg_user_files."),
+ NULL,
+ GUC_NOT_IN_SAMPLE
+ },
+ &allow_relative_tablespaces,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"lo_compat_privileges", PGC_SUSET, COMPAT_OPTIONS_PREVIOUS,
gettext_noop("Enables backward compatibility mode for privilege checks on large objects."),
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 403adb0ee7..7c4b3e9626 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -225,6 +225,7 @@ static const char *const subdirs[] = {
"pg_tblspc",
"pg_stat",
"pg_stat_tmp",
+ "pg_user_files",
"pg_xact",
"pg_logical",
"pg_logical/snapshots",
diff --git a/src/common/string.c b/src/common/string.c
index 3aa378c051..7e7e34f1f5 100644
--- a/src/common/string.c
+++ b/src/common/string.c
@@ -24,6 +24,15 @@
#include "common/string.h"
+/*
+ * Returns whether the string `str' has the prefix `start'.
+ */
+bool
+pg_str_startswith(const char *str, const char *start)
+{
+ return strncmp(str, start, strlen(start)) == 0;
+}
+
/*
* Returns whether the string `str' has the postfix `end'.
*/
diff --git a/src/include/commands/tablespace.h b/src/include/commands/tablespace.h
index 49cfc8479a..d286ced1c8 100644
--- a/src/include/commands/tablespace.h
+++ b/src/include/commands/tablespace.h
@@ -19,6 +19,8 @@
#include "lib/stringinfo.h"
#include "nodes/parsenodes.h"
+extern bool allow_relative_tablespaces;
+
/* XLOG stuff */
#define XLOG_TBLSPC_CREATE 0x00
#define XLOG_TBLSPC_DROP 0x10
diff --git a/src/include/common/string.h b/src/include/common/string.h
index 8eb5271ec8..ddd384f6f9 100644
--- a/src/include/common/string.h
+++ b/src/include/common/string.h
@@ -21,6 +21,7 @@ typedef struct PromptInterruptContext
} PromptInterruptContext;
/* functions in src/common/string.c */
+extern bool pg_str_startswith(const char *str, const char *start);
extern bool pg_str_endswith(const char *str, const char *end);
extern int strtoint(const char *pg_restrict str, char **pg_restrict endptr,
int base);
diff --git a/src/port/dirmod.c b/src/port/dirmod.c
index 763bc5f915..8787912d03 100644
--- a/src/port/dirmod.c
+++ b/src/port/dirmod.c
@@ -182,6 +182,10 @@ pgsymlink(const char *oldpath, const char *newpath)
while ((p = strchr(p, '/')) != NULL)
*p++ = '\\';
+ /* collapse useless instances of "\.\" to "\" */
+ while ((p = strstr(nativeTarget, "\\.\\")) != NULL)
+ memmove(p, p + 2, strlen(p + 2) + 1);
+
len = strlen(nativeTarget) * sizeof(WCHAR);
reparseBuf->ReparseTag = IO_REPARSE_TAG_MOUNT_POINT;
reparseBuf->ReparseDataLength = len + 12;
--
2.33.1
v7-0002-Use-relative-paths-for-tablespace-regression-test.patch (text/x-patch, 10.4K) download
From 9c4937895d969389157e7f9df9d031ecd1cbc8f4 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Wed, 4 Aug 2021 22:17:54 +1200
Subject: [PATCH v7 2/5] Use relative paths for tablespace regression test.
Remove the machinery from pg_regress that manages the testtablespace
directory. Instead, use a relative path.
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
src/test/regress/GNUmakefile | 3 +-
.../tablespace.out} | 18 +++++++--
src/test/regress/pg_regress.c | 40 -------------------
.../tablespace.source => sql/tablespace.sql} | 19 +++++++--
src/tools/msvc/vcregress.pl | 2 -
5 files changed, 32 insertions(+), 50 deletions(-)
rename src/test/regress/{output/tablespace.source => expected/tablespace.out} (97%)
rename src/test/regress/{input/tablespace.source => sql/tablespace.sql} (95%)
diff --git a/src/test/regress/GNUmakefile b/src/test/regress/GNUmakefile
index fe6e0c98aa..d014b4a35d 100644
--- a/src/test/regress/GNUmakefile
+++ b/src/test/regress/GNUmakefile
@@ -119,7 +119,7 @@ submake-contrib-spi: | submake-libpgport submake-generated-headers
## Run tests
##
-REGRESS_OPTS = --dlpath=. --max-concurrent-tests=20 --make-testtablespace-dir \
+REGRESS_OPTS = --dlpath=. --max-concurrent-tests=20 \
$(EXTRA_REGRESS_OPTS)
check: all
@@ -163,5 +163,4 @@ clean distclean maintainer-clean: clean-lib
rm -f pg_regress_main.o pg_regress.o pg_regress$(X)
# things created by various check targets
rm -f $(output_files) $(input_files)
- rm -rf testtablespace
rm -rf $(pg_regress_clean_files)
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/expected/tablespace.out
similarity index 97%
rename from src/test/regress/output/tablespace.source
rename to src/test/regress/expected/tablespace.out
index e7629d470e..7386693e05 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/expected/tablespace.out
@@ -1,7 +1,19 @@
+-- Note: this test uses relative paths and expects the server to create the
+-- directory on the fly, so that a streaming replica on the same system can
+-- replay it. Normally we require absolute paths to directories that already
+-- exist.
+-- create a tablespace under a relative path
+SET allow_relative_tablespaces = off;
+CREATE TABLESPACE regress_tblspacewith LOCATION 'pg_user_files/testtablespace'; -- fail
+ERROR: tablespace location must be an absolute path
+-- enable developer option allowing relative paths, with unacceptable prefix
+SET allow_relative_tablespaces = on;
+CREATE TABLESPACE regress_tblspacewith LOCATION 'xyz/testtablespace'; -- fail
+ERROR: tablespace location must be an absolute path
-- create a tablespace using WITH clause
-CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (some_nonexistent_parameter = true); -- fail
+CREATE TABLESPACE regress_tblspacewith LOCATION 'pg_user_files/testtablespace' WITH (some_nonexistent_parameter = true); -- fail
ERROR: unrecognized parameter "some_nonexistent_parameter"
-CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (random_page_cost = 3.0); -- ok
+CREATE TABLESPACE regress_tblspacewith LOCATION 'pg_user_files/testtablespace' WITH (random_page_cost = 3.0); -- ok
-- check to see the parameter was used
SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
spcoptions
@@ -12,7 +24,7 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
-- drop the tablespace so we can re-use the location
DROP TABLESPACE regress_tblspacewith;
-- create a tablespace we can use
-CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
+CREATE TABLESPACE regress_tblspace LOCATION 'pg_user_files/testtablespace';
-- try setting and resetting some properties for the new tablespace
ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true); -- fail
diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index 2c8a600bad..7c1404dac1 100644
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -477,7 +477,6 @@ replace_string(StringInfo string, const char *replace, const char *replacement)
static void
convert_sourcefiles_in(const char *source_subdir, const char *dest_dir, const char *dest_subdir, const char *suffix)
{
- char testtablespace[MAXPGPATH];
char indir[MAXPGPATH];
char outdir_sub[MAXPGPATH];
char **name;
@@ -506,9 +505,6 @@ convert_sourcefiles_in(const char *source_subdir, const char *dest_dir, const ch
if (!directory_exists(outdir_sub))
make_directory(outdir_sub);
- /* We might need to replace @testtablespace@ */
- snprintf(testtablespace, MAXPGPATH, "%s/testtablespace", outputdir);
-
/* finally loop on each file and do the replacement */
for (name = names; *name; name++)
{
@@ -554,7 +550,6 @@ convert_sourcefiles_in(const char *source_subdir, const char *dest_dir, const ch
{
replace_string(&line, "@abs_srcdir@", inputdir);
replace_string(&line, "@abs_builddir@", outputdir);
- replace_string(&line, "@testtablespace@", testtablespace);
replace_string(&line, "@libdir@", dlpath);
replace_string(&line, "@DLSUFFIX@", DLSUFFIX);
fputs(line.data, outfile);
@@ -587,32 +582,6 @@ convert_sourcefiles(void)
convert_sourcefiles_in("output", outputdir, "expected", "out");
}
-/*
- * Clean out the test tablespace dir, or create it if it doesn't exist.
- *
- * On Windows, doing this cleanup here makes it possible to run the
- * regression tests under a Windows administrative user account with the
- * restricted token obtained when starting pg_regress.
- */
-static void
-prepare_testtablespace_dir(void)
-{
- char testtablespace[MAXPGPATH];
-
- snprintf(testtablespace, MAXPGPATH, "%s/testtablespace", outputdir);
-
- if (directory_exists(testtablespace))
- {
- if (!rmtree(testtablespace, true))
- {
- fprintf(stderr, _("\n%s: could not remove test tablespace \"%s\"\n"),
- progname, testtablespace);
- exit(2);
- }
- }
- make_directory(testtablespace);
-}
-
/*
* Scan resultmap file to find which platform-specific expected files to use.
*
@@ -2152,7 +2121,6 @@ help(void)
printf(_(" --launcher=CMD use CMD as launcher of psql\n"));
printf(_(" --load-extension=EXT load the named extension before running the\n"));
printf(_(" tests; can appear multiple times\n"));
- printf(_(" --make-testtablespace-dir create testtablespace directory\n"));
printf(_(" --max-connections=N maximum number of concurrent connections\n"));
printf(_(" (default is 0, meaning unlimited)\n"));
printf(_(" --max-concurrent-tests=N maximum number of concurrent tests in schedule\n"));
@@ -2211,12 +2179,10 @@ regression_main(int argc, char *argv[],
{"load-extension", required_argument, NULL, 22},
{"config-auth", required_argument, NULL, 24},
{"max-concurrent-tests", required_argument, NULL, 25},
- {"make-testtablespace-dir", no_argument, NULL, 26},
{NULL, 0, NULL, 0}
};
bool use_unix_sockets;
- bool make_testtablespace_dir = false;
_stringlist *sl;
int c;
int i;
@@ -2342,9 +2308,6 @@ regression_main(int argc, char *argv[],
case 25:
max_concurrent_tests = atoi(optarg);
break;
- case 26:
- make_testtablespace_dir = true;
- break;
default:
/* getopt_long already emitted a complaint */
fprintf(stderr, _("\nTry \"%s -h\" for more information.\n"),
@@ -2397,9 +2360,6 @@ regression_main(int argc, char *argv[],
unlimit_core_size();
#endif
- if (make_testtablespace_dir)
- prepare_testtablespace_dir();
-
if (temp_instance)
{
FILE *pg_conf;
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/sql/tablespace.sql
similarity index 95%
rename from src/test/regress/input/tablespace.source
rename to src/test/regress/sql/tablespace.sql
index cb9774ecc8..6c79eb0333 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/sql/tablespace.sql
@@ -1,6 +1,19 @@
+-- Note: this test uses relative paths and expects the server to create the
+-- directory on the fly, so that a streaming replica on the same system can
+-- replay it. Normally we require absolute paths to directories that already
+-- exist.
+
+-- create a tablespace under a relative path
+SET allow_relative_tablespaces = off;
+CREATE TABLESPACE regress_tblspacewith LOCATION 'pg_user_files/testtablespace'; -- fail
+
+-- enable developer option allowing relative paths, with unacceptable prefix
+SET allow_relative_tablespaces = on;
+CREATE TABLESPACE regress_tblspacewith LOCATION 'xyz/testtablespace'; -- fail
+
-- create a tablespace using WITH clause
-CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (some_nonexistent_parameter = true); -- fail
-CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (random_page_cost = 3.0); -- ok
+CREATE TABLESPACE regress_tblspacewith LOCATION 'pg_user_files/testtablespace' WITH (some_nonexistent_parameter = true); -- fail
+CREATE TABLESPACE regress_tblspacewith LOCATION 'pg_user_files/testtablespace' WITH (random_page_cost = 3.0); -- ok
-- check to see the parameter was used
SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
@@ -9,7 +22,7 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
DROP TABLESPACE regress_tblspacewith;
-- create a tablespace we can use
-CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
+CREATE TABLESPACE regress_tblspace LOCATION 'pg_user_files/testtablespace';
-- try setting and resetting some properties for the new tablespace
ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index f3357b3292..92debf9b0f 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -118,7 +118,6 @@ sub installcheck_internal
"--bindir=../../../$Config/psql",
"--schedule=${schedule}_schedule",
"--max-concurrent-tests=20",
- "--make-testtablespace-dir",
"--encoding=SQL_ASCII",
"--no-locale");
push(@args, $maxconn) if $maxconn;
@@ -153,7 +152,6 @@ sub check
"--bindir=",
"--schedule=${schedule}_schedule",
"--max-concurrent-tests=20",
- "--make-testtablespace-dir",
"--encoding=SQL_ASCII",
"--no-locale",
"--temp-instance=./tmp_check");
--
2.33.1
v7-0003-Test-replay-of-regression-tests.patch (text/x-patch, 5.1K) download
From 5ad0e41cdad1539221f61d2a47a6f3486310fad8 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 5 Oct 2021 21:01:02 +1300
Subject: [PATCH v7 3/5] Test replay of regression tests.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Add a new TAP test under src/test/recovery to run the standard
regression tests while a streaming replica replays the WAL. This
provides a basic workout for WAL decoding and redo code, and compares
the replicated result.
Optionally, enable (expensive) wal_consistency_checking if listed in
the env variable PG_TEST_EXTRA.
Reviewed-by: 綱川 貴之 (Takayuki Tsunakawa) <tsunakawa.takay@fujitsu.com>
Reviewed-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Anastasia Lubennikova <lubennikovaav@gmail.com>
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
doc/src/sgml/regress.sgml | 11 ++++
src/test/perl/PostgreSQL/Test/Cluster.pm | 2 +-
src/test/recovery/t/027_stream_regress.pl | 69 +++++++++++++++++++++++
3 files changed, 81 insertions(+), 1 deletion(-)
create mode 100644 src/test/recovery/t/027_stream_regress.pl
diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml
index 724ef566e7..64fc81776d 100644
--- a/doc/src/sgml/regress.sgml
+++ b/doc/src/sgml/regress.sgml
@@ -289,6 +289,17 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl'
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>wal_consistency_checking</literal></term>
+ <listitem>
+ <para>
+ Uses <literal>wal_consistency_checking=all</literal> while running
+ some of the tests under <filename>src/test/recovery</filename>. Not
+ enabled by default because it is resource intensive.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
Tests for features that are not supported by the current build
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 9467a199c8..5cfa137cde 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -460,7 +460,7 @@ sub init
print $conf "hot_standby = on\n";
# conservative settings to ensure we can run multiple postmasters:
print $conf "shared_buffers = 1MB\n";
- print $conf "max_connections = 10\n";
+ print $conf "max_connections = 25\n";
# limit disk space consumption, too:
print $conf "max_wal_size = 128MB\n";
}
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
new file mode 100644
index 0000000000..a9eead02cf
--- /dev/null
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -0,0 +1,69 @@
+# Run the standard regression tests with streaming replication
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 4;
+
+# Initialize primary node
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+$node_primary->init(allows_streaming => 1);
+
+# WAL consistency checking is resource intensive so require opt-in with the
+# PG_TEST_EXTRA environment variable.
+if ($ENV{PG_TEST_EXTRA} &&
+ $ENV{PG_TEST_EXTRA} =~ m/\bwal_consistency_checking\b/) {
+ $node_primary->append_conf('postgresql.conf',
+ 'wal_consistency_checking = all');
+}
+
+$node_primary->start;
+is( $node_primary->psql(
+ 'postgres',
+ qq[SELECT pg_create_physical_replication_slot('standby_1');]),
+ 0,
+ 'physical slot created on primary');
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_primary->backup($backup_name);
+
+# Create streaming standby linking to primary
+my $node_standby_1 = PostgreSQL::Test::Cluster->new('standby_1');
+$node_standby_1->init_from_backup($node_primary, $backup_name,
+ has_streaming => 1);
+$node_standby_1->append_conf('postgresql.conf',
+ "primary_slot_name = standby_1");
+$node_standby_1->start;
+
+# Run the regression tests against the primary.
+# XXX How should we find the pg_regress binary and bindir?
+system_or_bail("../regress/pg_regress",
+ "--bindir=../../bin/psql",
+ "--port=" . $node_primary->port,
+ "--schedule=../regress/parallel_schedule",
+ "--dlpath=../regress",
+ "--inputdir=../regress");
+
+# Clobber all sequences with their next value, so that we don't have
+# differences between nodes due to caching.
+$node_primary->psql('regression',
+ "select setval(seqrelid, nextval(seqrelid)) from pg_sequence");
+
+# Wait for standby to catch up
+$node_primary->wait_for_catchup($node_standby_1, 'replay',
+ $node_primary->lsn('insert'));
+
+# Perform a logical dump of primary and standby, and check that they match
+command_ok(
+ [ "pg_dump", '-f', 'primary.dump', '--no-sync', '-p', $node_primary->port, 'regression' ],
+ "dump primary server");
+command_ok(
+ [ "pg_dump", '-f', 'standby.dump', '--no-sync', '-p', $node_standby_1->port, 'regression' ],
+ "dump standby server");
+command_ok(
+ [ "diff", 'primary.dump', 'standby.dump' ],
+ "compare primary and standby dumps");
+
+$node_standby_1->stop;
+$node_primary->stop;
--
2.33.1
* Re: A test for replay of regression tests
@ 2021-12-04 15:16 Andrew Dunstan <andrew@dunslane.net>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Andrew Dunstan @ 2021-12-04 15:16 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Anastasia Lubennikova <lubennikovaav@gmail.com>; Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On 12/3/21 23:21, Thomas Munro wrote:
>
> Next problem: The below is clearly not the right way to find the
> pg_regress binary and bindir, and doesn't work on Windows or VPATH.
> Any suggestions for how to do this? I feel like something like
> $node->installed_command() or similar logic is needed...
>
> # Run the regression tests against the primary.
> # XXX How should we find the pg_regress binary and bindir?
> system_or_bail("../regress/pg_regress",
> "--bindir=../../bin/psql",
> "--port=" . $node_primary->port,
> "--schedule=../regress/parallel_schedule",
> "--dlpath=../regress",
> "--inputdir=../regress");
>
TAP tests are passed a path to pg_regress as $ENV{PG_REGRESS}. You
should be using that. On non-MSVC, the path to a non-installed psql is
in this case "$TESTDIR/../../bin/psql" - this should work for VPATH
builds as well as non-VPATH. On MSVC it's a bit harder - it's
"$top_builddir/$releasetype/psql" but we don't expose that. Perhaps we
should. c.f. commit f4ce6c4d3a
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-12-08 23:10 Thomas Munro <thomas.munro@gmail.com>
parent: Andrew Dunstan <andrew@dunslane.net>
0 siblings, 3 replies; 82+ messages in thread
From: Thomas Munro @ 2021-12-08 23:10 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Anastasia Lubennikova <lubennikovaav@gmail.com>; Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Sun, Dec 5, 2021 at 4:16 AM Andrew Dunstan <andrew@dunslane.net> wrote:
> TAP tests are passed a path to pg_regress as $ENV{PG_REGRESS}. You
> should be using that. On non-MSVC, the path to a non-installed psql is
> in this case "$TESTDIR/../../bin/psql" - this should work for VPATH
> builds as well as non-VPATH. On MSVC it's a bit harder - it's
> "$top_builddir/$releasetype/psql" but we don't expose that. Perhaps we
> should. c.f. commit f4ce6c4d3a
Thanks, that helped. Here's a new version that passes on Windows,
Unix and Unix with VPATH. I also had to figure out where the DLLs
are, and make sure that various output files go to the build
directory, not source directory, if different, which I did by passing
down another similar environment variable. Better ideas? (It
confused me for some time that make follows the symlink and runs the
perl code from inside the source directory.)
This adds 2 whole minutes to the recovery check, when running with the
Windows serial-only scripts on Cirrus CI (using Andres's CI patches).
For Linux it adds ~20 seconds to the total of -j8 check-world.
Hopefully that's time well spent, because it adds test coverage for
all the redo routines, and hopefully soon we won't have to run 'em in
series on Windows.
Does anyone want to object to the concept of the "pg_user_files"
directory or the developer-only GUC "allow_relative_tablespaces"?
There's room for discussion about names; maybe initdb shouldn't create
the directory unless you ask it to, or something.
Attachments: download all (3 files, .tar.gz)
v8-0001-Allow-restricted-relative-tablespace-paths.patch (text/x-patch, 7.7K) download
From 6e4cbfea69ceb279b18d3312f37a77beb1f33506 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 25 May 2021 20:42:17 +1200
Subject: [PATCH v8 1/5] Allow restricted relative tablespace paths.
Traditionally, tablespace paths provided to LOCATION had to be absolute,
because relative paths risked colliding with PostgreSQL-managed files
and generally blurring the boundary between system-managed and
user-managed data.
This commit adds a very small exception: it allows relative paths to be
used, but only under a new directory "pg_user_files", and only if the
developer-only option allow_relative_tablespaces is enabled. This is
intended to be used in automated testing.
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
src/backend/commands/tablespace.c | 55 +++++++++++++++++++++++++++----
src/backend/utils/misc/guc.c | 12 +++++++
src/bin/initdb/initdb.c | 1 +
src/common/string.c | 9 +++++
src/include/commands/tablespace.h | 2 ++
src/include/common/string.h | 1 +
src/port/dirmod.c | 4 +++
7 files changed, 77 insertions(+), 7 deletions(-)
diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c
index 4b96eec9df..7e98ca5b76 100644
--- a/src/backend/commands/tablespace.c
+++ b/src/backend/commands/tablespace.c
@@ -70,6 +70,7 @@
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
#include "common/file_perm.h"
+#include "common/string.h"
#include "miscadmin.h"
#include "postmaster/bgwriter.h"
#include "storage/fd.h"
@@ -87,6 +88,7 @@
/* GUC variables */
char *default_tablespace = NULL;
char *temp_tablespaces = NULL;
+bool allow_relative_tablespaces = false;
static void create_tablespace_directories(const char *location,
@@ -267,11 +269,15 @@ CreateTableSpace(CreateTableSpaceStmt *stmt)
errmsg("tablespace location cannot contain single quotes")));
/*
- * Allowing relative paths seems risky
+ * Allowing relative paths seems risky in general, but we'll allow it under
+ * pg_user_files if a developer-mode option is enabled, for the purpose
+ * of testing.
*
* this also helps us ensure that location is not empty or whitespace
*/
- if (!is_absolute_path(location))
+ if (!is_absolute_path(location) &&
+ (!allow_relative_tablespaces ||
+ !pg_str_startswith(location, "pg_user_files/")))
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
errmsg("tablespace location must be an absolute path")));
@@ -587,6 +593,7 @@ DropTableSpace(DropTableSpaceStmt *stmt)
static void
create_tablespace_directories(const char *location, const Oid tablespaceoid)
{
+ char buffer[MAXPGPATH];
char *linkloc;
char *location_with_version_dir;
struct stat st;
@@ -602,11 +609,27 @@ create_tablespace_directories(const char *location, const Oid tablespaceoid)
if (chmod(location, pg_dir_create_mode) != 0)
{
if (errno == ENOENT)
- ereport(ERROR,
- (errcode(ERRCODE_UNDEFINED_FILE),
- errmsg("directory \"%s\" does not exist", location),
- InRecovery ? errhint("Create this directory for the tablespace before "
- "restarting the server.") : 0));
+ {
+ /*
+ * We're prepared to create directories under pg_user_files on the
+ * fly, for the benefit of regression tests.
+ */
+ if (pg_str_startswith(location, "pg_user_files/"))
+ {
+ Assert(allow_relative_tablespaces || InRecovery);
+ if (mkdir(location, pg_dir_create_mode) != 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_FILE),
+ errmsg("could not create directory \"%s\"",
+ location)));
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_FILE),
+ errmsg("directory \"%s\" does not exist", location),
+ InRecovery ? errhint("Create this directory for the tablespace before "
+ "restarting the server.") : 0));
+ }
else
ereport(ERROR,
(errcode_for_file_access(),
@@ -651,6 +674,24 @@ create_tablespace_directories(const char *location, const Oid tablespaceoid)
if (InRecovery)
remove_tablespace_symlink(linkloc);
+ if (!is_absolute_path(location))
+ {
+#ifdef WIN32
+ /*
+ * Our Windows junction-based symlink() emulation can't handle relative
+ * paths, so convert "pg_user_files" to an absolute path.
+ */
+ snprintf(buffer, sizeof(buffer), "%s/%s", DataDir, location);
+#else
+ /*
+ * Add a ".." prefix so that we have a path relative to the symlink,
+ * rather than DataDir.
+ */
+ snprintf(buffer, sizeof(buffer), "../%s", location);
+#endif
+ location = buffer;
+ }
+
/*
* Create the symlink under PGDATA
*/
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index ee6a838b3a..ab11bee118 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -46,6 +46,7 @@
#include "catalog/storage.h"
#include "commands/async.h"
#include "commands/prepare.h"
+#include "commands/tablespace.h"
#include "commands/trigger.h"
#include "commands/user.h"
#include "commands/vacuum.h"
@@ -1963,6 +1964,17 @@ static struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"allow_relative_tablespaces", PGC_SUSET, DEVELOPER_OPTIONS,
+ gettext_noop("Allows the tablespaces located under pg_user_files."),
+ NULL,
+ GUC_NOT_IN_SAMPLE
+ },
+ &allow_relative_tablespaces,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"lo_compat_privileges", PGC_SUSET, COMPAT_OPTIONS_PREVIOUS,
gettext_noop("Enables backward compatibility mode for privilege checks on large objects."),
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 403adb0ee7..7c4b3e9626 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -225,6 +225,7 @@ static const char *const subdirs[] = {
"pg_tblspc",
"pg_stat",
"pg_stat_tmp",
+ "pg_user_files",
"pg_xact",
"pg_logical",
"pg_logical/snapshots",
diff --git a/src/common/string.c b/src/common/string.c
index 3aa378c051..7e7e34f1f5 100644
--- a/src/common/string.c
+++ b/src/common/string.c
@@ -24,6 +24,15 @@
#include "common/string.h"
+/*
+ * Returns whether the string `str' has the prefix `start'.
+ */
+bool
+pg_str_startswith(const char *str, const char *start)
+{
+ return strncmp(str, start, strlen(start)) == 0;
+}
+
/*
* Returns whether the string `str' has the postfix `end'.
*/
diff --git a/src/include/commands/tablespace.h b/src/include/commands/tablespace.h
index 49cfc8479a..d286ced1c8 100644
--- a/src/include/commands/tablespace.h
+++ b/src/include/commands/tablespace.h
@@ -19,6 +19,8 @@
#include "lib/stringinfo.h"
#include "nodes/parsenodes.h"
+extern bool allow_relative_tablespaces;
+
/* XLOG stuff */
#define XLOG_TBLSPC_CREATE 0x00
#define XLOG_TBLSPC_DROP 0x10
diff --git a/src/include/common/string.h b/src/include/common/string.h
index 8eb5271ec8..ddd384f6f9 100644
--- a/src/include/common/string.h
+++ b/src/include/common/string.h
@@ -21,6 +21,7 @@ typedef struct PromptInterruptContext
} PromptInterruptContext;
/* functions in src/common/string.c */
+extern bool pg_str_startswith(const char *str, const char *start);
extern bool pg_str_endswith(const char *str, const char *end);
extern int strtoint(const char *pg_restrict str, char **pg_restrict endptr,
int base);
diff --git a/src/port/dirmod.c b/src/port/dirmod.c
index 763bc5f915..8787912d03 100644
--- a/src/port/dirmod.c
+++ b/src/port/dirmod.c
@@ -182,6 +182,10 @@ pgsymlink(const char *oldpath, const char *newpath)
while ((p = strchr(p, '/')) != NULL)
*p++ = '\\';
+ /* collapse useless instances of "\.\" to "\" */
+ while ((p = strstr(nativeTarget, "\\.\\")) != NULL)
+ memmove(p, p + 2, strlen(p + 2) + 1);
+
len = strlen(nativeTarget) * sizeof(WCHAR);
reparseBuf->ReparseTag = IO_REPARSE_TAG_MOUNT_POINT;
reparseBuf->ReparseDataLength = len + 12;
--
2.30.2
v8-0002-Use-relative-paths-for-tablespace-regression-test.patch (text/x-patch, 10.4K) download
From a60ada37f3ff7d311d56fe453b2abeadf0b350e5 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Wed, 4 Aug 2021 22:17:54 +1200
Subject: [PATCH v8 2/5] Use relative paths for tablespace regression test.
Remove the machinery from pg_regress that manages the testtablespace
directory. Instead, use a relative path.
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
src/test/regress/GNUmakefile | 3 +-
.../tablespace.out} | 18 +++++++--
src/test/regress/pg_regress.c | 40 -------------------
.../tablespace.source => sql/tablespace.sql} | 19 +++++++--
src/tools/msvc/vcregress.pl | 2 -
5 files changed, 32 insertions(+), 50 deletions(-)
rename src/test/regress/{output/tablespace.source => expected/tablespace.out} (97%)
rename src/test/regress/{input/tablespace.source => sql/tablespace.sql} (95%)
diff --git a/src/test/regress/GNUmakefile b/src/test/regress/GNUmakefile
index fe6e0c98aa..d014b4a35d 100644
--- a/src/test/regress/GNUmakefile
+++ b/src/test/regress/GNUmakefile
@@ -119,7 +119,7 @@ submake-contrib-spi: | submake-libpgport submake-generated-headers
## Run tests
##
-REGRESS_OPTS = --dlpath=. --max-concurrent-tests=20 --make-testtablespace-dir \
+REGRESS_OPTS = --dlpath=. --max-concurrent-tests=20 \
$(EXTRA_REGRESS_OPTS)
check: all
@@ -163,5 +163,4 @@ clean distclean maintainer-clean: clean-lib
rm -f pg_regress_main.o pg_regress.o pg_regress$(X)
# things created by various check targets
rm -f $(output_files) $(input_files)
- rm -rf testtablespace
rm -rf $(pg_regress_clean_files)
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/expected/tablespace.out
similarity index 97%
rename from src/test/regress/output/tablespace.source
rename to src/test/regress/expected/tablespace.out
index e7629d470e..7386693e05 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/expected/tablespace.out
@@ -1,7 +1,19 @@
+-- Note: this test uses relative paths and expects the server to create the
+-- directory on the fly, so that a streaming replica on the same system can
+-- replay it. Normally we require absolute paths to directories that already
+-- exist.
+-- create a tablespace under a relative path
+SET allow_relative_tablespaces = off;
+CREATE TABLESPACE regress_tblspacewith LOCATION 'pg_user_files/testtablespace'; -- fail
+ERROR: tablespace location must be an absolute path
+-- enable developer option allowing relative paths, with unacceptable prefix
+SET allow_relative_tablespaces = on;
+CREATE TABLESPACE regress_tblspacewith LOCATION 'xyz/testtablespace'; -- fail
+ERROR: tablespace location must be an absolute path
-- create a tablespace using WITH clause
-CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (some_nonexistent_parameter = true); -- fail
+CREATE TABLESPACE regress_tblspacewith LOCATION 'pg_user_files/testtablespace' WITH (some_nonexistent_parameter = true); -- fail
ERROR: unrecognized parameter "some_nonexistent_parameter"
-CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (random_page_cost = 3.0); -- ok
+CREATE TABLESPACE regress_tblspacewith LOCATION 'pg_user_files/testtablespace' WITH (random_page_cost = 3.0); -- ok
-- check to see the parameter was used
SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
spcoptions
@@ -12,7 +24,7 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
-- drop the tablespace so we can re-use the location
DROP TABLESPACE regress_tblspacewith;
-- create a tablespace we can use
-CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
+CREATE TABLESPACE regress_tblspace LOCATION 'pg_user_files/testtablespace';
-- try setting and resetting some properties for the new tablespace
ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true); -- fail
diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index 2c8a600bad..7c1404dac1 100644
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -477,7 +477,6 @@ replace_string(StringInfo string, const char *replace, const char *replacement)
static void
convert_sourcefiles_in(const char *source_subdir, const char *dest_dir, const char *dest_subdir, const char *suffix)
{
- char testtablespace[MAXPGPATH];
char indir[MAXPGPATH];
char outdir_sub[MAXPGPATH];
char **name;
@@ -506,9 +505,6 @@ convert_sourcefiles_in(const char *source_subdir, const char *dest_dir, const ch
if (!directory_exists(outdir_sub))
make_directory(outdir_sub);
- /* We might need to replace @testtablespace@ */
- snprintf(testtablespace, MAXPGPATH, "%s/testtablespace", outputdir);
-
/* finally loop on each file and do the replacement */
for (name = names; *name; name++)
{
@@ -554,7 +550,6 @@ convert_sourcefiles_in(const char *source_subdir, const char *dest_dir, const ch
{
replace_string(&line, "@abs_srcdir@", inputdir);
replace_string(&line, "@abs_builddir@", outputdir);
- replace_string(&line, "@testtablespace@", testtablespace);
replace_string(&line, "@libdir@", dlpath);
replace_string(&line, "@DLSUFFIX@", DLSUFFIX);
fputs(line.data, outfile);
@@ -587,32 +582,6 @@ convert_sourcefiles(void)
convert_sourcefiles_in("output", outputdir, "expected", "out");
}
-/*
- * Clean out the test tablespace dir, or create it if it doesn't exist.
- *
- * On Windows, doing this cleanup here makes it possible to run the
- * regression tests under a Windows administrative user account with the
- * restricted token obtained when starting pg_regress.
- */
-static void
-prepare_testtablespace_dir(void)
-{
- char testtablespace[MAXPGPATH];
-
- snprintf(testtablespace, MAXPGPATH, "%s/testtablespace", outputdir);
-
- if (directory_exists(testtablespace))
- {
- if (!rmtree(testtablespace, true))
- {
- fprintf(stderr, _("\n%s: could not remove test tablespace \"%s\"\n"),
- progname, testtablespace);
- exit(2);
- }
- }
- make_directory(testtablespace);
-}
-
/*
* Scan resultmap file to find which platform-specific expected files to use.
*
@@ -2152,7 +2121,6 @@ help(void)
printf(_(" --launcher=CMD use CMD as launcher of psql\n"));
printf(_(" --load-extension=EXT load the named extension before running the\n"));
printf(_(" tests; can appear multiple times\n"));
- printf(_(" --make-testtablespace-dir create testtablespace directory\n"));
printf(_(" --max-connections=N maximum number of concurrent connections\n"));
printf(_(" (default is 0, meaning unlimited)\n"));
printf(_(" --max-concurrent-tests=N maximum number of concurrent tests in schedule\n"));
@@ -2211,12 +2179,10 @@ regression_main(int argc, char *argv[],
{"load-extension", required_argument, NULL, 22},
{"config-auth", required_argument, NULL, 24},
{"max-concurrent-tests", required_argument, NULL, 25},
- {"make-testtablespace-dir", no_argument, NULL, 26},
{NULL, 0, NULL, 0}
};
bool use_unix_sockets;
- bool make_testtablespace_dir = false;
_stringlist *sl;
int c;
int i;
@@ -2342,9 +2308,6 @@ regression_main(int argc, char *argv[],
case 25:
max_concurrent_tests = atoi(optarg);
break;
- case 26:
- make_testtablespace_dir = true;
- break;
default:
/* getopt_long already emitted a complaint */
fprintf(stderr, _("\nTry \"%s -h\" for more information.\n"),
@@ -2397,9 +2360,6 @@ regression_main(int argc, char *argv[],
unlimit_core_size();
#endif
- if (make_testtablespace_dir)
- prepare_testtablespace_dir();
-
if (temp_instance)
{
FILE *pg_conf;
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/sql/tablespace.sql
similarity index 95%
rename from src/test/regress/input/tablespace.source
rename to src/test/regress/sql/tablespace.sql
index cb9774ecc8..6c79eb0333 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/sql/tablespace.sql
@@ -1,6 +1,19 @@
+-- Note: this test uses relative paths and expects the server to create the
+-- directory on the fly, so that a streaming replica on the same system can
+-- replay it. Normally we require absolute paths to directories that already
+-- exist.
+
+-- create a tablespace under a relative path
+SET allow_relative_tablespaces = off;
+CREATE TABLESPACE regress_tblspacewith LOCATION 'pg_user_files/testtablespace'; -- fail
+
+-- enable developer option allowing relative paths, with unacceptable prefix
+SET allow_relative_tablespaces = on;
+CREATE TABLESPACE regress_tblspacewith LOCATION 'xyz/testtablespace'; -- fail
+
-- create a tablespace using WITH clause
-CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (some_nonexistent_parameter = true); -- fail
-CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (random_page_cost = 3.0); -- ok
+CREATE TABLESPACE regress_tblspacewith LOCATION 'pg_user_files/testtablespace' WITH (some_nonexistent_parameter = true); -- fail
+CREATE TABLESPACE regress_tblspacewith LOCATION 'pg_user_files/testtablespace' WITH (random_page_cost = 3.0); -- ok
-- check to see the parameter was used
SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
@@ -9,7 +22,7 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
DROP TABLESPACE regress_tblspacewith;
-- create a tablespace we can use
-CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
+CREATE TABLESPACE regress_tblspace LOCATION 'pg_user_files/testtablespace';
-- try setting and resetting some properties for the new tablespace
ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index 5975e7c9cd..bf66083f73 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -133,7 +133,6 @@ sub installcheck_internal
"--bindir=../../../$Config/psql",
"--schedule=${schedule}_schedule",
"--max-concurrent-tests=20",
- "--make-testtablespace-dir",
"--encoding=SQL_ASCII",
"--no-locale");
push(@args, $maxconn) if $maxconn;
@@ -168,7 +167,6 @@ sub check
"--bindir=",
"--schedule=${schedule}_schedule",
"--max-concurrent-tests=20",
- "--make-testtablespace-dir",
"--encoding=SQL_ASCII",
"--no-locale",
"--temp-instance=./tmp_check");
--
2.30.2
v8-0003-Test-replay-of-regression-tests.patch (text/x-patch, 6.4K) download
From 5d76158609f6aa338fad7dd0ef872715582b0b08 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 5 Oct 2021 21:01:02 +1300
Subject: [PATCH v8 3/5] Test replay of regression tests.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Add a new TAP test under src/test/recovery to run the standard
regression tests while a streaming replica replays the WAL. This
provides a basic workout for WAL decoding and redo code, and compares
the replicated result.
Optionally, enable (expensive) wal_consistency_checking if listed in
the env variable PG_TEST_EXTRA.
Reviewed-by: 綱川 貴之 (Takayuki Tsunakawa) <tsunakawa.takay@fujitsu.com>
Reviewed-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Anastasia Lubennikova <lubennikovaav@gmail.com>
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
doc/src/sgml/regress.sgml | 11 ++++
src/test/perl/PostgreSQL/Test/Cluster.pm | 2 +-
src/test/recovery/Makefile | 6 +-
src/test/recovery/t/027_stream_regress.pl | 75 +++++++++++++++++++++++
src/tools/msvc/vcregress.pl | 2 +
5 files changed, 94 insertions(+), 2 deletions(-)
create mode 100644 src/test/recovery/t/027_stream_regress.pl
diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml
index 724ef566e7..64fc81776d 100644
--- a/doc/src/sgml/regress.sgml
+++ b/doc/src/sgml/regress.sgml
@@ -289,6 +289,17 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl'
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>wal_consistency_checking</literal></term>
+ <listitem>
+ <para>
+ Uses <literal>wal_consistency_checking=all</literal> while running
+ some of the tests under <filename>src/test/recovery</filename>. Not
+ enabled by default because it is resource intensive.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
Tests for features that are not supported by the current build
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 9467a199c8..5cfa137cde 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -460,7 +460,7 @@ sub init
print $conf "hot_standby = on\n";
# conservative settings to ensure we can run multiple postmasters:
print $conf "shared_buffers = 1MB\n";
- print $conf "max_connections = 10\n";
+ print $conf "max_connections = 25\n";
# limit disk space consumption, too:
print $conf "max_wal_size = 128MB\n";
}
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
index 288c04b861..c557018b38 100644
--- a/src/test/recovery/Makefile
+++ b/src/test/recovery/Makefile
@@ -15,10 +15,14 @@ subdir = src/test/recovery
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-# required for 017_shm.pl
+# required for 017_shm.pl and 027_stream_regress.pl
REGRESS_SHLIB=$(abs_top_builddir)/src/test/regress/regress$(DLSUFFIX)
export REGRESS_SHLIB
+# required for 027_stream_regress.pl
+REGRESS_OUTPUTDIR=$(abs_top_builddir)/src/test/recovery
+export REGRESS_OUTPUTDIR
+
check:
$(prove_check)
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
new file mode 100644
index 0000000000..d656762d29
--- /dev/null
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -0,0 +1,75 @@
+# Run the standard regression tests with streaming replication
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 4;
+use File::Basename;
+
+# Initialize primary node
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+$node_primary->init(allows_streaming => 1);
+
+# WAL consistency checking is resource intensive so require opt-in with the
+# PG_TEST_EXTRA environment variable.
+if ($ENV{PG_TEST_EXTRA} &&
+ $ENV{PG_TEST_EXTRA} =~ m/\bwal_consistency_checking\b/) {
+ $node_primary->append_conf('postgresql.conf',
+ 'wal_consistency_checking = all');
+}
+
+$node_primary->start;
+is( $node_primary->psql(
+ 'postgres',
+ qq[SELECT pg_create_physical_replication_slot('standby_1');]),
+ 0,
+ 'physical slot created on primary');
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_primary->backup($backup_name);
+
+# Create streaming standby linking to primary
+my $node_standby_1 = PostgreSQL::Test::Cluster->new('standby_1');
+$node_standby_1->init_from_backup($node_primary, $backup_name,
+ has_streaming => 1);
+$node_standby_1->append_conf('postgresql.conf',
+ "primary_slot_name = standby_1");
+$node_standby_1->start;
+
+my $dlpath = PostgreSQL::Test::Utils::perl2host(dirname($ENV{REGRESS_SHLIB}));
+my $outputdir = PostgreSQL::Test::Utils::perl2host($ENV{REGRESS_OUTPUTDIR});
+
+# Run the regression tests against the primary.
+system_or_bail($ENV{PG_REGRESS},
+ "--dlpath=" . $dlpath,
+ "--bindir=",
+ "--port=" . $node_primary->port,
+ "--schedule=../regress/parallel_schedule",
+ "--inputdir=../regress",
+ "--outputdir=" . $outputdir);
+
+# Clobber all sequences with their next value, so that we don't have
+# differences between nodes due to caching.
+$node_primary->psql('regression',
+ "select setval(seqrelid, nextval(seqrelid)) from pg_sequence");
+
+# Wait for standby to catch up
+$node_primary->wait_for_catchup($node_standby_1, 'replay',
+ $node_primary->lsn('insert'));
+
+# Perform a logical dump of primary and standby, and check that they match
+command_ok(
+ [ "pg_dump", '-f', $outputdir . '/primary.dump', '--no-sync',
+ '-p', $node_primary->port, 'regression' ],
+ "dump primary server");
+command_ok(
+ [ "pg_dump", '-f', $outputdir . '/standby.dump', '--no-sync',
+ '-p', $node_standby_1->port, 'regression' ],
+ "dump standby server");
+command_ok(
+ [ "diff", $outputdir . '/primary.dump', $outputdir . '/standby.dump' ],
+ "compare primary and standby dumps");
+
+$node_standby_1->stop;
+$node_primary->stop;
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index bf66083f73..d002bdac9d 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -535,6 +535,8 @@ sub recoverycheck
{
InstallTemp();
+ $ENV{REGRESS_OUTPUTDIR} = "$topdir/src/test/recovery";
+
my $mstat = 0;
my $dir = "$topdir/src/test/recovery";
my $status = tap_check($dir);
--
2.30.2
* Re: A test for replay of regression tests
@ 2021-12-09 13:12 Andrew Dunstan <andrew@dunslane.net>
parent: Thomas Munro <thomas.munro@gmail.com>
2 siblings, 2 replies; 82+ messages in thread
From: Andrew Dunstan @ 2021-12-09 13:12 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Anastasia Lubennikova <lubennikovaav@gmail.com>; Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On 12/8/21 18:10, Thomas Munro wrote:
> On Sun, Dec 5, 2021 at 4:16 AM Andrew Dunstan <andrew@dunslane.net> wrote:
>> TAP tests are passed a path to pg_regress as $ENV{PG_REGRESS}. You
>> should be using that. On non-MSVC, the path to a non-installed psql is
>> in this case "$TESTDIR/../../bin/psql" - this should work for VPATH
>> builds as well as non-VPATH. On MSVC it's a bit harder - it's
>> "$top_builddir/$releasetype/psql" but we don't expose that. Perhaps we
>> should. c.f. commit f4ce6c4d3a
> Thanks, that helped. Here's a new version that passes on Windows,
> Unix and Unix with VPATH. I also had to figure out where the DLLs
> are, and make sure that various output files go to the build
> directory, not source directory, if different, which I did by passing
> down another similar environment variable. Better ideas? (It
> confused me for some time that make follows the symlink and runs the
> perl code from inside the source directory.)
The new version appears to set an empty --bindir for pg_regress. Is that
right?
> This adds 2 whole minutes to the recovery check, when running with the
> Windows serial-only scripts on Cirrus CI (using Andres's CI patches).
> For Linux it adds ~20 seconds to the total of -j8 check-world.
> Hopefully that's time well spent, because it adds test coverage for
> all the redo routines, and hopefully soon we won't have to run 'em in
> series on Windows.
>
> Does anyone want to object to the concept of the "pg_user_files"
> directory or the developer-only GUC "allow_relative_tablespaces"?
> There's room for discussion about names; maybe initdb shouldn't create
> the directory unless you ask it to, or something.
I'm slightly worried that some bright spark will discover it and think
it's a good idea for a production setup.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-12-09 19:38 Andres Freund <andres@anarazel.de>
parent: Andrew Dunstan <andrew@dunslane.net>
1 sibling, 1 reply; 82+ messages in thread
From: Andres Freund @ 2021-12-09 19:38 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
Hi,
On 2021-12-09 08:12:14 -0500, Andrew Dunstan wrote:
> > Does anyone want to object to the concept of the "pg_user_files"
> > directory or the developer-only GUC "allow_relative_tablespaces"?
> > There's room for discussion about names; maybe initdb shouldn't create
> > the directory unless you ask it to, or something.
Personally I'd rather put relative tablespaces into a dedicated directory or
just into pg_tblspc, but without a symlink. Some tools need to understand
tablespace layout etc, and having them in a directory that, by the name, will
also contain other things seems likely to cause confusion.
> I'm slightly worried that some bright spark will discover it and think
> it's a good idea for a production setup.
It'd not really be worse than the current situation of accidentally corrupting
a local replica or such :/.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-12-09 20:15 Thomas Munro <thomas.munro@gmail.com>
parent: Andrew Dunstan <andrew@dunslane.net>
1 sibling, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2021-12-09 20:15 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Anastasia Lubennikova <lubennikovaav@gmail.com>; Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Fri, Dec 10, 2021 at 2:12 AM Andrew Dunstan <andrew@dunslane.net> wrote:
> The new version appears to set an empty --bindir for pg_regress. Is that
> right?
It seems to be necessary to find eg psql, since --bindir='' means
"expect $PATH to contain the installed binaries", and that's working
on both build systems. The alternative would be to export yet another
environment variable, $PG_INSTALL or such -- do you think that'd be
better, or did I miss something that exists already like that?
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-12-09 21:35 Thomas Munro <thomas.munro@gmail.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 82+ messages in thread
From: Thomas Munro @ 2021-12-09 21:35 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Fri, Dec 10, 2021 at 8:38 AM Andres Freund <andres@anarazel.de> wrote:
> On 2021-12-09 08:12:14 -0500, Andrew Dunstan wrote:
> > > Does anyone want to object to the concept of the "pg_user_files"
> > > directory or the developer-only GUC "allow_relative_tablespaces"?
> > > There's room for discussion about names; maybe initdb shouldn't create
> > > the directory unless you ask it to, or something.
>
> Personally I'd rather put relative tablespaces into a dedicated directory or
> just into pg_tblspc, but without a symlink. Some tools need to understand
> tablespace layout etc, and having them in a directory that, by the name, will
> also contain other things seems likely to cause confusion.
Alright, let me try it that way... more soon.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-12-09 21:36 Andres Freund <andres@anarazel.de>
parent: Thomas Munro <thomas.munro@gmail.com>
2 siblings, 1 reply; 82+ messages in thread
From: Andres Freund @ 2021-12-09 21:36 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
Hi,
On 2021-12-09 12:10:23 +1300, Thomas Munro wrote:
> From a60ada37f3ff7d311d56fe453b2abeadf0b350e5 Mon Sep 17 00:00:00 2001
> From: Thomas Munro <thomas.munro@gmail.com>
> Date: Wed, 4 Aug 2021 22:17:54 +1200
> Subject: [PATCH v8 2/5] Use relative paths for tablespace regression test.
>
> Remove the machinery from pg_regress that manages the testtablespace
> directory. Instead, use a relative path.
>
> Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
Seems like we ought to add a tiny tap test or such for this - otherwise we
won't have any coverage of "normal" tablespaces? I don't think they need to be
exercised as part of the normal tests, but having some very basic testing
in a tap test seems worthwhile.
> diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
> index 9467a199c8..5cfa137cde 100644
> --- a/src/test/perl/PostgreSQL/Test/Cluster.pm
> +++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
> @@ -460,7 +460,7 @@ sub init
> print $conf "hot_standby = on\n";
> # conservative settings to ensure we can run multiple postmasters:
> print $conf "shared_buffers = 1MB\n";
> - print $conf "max_connections = 10\n";
> + print $conf "max_connections = 25\n";
> # limit disk space consumption, too:
> print $conf "max_wal_size = 128MB\n";
> }
What's the relation of this to the rest?
> +# Perform a logical dump of primary and standby, and check that they match
> +command_ok(
> + [ "pg_dump", '-f', $outputdir . '/primary.dump', '--no-sync',
> + '-p', $node_primary->port, 'regression' ],
> + "dump primary server");
> +command_ok(
> + [ "pg_dump", '-f', $outputdir . '/standby.dump', '--no-sync',
> + '-p', $node_standby_1->port, 'regression' ],
> + "dump standby server");
> +command_ok(
> + [ "diff", $outputdir . '/primary.dump', $outputdir . '/standby.dump' ],
> + "compare primary and standby dumps");
> +
Absurd nitpick: What's the deal with using "" for one part, and '' for the
rest?
Separately: I think the case of seeing diffs will be too hard to debug like
this, as the difference isn't shown afaict?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-12-09 23:58 Thomas Munro <thomas.munro@gmail.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 3 replies; 82+ messages in thread
From: Thomas Munro @ 2021-12-09 23:58 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Fri, Dec 10, 2021 at 10:36 AM Andres Freund <andres@anarazel.de> wrote:
> Seems like we ought to add a tiny tap test or such for this - otherwise we
> won't have any coverage of "normal" tablespaces? I don't think they need to be
> exercised as part of the normal tests, but having some very basic testing
> in a tap test seems worthwhile.
Good idea, that was bothering me too. Done.
> > - print $conf "max_connections = 10\n";
> > + print $conf "max_connections = 25\n";
> What's the relation of this to the rest?
Someone decided that allow_streaming should imply max_connections =
10, but we need ~20 to run the parallel regression test schedule.
However, I can just as easily move that to a local adjustment in the
TAP test file. Done, like so:
+$node_primary->adjust_conf('postgresql.conf', 'max_connections', '25', 1);
> Absurd nitpick: What's the deal with using "" for one part, and '' for the
> rest?
Fixed.
> Separately: I think the case of seeing diffs will be too hard to debug like
> this, as the difference isn't shown afaict?
Seems to be OK. The output goes to
src/test/recovery/tmp_check/log/regress_log_027_stream_regress, so for
example if you comment out the bit that deals with SEQUENCE caching
you'll see:
# Running: pg_dump -f
/usr/home/tmunro/projects/postgresql/src/test/recovery/primary.dump
--no-sync -p 63693 regression
ok 2 - dump primary server
# Running: pg_dump -f
/usr/home/tmunro/projects/postgresql/src/test/recovery/standby.dump
--no-sync -p 63694 regression
ok 3 - dump standby server
# Running: diff
/usr/home/tmunro/projects/postgresql/src/test/recovery/primary.dump
/usr/home/tmunro/projects/postgresql/src/test/recovery/standby.dump
436953c436953
< SELECT pg_catalog.setval('public.clstr_tst_s_rf_a_seq', 32, true);
---
> SELECT pg_catalog.setval('public.clstr_tst_s_rf_a_seq', 33, true);
... more hunks ...
And from the previous email:
On Fri, Dec 10, 2021 at 10:35 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Fri, Dec 10, 2021 at 8:38 AM Andres Freund <andres@anarazel.de> wrote:
> > Personally I'd rather put relative tablespaces into a dedicated directory or
> > just into pg_tblspc, but without a symlink. Some tools need to understand
> > tablespace layout etc, and having them in a directory that, by the name, will
> > also contain other things seems likely to cause confusion.
Ok, in this version I have a developer-only GUC
allow_in_place_tablespaces instead. If you turn it on, you can do:
CREATE TABLESPACE regress_blah LOCATION = '';
... and then pg_tblspc/OID is created directly as a directory. Not
allowed by default or documented.
Attachments: download all (4 files, .tar.gz)
v9-0001-Allow-in-place-tablespaces.patch (text/x-patch, 5.2K) download
From 345fe049309ed84a377cda8e5a3c6df10482ae9a Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Fri, 10 Dec 2021 11:45:24 +1300
Subject: [PATCH v9 1/6] Allow "in place" tablespaces.
Provide a developer-only GUC allow_in_place_tablespaces, disabled by
default. When enabled, tablespaces can be created with an empty
LOCATION string, meaning that they should be created as a directory
directly beneath pg_tblspc. This can be used for new testing scenarios,
in a follow-up patch. Not intended for end-user usage, since it might
confuse backup tools that expect symlinks.
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
src/backend/commands/tablespace.c | 36 +++++++++++++++++++++++++------
src/backend/utils/misc/guc.c | 12 +++++++++++
src/include/commands/tablespace.h | 2 ++
3 files changed, 44 insertions(+), 6 deletions(-)
diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c
index 4b96eec9df..7a950c75a7 100644
--- a/src/backend/commands/tablespace.c
+++ b/src/backend/commands/tablespace.c
@@ -87,6 +87,7 @@
/* GUC variables */
char *default_tablespace = NULL;
char *temp_tablespaces = NULL;
+bool allow_in_place_tablespaces = false;
static void create_tablespace_directories(const char *location,
@@ -241,6 +242,7 @@ CreateTableSpace(CreateTableSpaceStmt *stmt)
char *location;
Oid ownerId;
Datum newOptions;
+ bool in_place;
/* Must be superuser */
if (!superuser())
@@ -266,12 +268,15 @@ CreateTableSpace(CreateTableSpaceStmt *stmt)
(errcode(ERRCODE_INVALID_NAME),
errmsg("tablespace location cannot contain single quotes")));
+ in_place = allow_in_place_tablespaces && strlen(location) == 0;
+
/*
* Allowing relative paths seems risky
*
- * this also helps us ensure that location is not empty or whitespace
+ * This also helps us ensure that location is not empty or whitespace,
+ * unless specifying a developer-only in-place tablespace.
*/
- if (!is_absolute_path(location))
+ if (!in_place && !is_absolute_path(location))
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
errmsg("tablespace location must be an absolute path")));
@@ -590,16 +595,35 @@ create_tablespace_directories(const char *location, const Oid tablespaceoid)
char *linkloc;
char *location_with_version_dir;
struct stat st;
+ bool in_place;
linkloc = psprintf("pg_tblspc/%u", tablespaceoid);
- location_with_version_dir = psprintf("%s/%s", location,
+
+ /*
+ * If we're asked to make an 'in place' tablespace, create the directory
+ * directly where the symlink would normally go. This is a developer-only
+ * option for now, to facilitate regression testing.
+ */
+ in_place =
+ (allow_in_place_tablespaces || InRecovery) && strlen(location) == 0;
+
+ if (in_place)
+ {
+ if (MakePGDirectory(linkloc) < 0 && errno != EEXIST)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create directory \"%s\": %m",
+ linkloc)));
+ }
+
+ location_with_version_dir = psprintf("%s/%s", in_place ? linkloc : location,
TABLESPACE_VERSION_DIRECTORY);
/*
* Attempt to coerce target directory to safe permissions. If this fails,
* it doesn't exist or has the wrong owner.
*/
- if (chmod(location, pg_dir_create_mode) != 0)
+ if (!in_place && chmod(location, pg_dir_create_mode) != 0)
{
if (errno == ENOENT)
ereport(ERROR,
@@ -648,13 +672,13 @@ create_tablespace_directories(const char *location, const Oid tablespaceoid)
/*
* In recovery, remove old symlink, in case it points to the wrong place.
*/
- if (InRecovery)
+ if (!in_place && InRecovery)
remove_tablespace_symlink(linkloc);
/*
* Create the symlink under PGDATA
*/
- if (symlink(location, linkloc) < 0)
+ if (!in_place && symlink(location, linkloc) < 0)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not create symbolic link \"%s\": %m",
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index ee6a838b3a..0fafacd726 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -46,6 +46,7 @@
#include "catalog/storage.h"
#include "commands/async.h"
#include "commands/prepare.h"
+#include "commands/tablespace.h"
#include "commands/trigger.h"
#include "commands/user.h"
#include "commands/vacuum.h"
@@ -1963,6 +1964,17 @@ static struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"allow_in_place_tablespaces", PGC_SUSET, DEVELOPER_OPTIONS,
+ gettext_noop("Allows tablespaces directly inside pg_tblspc, for testing."),
+ NULL,
+ GUC_NOT_IN_SAMPLE
+ },
+ &allow_in_place_tablespaces,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"lo_compat_privileges", PGC_SUSET, COMPAT_OPTIONS_PREVIOUS,
gettext_noop("Enables backward compatibility mode for privilege checks on large objects."),
diff --git a/src/include/commands/tablespace.h b/src/include/commands/tablespace.h
index 49cfc8479a..f33ba51bd6 100644
--- a/src/include/commands/tablespace.h
+++ b/src/include/commands/tablespace.h
@@ -19,6 +19,8 @@
#include "lib/stringinfo.h"
#include "nodes/parsenodes.h"
+extern bool allow_in_place_tablespaces;
+
/* XLOG stuff */
#define XLOG_TBLSPC_CREATE 0x00
#define XLOG_TBLSPC_DROP 0x10
--
2.33.1
v9-0002-Use-in-place-tablespaces-in-regression-test.patch (text/x-patch, 10.2K) download
From 02554d87073ca3019558b5a02bd4b89bcb1e7e73 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Wed, 4 Aug 2021 22:17:54 +1200
Subject: [PATCH v9 2/6] Use in-place tablespaces in regression test.
Remove the machinery from pg_regress that manages the testtablespace
directory. Instead, use "in-place" tablespaces, because they work
correctly when there is a streaming replica running on the same host.
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
src/test/regress/GNUmakefile | 3 +-
.../tablespace.out} | 17 ++++++--
src/test/regress/pg_regress.c | 40 -------------------
.../tablespace.source => sql/tablespace.sql} | 18 +++++++--
src/tools/msvc/vcregress.pl | 2 -
5 files changed, 30 insertions(+), 50 deletions(-)
rename src/test/regress/{output/tablespace.source => expected/tablespace.out} (97%)
rename src/test/regress/{input/tablespace.source => sql/tablespace.sql} (96%)
diff --git a/src/test/regress/GNUmakefile b/src/test/regress/GNUmakefile
index fe6e0c98aa..d014b4a35d 100644
--- a/src/test/regress/GNUmakefile
+++ b/src/test/regress/GNUmakefile
@@ -119,7 +119,7 @@ submake-contrib-spi: | submake-libpgport submake-generated-headers
## Run tests
##
-REGRESS_OPTS = --dlpath=. --max-concurrent-tests=20 --make-testtablespace-dir \
+REGRESS_OPTS = --dlpath=. --max-concurrent-tests=20 \
$(EXTRA_REGRESS_OPTS)
check: all
@@ -163,5 +163,4 @@ clean distclean maintainer-clean: clean-lib
rm -f pg_regress_main.o pg_regress.o pg_regress$(X)
# things created by various check targets
rm -f $(output_files) $(input_files)
- rm -rf testtablespace
rm -rf $(pg_regress_clean_files)
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/expected/tablespace.out
similarity index 97%
rename from src/test/regress/output/tablespace.source
rename to src/test/regress/expected/tablespace.out
index e7629d470e..4169c36cde 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/expected/tablespace.out
@@ -1,7 +1,18 @@
+-- relative tablespace locations are not allowed
+CREATE TABLESPACE regress_tblspace LOCATION 'relative'; -- fail
+ERROR: tablespace location must be an absolute path
+-- empty tablespace locations are not usually allowed
+CREATE TABLESPACE regress_tblspace LOCATION ''; -- fail
+ERROR: tablespace location must be an absolute path
+-- as a special developer-only option to allow us to use tablespaces
+-- with streaming replication on the same server, an empty location
+-- can be allowed as a way to say that the tablespace should be created
+-- as a directory in pg_tblspc, rather than being a symlink
+SET allow_in_place_tablespaces = true;
-- create a tablespace using WITH clause
-CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (some_nonexistent_parameter = true); -- fail
+CREATE TABLESPACE regress_tblspacewith LOCATION '' WITH (some_nonexistent_parameter = true); -- fail
ERROR: unrecognized parameter "some_nonexistent_parameter"
-CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (random_page_cost = 3.0); -- ok
+CREATE TABLESPACE regress_tblspacewith LOCATION '' WITH (random_page_cost = 3.0); -- ok
-- check to see the parameter was used
SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
spcoptions
@@ -12,7 +23,7 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
-- drop the tablespace so we can re-use the location
DROP TABLESPACE regress_tblspacewith;
-- create a tablespace we can use
-CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
+CREATE TABLESPACE regress_tblspace LOCATION '';
-- try setting and resetting some properties for the new tablespace
ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true); -- fail
diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index 2c8a600bad..7c1404dac1 100644
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -477,7 +477,6 @@ replace_string(StringInfo string, const char *replace, const char *replacement)
static void
convert_sourcefiles_in(const char *source_subdir, const char *dest_dir, const char *dest_subdir, const char *suffix)
{
- char testtablespace[MAXPGPATH];
char indir[MAXPGPATH];
char outdir_sub[MAXPGPATH];
char **name;
@@ -506,9 +505,6 @@ convert_sourcefiles_in(const char *source_subdir, const char *dest_dir, const ch
if (!directory_exists(outdir_sub))
make_directory(outdir_sub);
- /* We might need to replace @testtablespace@ */
- snprintf(testtablespace, MAXPGPATH, "%s/testtablespace", outputdir);
-
/* finally loop on each file and do the replacement */
for (name = names; *name; name++)
{
@@ -554,7 +550,6 @@ convert_sourcefiles_in(const char *source_subdir, const char *dest_dir, const ch
{
replace_string(&line, "@abs_srcdir@", inputdir);
replace_string(&line, "@abs_builddir@", outputdir);
- replace_string(&line, "@testtablespace@", testtablespace);
replace_string(&line, "@libdir@", dlpath);
replace_string(&line, "@DLSUFFIX@", DLSUFFIX);
fputs(line.data, outfile);
@@ -587,32 +582,6 @@ convert_sourcefiles(void)
convert_sourcefiles_in("output", outputdir, "expected", "out");
}
-/*
- * Clean out the test tablespace dir, or create it if it doesn't exist.
- *
- * On Windows, doing this cleanup here makes it possible to run the
- * regression tests under a Windows administrative user account with the
- * restricted token obtained when starting pg_regress.
- */
-static void
-prepare_testtablespace_dir(void)
-{
- char testtablespace[MAXPGPATH];
-
- snprintf(testtablespace, MAXPGPATH, "%s/testtablespace", outputdir);
-
- if (directory_exists(testtablespace))
- {
- if (!rmtree(testtablespace, true))
- {
- fprintf(stderr, _("\n%s: could not remove test tablespace \"%s\"\n"),
- progname, testtablespace);
- exit(2);
- }
- }
- make_directory(testtablespace);
-}
-
/*
* Scan resultmap file to find which platform-specific expected files to use.
*
@@ -2152,7 +2121,6 @@ help(void)
printf(_(" --launcher=CMD use CMD as launcher of psql\n"));
printf(_(" --load-extension=EXT load the named extension before running the\n"));
printf(_(" tests; can appear multiple times\n"));
- printf(_(" --make-testtablespace-dir create testtablespace directory\n"));
printf(_(" --max-connections=N maximum number of concurrent connections\n"));
printf(_(" (default is 0, meaning unlimited)\n"));
printf(_(" --max-concurrent-tests=N maximum number of concurrent tests in schedule\n"));
@@ -2211,12 +2179,10 @@ regression_main(int argc, char *argv[],
{"load-extension", required_argument, NULL, 22},
{"config-auth", required_argument, NULL, 24},
{"max-concurrent-tests", required_argument, NULL, 25},
- {"make-testtablespace-dir", no_argument, NULL, 26},
{NULL, 0, NULL, 0}
};
bool use_unix_sockets;
- bool make_testtablespace_dir = false;
_stringlist *sl;
int c;
int i;
@@ -2342,9 +2308,6 @@ regression_main(int argc, char *argv[],
case 25:
max_concurrent_tests = atoi(optarg);
break;
- case 26:
- make_testtablespace_dir = true;
- break;
default:
/* getopt_long already emitted a complaint */
fprintf(stderr, _("\nTry \"%s -h\" for more information.\n"),
@@ -2397,9 +2360,6 @@ regression_main(int argc, char *argv[],
unlimit_core_size();
#endif
- if (make_testtablespace_dir)
- prepare_testtablespace_dir();
-
if (temp_instance)
{
FILE *pg_conf;
diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/sql/tablespace.sql
similarity index 96%
rename from src/test/regress/input/tablespace.source
rename to src/test/regress/sql/tablespace.sql
index cb9774ecc8..a42e33ed72 100644
--- a/src/test/regress/input/tablespace.source
+++ b/src/test/regress/sql/tablespace.sql
@@ -1,6 +1,18 @@
+-- relative tablespace locations are not allowed
+CREATE TABLESPACE regress_tblspace LOCATION 'relative'; -- fail
+
+-- empty tablespace locations are not usually allowed
+CREATE TABLESPACE regress_tblspace LOCATION ''; -- fail
+
+-- as a special developer-only option to allow us to use tablespaces
+-- with streaming replication on the same server, an empty location
+-- can be allowed as a way to say that the tablespace should be created
+-- as a directory in pg_tblspc, rather than being a symlink
+SET allow_in_place_tablespaces = true;
+
-- create a tablespace using WITH clause
-CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (some_nonexistent_parameter = true); -- fail
-CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (random_page_cost = 3.0); -- ok
+CREATE TABLESPACE regress_tblspacewith LOCATION '' WITH (some_nonexistent_parameter = true); -- fail
+CREATE TABLESPACE regress_tblspacewith LOCATION '' WITH (random_page_cost = 3.0); -- ok
-- check to see the parameter was used
SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
@@ -9,7 +21,7 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
DROP TABLESPACE regress_tblspacewith;
-- create a tablespace we can use
-CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
+CREATE TABLESPACE regress_tblspace LOCATION '';
-- try setting and resetting some properties for the new tablespace
ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index 5975e7c9cd..bf66083f73 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -133,7 +133,6 @@ sub installcheck_internal
"--bindir=../../../$Config/psql",
"--schedule=${schedule}_schedule",
"--max-concurrent-tests=20",
- "--make-testtablespace-dir",
"--encoding=SQL_ASCII",
"--no-locale");
push(@args, $maxconn) if $maxconn;
@@ -168,7 +167,6 @@ sub check
"--bindir=",
"--schedule=${schedule}_schedule",
"--max-concurrent-tests=20",
- "--make-testtablespace-dir",
"--encoding=SQL_ASCII",
"--no-locale",
"--temp-instance=./tmp_check");
--
2.33.1
v9-0003-Add-new-simple-TAP-test-for-tablespaces.patch (text/x-patch, 2.6K) download
From 0f8e296db9c870f9c59d62dad4cb07f8b7137d05 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Fri, 10 Dec 2021 12:22:55 +1300
Subject: [PATCH v9 3/6] Add new simple TAP test for tablespaces.
The tablespace tests in the main regression tests have been changed to
use "in-place" tablespaces, so that they work when streamed to a replica
on the same host. Add a new TAP test that exercises tablespaces with
absolute paths (but couldn't be replicated), for coverage.
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
.../modules/test_misc/t/002_tablespace.pl | 49 +++++++++++++++++++
1 file changed, 49 insertions(+)
create mode 100644 src/test/modules/test_misc/t/002_tablespace.pl
diff --git a/src/test/modules/test_misc/t/002_tablespace.pl b/src/test/modules/test_misc/t/002_tablespace.pl
new file mode 100644
index 0000000000..3864e5c24b
--- /dev/null
+++ b/src/test/modules/test_misc/t/002_tablespace.pl
@@ -0,0 +1,49 @@
+# Simple tablespace tests that can't be replicated on the same host
+# due to the use of absolute paths, so we keep them out of the regular
+# regression tests.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 6;
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->start;
+
+# Create a directory to hold the tablespace.
+my $LOCATION = $node->basedir() . "/testtablespace";
+mkdir($LOCATION);
+
+my $result;
+
+# Create a tablespace with an absolute path
+$result = $node->psql('postgres',
+ "CREATE TABLESPACE regress_tablespace LOCATION '$LOCATION'");
+ok($result == 0, 'create tablespace with absolute path');
+
+# Can't create a tablespace where there is one already
+$result = $node->psql('postgres',
+ "CREATE TABLESPACE regress_tablespace LOCATION '$LOCATION'");
+ok($result != 0, 'clobber tablespace with absolute path');
+
+# Create table in it
+$result = $node->psql('postgres',
+ "CREATE TABLE t () TABLESPACE regress_tablespace");
+ok($result == 0, 'create table in tablespace with absolute path');
+
+# Can't drop a tablespace that still has a table in it
+$result = $node->psql('postgres',
+ "DROP TABLESPACE regress_tablespace LOCATION '$LOCATION'");
+ok($result != 0, 'drop tablespace with absolute path');
+
+# Drop the table
+$result = $node->psql('postgres', "DROP TABLE t");
+ok($result == 0, 'drop table in tablespace with absolute path');
+
+# Drop the tablespace
+$result = $node->psql('postgres', "DROP TABLESPACE regress_tablespace");
+ok($result == 0, 'drop tablespace with absolute path');
+
+$node->stop;
--
2.33.1
v9-0004-Test-replay-of-regression-tests.patch (text/x-patch, 5.9K) download
From 48d28b222c518330cf6cea262f2b3ab4ee504238 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 5 Oct 2021 21:01:02 +1300
Subject: [PATCH v9 4/6] Test replay of regression tests.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Add a new TAP test under src/test/recovery to run the standard
regression tests while a streaming replica replays the WAL. This
provides a basic workout for WAL decoding and redo code, and compares
the replicated result.
Optionally, enable (expensive) wal_consistency_checking if listed in
the env variable PG_TEST_EXTRA.
Reviewed-by: 綱川 貴之 (Takayuki Tsunakawa) <tsunakawa.takay@fujitsu.com>
Reviewed-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Anastasia Lubennikova <lubennikovaav@gmail.com>
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
doc/src/sgml/regress.sgml | 11 ++++
src/test/recovery/Makefile | 6 +-
src/test/recovery/t/027_stream_regress.pl | 76 +++++++++++++++++++++++
src/tools/msvc/vcregress.pl | 2 +
4 files changed, 94 insertions(+), 1 deletion(-)
create mode 100644 src/test/recovery/t/027_stream_regress.pl
diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml
index 724ef566e7..64fc81776d 100644
--- a/doc/src/sgml/regress.sgml
+++ b/doc/src/sgml/regress.sgml
@@ -289,6 +289,17 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl'
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>wal_consistency_checking</literal></term>
+ <listitem>
+ <para>
+ Uses <literal>wal_consistency_checking=all</literal> while running
+ some of the tests under <filename>src/test/recovery</filename>. Not
+ enabled by default because it is resource intensive.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
Tests for features that are not supported by the current build
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
index 288c04b861..c557018b38 100644
--- a/src/test/recovery/Makefile
+++ b/src/test/recovery/Makefile
@@ -15,10 +15,14 @@ subdir = src/test/recovery
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-# required for 017_shm.pl
+# required for 017_shm.pl and 027_stream_regress.pl
REGRESS_SHLIB=$(abs_top_builddir)/src/test/regress/regress$(DLSUFFIX)
export REGRESS_SHLIB
+# required for 027_stream_regress.pl
+REGRESS_OUTPUTDIR=$(abs_top_builddir)/src/test/recovery
+export REGRESS_OUTPUTDIR
+
check:
$(prove_check)
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
new file mode 100644
index 0000000000..74993cd30e
--- /dev/null
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -0,0 +1,76 @@
+# Run the standard regression tests with streaming replication
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 4;
+use File::Basename;
+
+# Initialize primary node
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+$node_primary->init(allows_streaming => 1);
+$node_primary->adjust_conf('postgresql.conf', 'max_connections', '25', 1);
+
+# WAL consistency checking is resource intensive so require opt-in with the
+# PG_TEST_EXTRA environment variable.
+if ($ENV{PG_TEST_EXTRA} &&
+ $ENV{PG_TEST_EXTRA} =~ m/\bwal_consistency_checking\b/) {
+ $node_primary->append_conf('postgresql.conf',
+ 'wal_consistency_checking = all');
+}
+
+$node_primary->start;
+is( $node_primary->psql(
+ 'postgres',
+ qq[SELECT pg_create_physical_replication_slot('standby_1');]),
+ 0,
+ 'physical slot created on primary');
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_primary->backup($backup_name);
+
+# Create streaming standby linking to primary
+my $node_standby_1 = PostgreSQL::Test::Cluster->new('standby_1');
+$node_standby_1->init_from_backup($node_primary, $backup_name,
+ has_streaming => 1);
+$node_standby_1->append_conf('postgresql.conf',
+ "primary_slot_name = standby_1");
+$node_standby_1->start;
+
+my $dlpath = PostgreSQL::Test::Utils::perl2host(dirname($ENV{REGRESS_SHLIB}));
+my $outputdir = PostgreSQL::Test::Utils::perl2host($ENV{REGRESS_OUTPUTDIR});
+
+# Run the regression tests against the primary.
+system_or_bail($ENV{PG_REGRESS},
+ "--dlpath=" . $dlpath,
+ "--bindir=",
+ "--port=" . $node_primary->port,
+ "--schedule=../regress/parallel_schedule",
+ "--inputdir=../regress",
+ "--outputdir=" . $outputdir);
+
+# Clobber all sequences with their next value, so that we don't have
+# differences between nodes due to caching.
+$node_primary->psql('regression',
+ "select setval(seqrelid, nextval(seqrelid)) from pg_sequence");
+
+# Wait for standby to catch up
+$node_primary->wait_for_catchup($node_standby_1, 'replay',
+ $node_primary->lsn('insert'));
+
+# Perform a logical dump of primary and standby, and check that they match
+command_ok(
+ [ 'pg_dump', '-f', $outputdir . '/primary.dump', '--no-sync',
+ '-p', $node_primary->port, 'regression' ],
+ 'dump primary server');
+command_ok(
+ [ 'pg_dump', '-f', $outputdir . '/standby.dump', '--no-sync',
+ '-p', $node_standby_1->port, 'regression' ],
+ 'dump standby server');
+command_ok(
+ [ 'diff', $outputdir . '/primary.dump', $outputdir . '/standby.dump' ],
+ 'compare primary and standby dumps');
+
+$node_standby_1->stop;
+$node_primary->stop;
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index bf66083f73..d002bdac9d 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -535,6 +535,8 @@ sub recoverycheck
{
InstallTemp();
+ $ENV{REGRESS_OUTPUTDIR} = "$topdir/src/test/recovery";
+
my $mstat = 0;
my $dir = "$topdir/src/test/recovery";
my $status = tap_check($dir);
--
2.33.1
* Re: A test for replay of regression tests
@ 2021-12-10 00:09 Thomas Munro <thomas.munro@gmail.com>
parent: Thomas Munro <thomas.munro@gmail.com>
2 siblings, 0 replies; 82+ messages in thread
From: Thomas Munro @ 2021-12-10 00:09 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Fri, Dec 10, 2021 at 12:58 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> +$node_primary->adjust_conf('postgresql.conf', 'max_connections', '25', 1);
Erm, in fact this requirement came about in an earlier version where I
was invoking make and getting --max-concurrent-tests=20 from
src/test/regress/GNUmakefile. Which I should probably replicate
here...
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-12-11 00:06 Andrew Dunstan <andrew@dunslane.net>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 0 replies; 82+ messages in thread
From: Andrew Dunstan @ 2021-12-11 00:06 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Anastasia Lubennikova <lubennikovaav@gmail.com>; Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On 12/9/21 15:15, Thomas Munro wrote:
> On Fri, Dec 10, 2021 at 2:12 AM Andrew Dunstan <andrew@dunslane.net> wrote:
>> The new version appears to set an empty --bindir for pg_regress. Is that
>> right?
> It seems to be necessary to find eg psql, since --bindir='' means
> "expect $PATH to contain the installed binaries", and that's working
> on both build systems. The alternative would be to export yet another
> environment variable, $PG_INSTALL or such -- do you think that'd be
> better, or did I miss something that exists already like that?
No, that seems ok. Might be worth a comment.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-12-11 00:17 Andres Freund <andres@anarazel.de>
parent: Thomas Munro <thomas.munro@gmail.com>
2 siblings, 0 replies; 82+ messages in thread
From: Andres Freund @ 2021-12-11 00:17 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
Hi,
On 2021-12-10 12:58:01 +1300, Thomas Munro wrote:
> > What's the relation of this to the rest?
>
> Someone decided that allow_streaming should imply max_connections =
> 10, but we need ~20 to run the parallel regression test schedule.
> However, I can just as easily move that to a local adjustment in the
> TAP test file. Done, like so:
>
> +$node_primary->adjust_conf('postgresql.conf', 'max_connections', '25', 1);
Possible that this will cause problem on some *BSD platform with a limited
count of semaphores. But we can deal with that if / when it happens.
> > Separately: I think the case of seeing diffs will be too hard to debug like
> > this, as the difference isn't shown afaict?
>
> Seems to be OK. The output goes to
> src/test/recovery/tmp_check/log/regress_log_027_stream_regress, so for
> example if you comment out the bit that deals with SEQUENCE caching
> you'll see:
Ah, ok. Not sure what I thought there...
> On Fri, Dec 10, 2021 at 10:35 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> > On Fri, Dec 10, 2021 at 8:38 AM Andres Freund <andres@anarazel.de> wrote:
> > > Personally I'd rather put relative tablespaces into a dedicated directory or
> > > just into pg_tblspc, but without a symlink. Some tools need to understand
> > > tablespace layout etc, and having them in a directory that, by the name, will
> > > also contain other things seems likely to cause confusion.
>
> Ok, in this version I have a developer-only GUC
> allow_in_place_tablespaces instead. If you turn it on, you can do:
>
> CREATE TABLESPACE regress_blah LOCATION = '';
> ... and then pg_tblspc/OID is created directly as a directory. Not
> allowed by default or documented.
WFM. I think we might eventually want to allow it by default, but we can deal
with that whenever somebody wants to spend the energy doing so.
> @@ -590,16 +595,35 @@ create_tablespace_directories(const char *location, const Oid tablespaceoid)
> char *linkloc;
> char *location_with_version_dir;
> struct stat st;
> + bool in_place;
>
> linkloc = psprintf("pg_tblspc/%u", tablespaceoid);
> - location_with_version_dir = psprintf("%s/%s", location,
> +
> + /*
> + * If we're asked to make an 'in place' tablespace, create the directory
> + * directly where the symlink would normally go. This is a developer-only
> + * option for now, to facilitate regression testing.
> + */
> + in_place =
> + (allow_in_place_tablespaces || InRecovery) && strlen(location) == 0;
Why is in_place set to true by InRecovery?
ISTM that allow_in_place_tablespaces should be checked in CreateTableSpace(),
and create_tablespace_directories() should just do whatever it's told?
Otherwise it seems there's ample potential for confusion, e.g. because of the
GUC differing between primary and replica, or between crash and a new start.
> + if (in_place)
> + {
> + if (MakePGDirectory(linkloc) < 0 && errno != EEXIST)
> + ereport(ERROR,
> + (errcode_for_file_access(),
> + errmsg("could not create directory \"%s\": %m",
> + linkloc)));
> + }
> +
> + location_with_version_dir = psprintf("%s/%s", in_place ? linkloc : location,
> TABLESPACE_VERSION_DIRECTORY);
>
> /*
> * Attempt to coerce target directory to safe permissions. If this fails,
> * it doesn't exist or has the wrong owner.
> */
> - if (chmod(location, pg_dir_create_mode) != 0)
> + if (!in_place && chmod(location, pg_dir_create_mode) != 0)
> {
> if (errno == ENOENT)
> ereport(ERROR,
Maybe add a coment saying that we don't need to chmod here because
MakePGDirectory() takes care of that?
> @@ -648,13 +672,13 @@ create_tablespace_directories(const char *location, const Oid tablespaceoid)
> /*
> * In recovery, remove old symlink, in case it points to the wrong place.
> */
> - if (InRecovery)
> + if (!in_place && InRecovery)
> remove_tablespace_symlink(linkloc);
I don't think it's right to check !in_place as you do here, given that it
currently depends on a GUC setting that's possibly differs between WAL
generation and replay time?
> --- a/src/test/regress/output/tablespace.source
> +++ b/src/test/regress/expected/tablespace.out
> @@ -1,7 +1,18 @@
> +-- relative tablespace locations are not allowed
> +CREATE TABLESPACE regress_tblspace LOCATION 'relative'; -- fail
> +ERROR: tablespace location must be an absolute path
> +-- empty tablespace locations are not usually allowed
> +CREATE TABLESPACE regress_tblspace LOCATION ''; -- fail
> +ERROR: tablespace location must be an absolute path
> +-- as a special developer-only option to allow us to use tablespaces
> +-- with streaming replication on the same server, an empty location
> +-- can be allowed as a way to say that the tablespace should be created
> +-- as a directory in pg_tblspc, rather than being a symlink
> +SET allow_in_place_tablespaces = true;
> -- create a tablespace using WITH clause
> -CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (some_nonexistent_parameter = true); -- fail
> +CREATE TABLESPACE regress_tblspacewith LOCATION '' WITH (some_nonexistent_parameter = true); -- fail
> ERROR: unrecognized parameter "some_nonexistent_parameter"
> -CREATE TABLESPACE regress_tblspacewith LOCATION '@testtablespace@' WITH (random_page_cost = 3.0); -- ok
> +CREATE TABLESPACE regress_tblspacewith LOCATION '' WITH (random_page_cost = 3.0); -- ok
Perhaps also add a test that we catch "in-place" tablespace creation with
allow_in_place_tablespaces = false? Although perhaps that's better done in the
previous commit...
> +++ b/src/test/modules/test_misc/t/002_tablespace.pl
Two minor things that I think would be worth testing here:
1) moving between two "absolute" tablespaces. That could conceivably break differently
between in-place and relative tablespaces.
2) Moving between absolute and relative tablespace.
> +# required for 027_stream_regress.pl
> +REGRESS_OUTPUTDIR=$(abs_top_builddir)/src/test/recovery
> +export REGRESS_OUTPUTDIR
Why do we need this?
> +# Initialize primary node
> +my $node_primary = PostgreSQL::Test::Cluster->new('primary');
> +$node_primary->init(allows_streaming => 1);
> +$node_primary->adjust_conf('postgresql.conf', 'max_connections', '25', 1);
Probably should set at least max_prepared_transactions > 0, so the tests
requiring prepared xacts can work. They have nontrivial replay routines, so
that seems worthwhile?
> +# Perform a logical dump of primary and standby, and check that they match
> +command_ok(
> + [ 'pg_dump', '-f', $outputdir . '/primary.dump', '--no-sync',
> + '-p', $node_primary->port, 'regression' ],
> + 'dump primary server');
> +command_ok(
> + [ 'pg_dump', '-f', $outputdir . '/standby.dump', '--no-sync',
> + '-p', $node_standby_1->port, 'regression' ],
> + 'dump standby server');
> +command_ok(
> + [ 'diff', $outputdir . '/primary.dump', $outputdir . '/standby.dump' ],
> + 'compare primary and standby dumps');
> +
> +$node_standby_1->stop;
> +$node_primary->stop;
This doesn't verify if global objects are replayed correctly. Perhaps it'd be
better to use pg_dumpall?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-12-15 08:50 Michael Paquier <michael@paquier.xyz>
parent: Thomas Munro <thomas.munro@gmail.com>
2 siblings, 1 reply; 82+ messages in thread
From: Michael Paquier @ 2021-12-15 08:50 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Fri, Dec 10, 2021 at 12:58:01PM +1300, Thomas Munro wrote:
> -# required for 017_shm.pl
> +# required for 017_shm.pl and 027_stream_regress.pl
> REGRESS_SHLIB=$(abs_top_builddir)/src/test/regress/regress$(DLSUFFIX)
> export REGRESS_SHLIB
Hmm. FWIW, I am working on doing similar for pg_upgrade to switch to
TAP there, and we share a lot in terms of running pg_regress on an
exising cluster. Wouldn't it be better to move this definition to
src/Makefile.global.in rather than src/test/recovery/?
My pg_regress command is actually very similar to yours, so I am
wondering if this would be better if somehow centralized, perhaps in
Cluster.pm.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-12-16 00:22 Michael Paquier <michael@paquier.xyz>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 82+ messages in thread
From: Michael Paquier @ 2021-12-16 00:22 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Wed, Dec 15, 2021 at 05:50:45PM +0900, Michael Paquier wrote:
> Hmm. FWIW, I am working on doing similar for pg_upgrade to switch to
> TAP there, and we share a lot in terms of running pg_regress on an
> exising cluster. Wouldn't it be better to move this definition to
> src/Makefile.global.in rather than src/test/recovery/?
>
> My pg_regress command is actually very similar to yours, so I am
> wondering if this would be better if somehow centralized, perhaps in
> Cluster.pm.
By the way, while I was sorting out my things, I have noticed that v4
does not handle EXTRA_REGRESS_OPT. Is that wanted? You could just
add that into your patch set and push the extra options to the
pg_regress command:
my $extra_opts_val = $ENV{EXTRA_REGRESS_OPT} || "";
my @extra_opts = split(/\s+/, $extra_opts_val);
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2021-12-21 22:41 Thomas Munro <thomas.munro@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2021-12-21 22:41 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Andres Freund <andres@anarazel.de>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
Hi,
Rebased and updated based on feedback. Responses to multiple emails below:
On Thu, Dec 16, 2021 at 1:22 PM Michael Paquier <michael@paquier.xyz> wrote:
> On Wed, Dec 15, 2021 at 05:50:45PM +0900, Michael Paquier wrote:
> > Hmm. FWIW, I am working on doing similar for pg_upgrade to switch to
> > TAP there, and we share a lot in terms of running pg_regress on an
> > exising cluster. Wouldn't it be better to move this definition to
> > src/Makefile.global.in rather than src/test/recovery/?
> >
> > My pg_regress command is actually very similar to yours, so I am
> > wondering if this would be better if somehow centralized, perhaps in
> > Cluster.pm.
Thanks for looking. Right, it sounds like you'll have the same
problems I ran into. I haven't updated this patch for that yet, as
I'm not sure exactly what you need and we could easily move it in a
later commit. Does that seem reasonable?
> By the way, while I was sorting out my things, I have noticed that v4
> does not handle EXTRA_REGRESS_OPT. Is that wanted? You could just
> add that into your patch set and push the extra options to the
> pg_regress command:
> my $extra_opts_val = $ENV{EXTRA_REGRESS_OPT} || "";
> my @extra_opts = split(/\s+/, $extra_opts_val);
Seems like a good idea for consistency, but isn't that a variable
that's supposed to be expanded by a shell, not naively split on
whitespace? Perhaps we should use the single-argument variant of
system(), so the whole escaped enchilada is passed to a shell? Tried
like that in this version (though now I'm wondering what the correct
perl incantation is to shell-escape $outputdir and $dlpath...)
On Sat, Dec 11, 2021 at 1:17 PM Andres Freund <andres@anarazel.de> wrote:
> On 2021-12-10 12:58:01 +1300, Thomas Munro wrote:
> > +$node_primary->adjust_conf('postgresql.conf', 'max_connections', '25', 1);
>
> Possible that this will cause problem on some *BSD platform with a limited
> count of semaphores. But we can deal with that if / when it happens.
Right, those systems don't work out of the box for us already without
sysctl tweaks, so it doesn't matter if animals have to be adjusted
further.
> > @@ -590,16 +595,35 @@ create_tablespace_directories(const char *location, const Oid tablespaceoid)
> > char *linkloc;
> > char *location_with_version_dir;
> > struct stat st;
> > + bool in_place;
> >
> > linkloc = psprintf("pg_tblspc/%u", tablespaceoid);
> > - location_with_version_dir = psprintf("%s/%s", location,
> > +
> > + /*
> > + * If we're asked to make an 'in place' tablespace, create the directory
> > + * directly where the symlink would normally go. This is a developer-only
> > + * option for now, to facilitate regression testing.
> > + */
> > + in_place =
> > + (allow_in_place_tablespaces || InRecovery) && strlen(location) == 0;
>
> Why is in_place set to true by InRecovery?
Well the real condition is strlen(location) == 0, and the other part
is a sort of bit belt-and-braces check, but yeah, I should just remove
that part. Done.
> ISTM that allow_in_place_tablespaces should be checked in CreateTableSpace(),
> and create_tablespace_directories() should just do whatever it's told?
> Otherwise it seems there's ample potential for confusion, e.g. because of the
> GUC differing between primary and replica, or between crash and a new start.
Agreed, that was the effect but the extra unnecessary check was a bit confusing.
> > /*
> > * Attempt to coerce target directory to safe permissions. If this fails,
> > * it doesn't exist or has the wrong owner.
> > */
> > - if (chmod(location, pg_dir_create_mode) != 0)
> > + if (!in_place && chmod(location, pg_dir_create_mode) != 0)
> > {
> > if (errno == ENOENT)
> > ereport(ERROR,
>
> Maybe add a coment saying that we don't need to chmod here because
> MakePGDirectory() takes care of that?
Done.
> > @@ -648,13 +672,13 @@ create_tablespace_directories(const char *location, const Oid tablespaceoid)
> > /*
> > * In recovery, remove old symlink, in case it points to the wrong place.
> > */
> > - if (InRecovery)
> > + if (!in_place && InRecovery)
> > remove_tablespace_symlink(linkloc);
>
> I don't think it's right to check !in_place as you do here, given that it
> currently depends on a GUC setting that's possibly differs between WAL
> generation and replay time?
I have to, because otherwise we'll remove the directory we just
created at the top of the function. It doesn't really depend on a GUC
(clearer after previous change).
> Perhaps also add a test that we catch "in-place" tablespace creation with
> allow_in_place_tablespaces = false? Although perhaps that's better done in the
> previous commit...
There was a test for that already, see this bit:
+-- empty tablespace locations are not usually allowed
+CREATE TABLESPACE regress_tblspace LOCATION ''; -- fail
+ERROR: tablespace location must be an absolute path
> > +++ b/src/test/modules/test_misc/t/002_tablespace.pl
>
> Two minor things that I think would be worth testing here:
> 1) moving between two "absolute" tablespaces. That could conceivably break differently
> between in-place and relative tablespaces.
> 2) Moving between absolute and relative tablespace.
Done.
> > +# required for 027_stream_regress.pl
> > +REGRESS_OUTPUTDIR=$(abs_top_builddir)/src/test/recovery
> > +export REGRESS_OUTPUTDIR
>
> Why do we need this?
The Make macro "prove_check" (src/Makefile.global.in) always changes
to the source directory to run TAP tests. Without an explicit
directive to control where regression test output goes, it got
splattered all over the source tree in VPATH builds. I didn't see an
existing way to adjust that (did I miss something?). Hence desire to
pass down a path in the build tree. Better ideas welcome.
> > +# Initialize primary node
> > +my $node_primary = PostgreSQL::Test::Cluster->new('primary');
> > +$node_primary->init(allows_streaming => 1);
> > +$node_primary->adjust_conf('postgresql.conf', 'max_connections', '25', 1);
>
> Probably should set at least max_prepared_transactions > 0, so the tests
> requiring prepared xacts can work. They have nontrivial replay routines, so
> that seems worthwhile?
Good idea. Done.
> > +# Perform a logical dump of primary and standby, and check that they match
> > +command_ok(
> > + [ 'pg_dump', '-f', $outputdir . '/primary.dump', '--no-sync',
> > + '-p', $node_primary->port, 'regression' ],
> > + 'dump primary server');
> > +command_ok(
> > + [ 'pg_dump', '-f', $outputdir . '/standby.dump', '--no-sync',
> > + '-p', $node_standby_1->port, 'regression' ],
> > + 'dump standby server');
> > +command_ok(
> > + [ 'diff', $outputdir . '/primary.dump', $outputdir . '/standby.dump' ],
> > + 'compare primary and standby dumps');
> > +
> > +$node_standby_1->stop;
> > +$node_primary->stop;
>
> This doesn't verify if global objects are replayed correctly. Perhaps it'd be
> better to use pg_dumpall?
Good idea. Done.
Attachments: download all (4 files, .tar.gz)
v10-0001-Allow-in-place-tablespaces.patch (text/x-patch, 5.4K) download
From 6a05a305f3e4c4e5b9bb368f3e18d502e779a3b8 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Fri, 10 Dec 2021 11:45:24 +1300
Subject: [PATCH v10 1/6] Allow "in place" tablespaces.
Provide a developer-only GUC allow_in_place_tablespaces, disabled by
default. When enabled, tablespaces can be created with an empty
LOCATION string, meaning that they should be created as a directory
directly beneath pg_tblspc. This can be used for new testing scenarios,
in a follow-up patch. Not intended for end-user usage, since it might
confuse backup tools that expect symlinks.
Reviewed-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
src/backend/commands/tablespace.c | 39 +++++++++++++++++++++++++------
src/backend/utils/misc/guc.c | 12 ++++++++++
src/include/commands/tablespace.h | 2 ++
3 files changed, 46 insertions(+), 7 deletions(-)
diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c
index 4b96eec9df..c198501cbf 100644
--- a/src/backend/commands/tablespace.c
+++ b/src/backend/commands/tablespace.c
@@ -87,6 +87,7 @@
/* GUC variables */
char *default_tablespace = NULL;
char *temp_tablespaces = NULL;
+bool allow_in_place_tablespaces = false;
static void create_tablespace_directories(const char *location,
@@ -241,6 +242,7 @@ CreateTableSpace(CreateTableSpaceStmt *stmt)
char *location;
Oid ownerId;
Datum newOptions;
+ bool in_place;
/* Must be superuser */
if (!superuser())
@@ -266,12 +268,15 @@ CreateTableSpace(CreateTableSpaceStmt *stmt)
(errcode(ERRCODE_INVALID_NAME),
errmsg("tablespace location cannot contain single quotes")));
+ in_place = allow_in_place_tablespaces && strlen(location) == 0;
+
/*
* Allowing relative paths seems risky
*
- * this also helps us ensure that location is not empty or whitespace
+ * This also helps us ensure that location is not empty or whitespace,
+ * unless specifying a developer-only in-place tablespace.
*/
- if (!is_absolute_path(location))
+ if (!in_place && !is_absolute_path(location))
ereport(ERROR,
(errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
errmsg("tablespace location must be an absolute path")));
@@ -590,16 +595,36 @@ create_tablespace_directories(const char *location, const Oid tablespaceoid)
char *linkloc;
char *location_with_version_dir;
struct stat st;
+ bool in_place;
linkloc = psprintf("pg_tblspc/%u", tablespaceoid);
- location_with_version_dir = psprintf("%s/%s", location,
+
+ /*
+ * If we're asked to make an 'in place' tablespace, create the directory
+ * directly where the symlink would normally go. This is a developer-only
+ * option for now, to facilitate regression testing.
+ */
+ in_place = strlen(location) == 0;
+
+ if (in_place)
+ {
+ if (MakePGDirectory(linkloc) < 0 && errno != EEXIST)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create directory \"%s\": %m",
+ linkloc)));
+ }
+
+ location_with_version_dir = psprintf("%s/%s", in_place ? linkloc : location,
TABLESPACE_VERSION_DIRECTORY);
/*
* Attempt to coerce target directory to safe permissions. If this fails,
- * it doesn't exist or has the wrong owner.
+ * it doesn't exist or has the wrong owner. Not needed for in-place mode,
+ * because in that case we created the directory with the desired
+ * permissions.
*/
- if (chmod(location, pg_dir_create_mode) != 0)
+ if (!in_place && chmod(location, pg_dir_create_mode) != 0)
{
if (errno == ENOENT)
ereport(ERROR,
@@ -648,13 +673,13 @@ create_tablespace_directories(const char *location, const Oid tablespaceoid)
/*
* In recovery, remove old symlink, in case it points to the wrong place.
*/
- if (InRecovery)
+ if (!in_place && InRecovery)
remove_tablespace_symlink(linkloc);
/*
* Create the symlink under PGDATA
*/
- if (symlink(location, linkloc) < 0)
+ if (!in_place && symlink(location, linkloc) < 0)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not create symbolic link \"%s\": %m",
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index bff949a40b..ea655bd98e 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -46,6 +46,7 @@
#include "catalog/storage.h"
#include "commands/async.h"
#include "commands/prepare.h"
+#include "commands/tablespace.h"
#include "commands/trigger.h"
#include "commands/user.h"
#include "commands/vacuum.h"
@@ -1963,6 +1964,17 @@ static struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
+ {
+ {"allow_in_place_tablespaces", PGC_SUSET, DEVELOPER_OPTIONS,
+ gettext_noop("Allows tablespaces directly inside pg_tblspc, for testing."),
+ NULL,
+ GUC_NOT_IN_SAMPLE
+ },
+ &allow_in_place_tablespaces,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"lo_compat_privileges", PGC_SUSET, COMPAT_OPTIONS_PREVIOUS,
gettext_noop("Enables backward compatibility mode for privilege checks on large objects."),
diff --git a/src/include/commands/tablespace.h b/src/include/commands/tablespace.h
index 49cfc8479a..f33ba51bd6 100644
--- a/src/include/commands/tablespace.h
+++ b/src/include/commands/tablespace.h
@@ -19,6 +19,8 @@
#include "lib/stringinfo.h"
#include "nodes/parsenodes.h"
+extern bool allow_in_place_tablespaces;
+
/* XLOG stuff */
#define XLOG_TBLSPC_CREATE 0x00
#define XLOG_TBLSPC_DROP 0x10
--
2.30.2
v10-0002-Use-in-place-tablespaces-in-regression-test.patch (text/x-patch, 8.8K) download
From b84478cd1db090cd308514a8db322ea0dc54cbb4 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Wed, 4 Aug 2021 22:17:54 +1200
Subject: [PATCH v10 2/6] Use in-place tablespaces in regression test.
Remove the machinery from pg_regress that manages the testtablespace
directory. Instead, use "in-place" tablespaces, because they work
correctly when there is a streaming replica running on the same host.
Reviewed-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
src/test/regress/GNUmakefile | 2 +-
src/test/regress/expected/tablespace.out | 20 ++++++++++----
src/test/regress/pg_regress.c | 35 ------------------------
src/test/regress/sql/tablespace.sql | 19 +++++++++----
src/tools/msvc/vcregress.pl | 2 --
5 files changed, 28 insertions(+), 50 deletions(-)
diff --git a/src/test/regress/GNUmakefile b/src/test/regress/GNUmakefile
index 330eca2b83..31b1c49100 100644
--- a/src/test/regress/GNUmakefile
+++ b/src/test/regress/GNUmakefile
@@ -112,7 +112,7 @@ submake-contrib-spi: | submake-libpgport submake-generated-headers
## Run tests
##
-REGRESS_OPTS = --dlpath=. --max-concurrent-tests=20 --make-testtablespace-dir \
+REGRESS_OPTS = --dlpath=. --max-concurrent-tests=20 \
$(EXTRA_REGRESS_OPTS)
check: all
diff --git a/src/test/regress/expected/tablespace.out b/src/test/regress/expected/tablespace.out
index 864f4b6e20..2dfbcfdebe 100644
--- a/src/test/regress/expected/tablespace.out
+++ b/src/test/regress/expected/tablespace.out
@@ -1,10 +1,18 @@
--- directory paths are passed to us in environment variables
-\getenv abs_builddir PG_ABS_BUILDDIR
-\set testtablespace :abs_builddir '/testtablespace'
+-- relative tablespace locations are not allowed
+CREATE TABLESPACE regress_tblspace LOCATION 'relative'; -- fail
+ERROR: tablespace location must be an absolute path
+-- empty tablespace locations are not usually allowed
+CREATE TABLESPACE regress_tblspace LOCATION ''; -- fail
+ERROR: tablespace location must be an absolute path
+-- as a special developer-only option to allow us to use tablespaces
+-- with streaming replication on the same server, an empty location
+-- can be allowed as a way to say that the tablespace should be created
+-- as a directory in pg_tblspc, rather than being a symlink
+SET allow_in_place_tablespaces = true;
-- create a tablespace using WITH clause
-CREATE TABLESPACE regress_tblspacewith LOCATION :'testtablespace' WITH (some_nonexistent_parameter = true); -- fail
+CREATE TABLESPACE regress_tblspacewith LOCATION '' WITH (some_nonexistent_parameter = true); -- fail
ERROR: unrecognized parameter "some_nonexistent_parameter"
-CREATE TABLESPACE regress_tblspacewith LOCATION :'testtablespace' WITH (random_page_cost = 3.0); -- ok
+CREATE TABLESPACE regress_tblspacewith LOCATION '' WITH (random_page_cost = 3.0); -- ok
-- check to see the parameter was used
SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
spcoptions
@@ -15,7 +23,7 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
-- drop the tablespace so we can re-use the location
DROP TABLESPACE regress_tblspacewith;
-- create a tablespace we can use
-CREATE TABLESPACE regress_tblspace LOCATION :'testtablespace';
+CREATE TABLESPACE regress_tblspace LOCATION '';
-- try setting and resetting some properties for the new tablespace
ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true); -- fail
diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index 589357ba59..d64b02d4ad 100644
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -438,32 +438,6 @@ string_matches_pattern(const char *str, const char *pattern)
return false;
}
-/*
- * Clean out the test tablespace dir, or create it if it doesn't exist.
- *
- * On Windows, doing this cleanup here makes it possible to run the
- * regression tests under a Windows administrative user account with the
- * restricted token obtained when starting pg_regress.
- */
-static void
-prepare_testtablespace_dir(void)
-{
- char testtablespace[MAXPGPATH];
-
- snprintf(testtablespace, MAXPGPATH, "%s/testtablespace", outputdir);
-
- if (directory_exists(testtablespace))
- {
- if (!rmtree(testtablespace, true))
- {
- fprintf(stderr, _("\n%s: could not remove test tablespace \"%s\"\n"),
- progname, testtablespace);
- exit(2);
- }
- }
- make_directory(testtablespace);
-}
-
/*
* Scan resultmap file to find which platform-specific expected files to use.
*
@@ -2010,7 +1984,6 @@ help(void)
printf(_(" --launcher=CMD use CMD as launcher of psql\n"));
printf(_(" --load-extension=EXT load the named extension before running the\n"));
printf(_(" tests; can appear multiple times\n"));
- printf(_(" --make-testtablespace-dir create testtablespace directory\n"));
printf(_(" --max-connections=N maximum number of concurrent connections\n"));
printf(_(" (default is 0, meaning unlimited)\n"));
printf(_(" --max-concurrent-tests=N maximum number of concurrent tests in schedule\n"));
@@ -2069,12 +2042,10 @@ regression_main(int argc, char *argv[],
{"load-extension", required_argument, NULL, 22},
{"config-auth", required_argument, NULL, 24},
{"max-concurrent-tests", required_argument, NULL, 25},
- {"make-testtablespace-dir", no_argument, NULL, 26},
{NULL, 0, NULL, 0}
};
bool use_unix_sockets;
- bool make_testtablespace_dir = false;
_stringlist *sl;
int c;
int i;
@@ -2200,9 +2171,6 @@ regression_main(int argc, char *argv[],
case 25:
max_concurrent_tests = atoi(optarg);
break;
- case 26:
- make_testtablespace_dir = true;
- break;
default:
/* getopt_long already emitted a complaint */
fprintf(stderr, _("\nTry \"%s -h\" for more information.\n"),
@@ -2255,9 +2223,6 @@ regression_main(int argc, char *argv[],
unlimit_core_size();
#endif
- if (make_testtablespace_dir)
- prepare_testtablespace_dir();
-
if (temp_instance)
{
FILE *pg_conf;
diff --git a/src/test/regress/sql/tablespace.sql b/src/test/regress/sql/tablespace.sql
index 92076db9a1..896f05cea3 100644
--- a/src/test/regress/sql/tablespace.sql
+++ b/src/test/regress/sql/tablespace.sql
@@ -1,11 +1,18 @@
--- directory paths are passed to us in environment variables
-\getenv abs_builddir PG_ABS_BUILDDIR
+-- relative tablespace locations are not allowed
+CREATE TABLESPACE regress_tblspace LOCATION 'relative'; -- fail
-\set testtablespace :abs_builddir '/testtablespace'
+-- empty tablespace locations are not usually allowed
+CREATE TABLESPACE regress_tblspace LOCATION ''; -- fail
+
+-- as a special developer-only option to allow us to use tablespaces
+-- with streaming replication on the same server, an empty location
+-- can be allowed as a way to say that the tablespace should be created
+-- as a directory in pg_tblspc, rather than being a symlink
+SET allow_in_place_tablespaces = true;
-- create a tablespace using WITH clause
-CREATE TABLESPACE regress_tblspacewith LOCATION :'testtablespace' WITH (some_nonexistent_parameter = true); -- fail
-CREATE TABLESPACE regress_tblspacewith LOCATION :'testtablespace' WITH (random_page_cost = 3.0); -- ok
+CREATE TABLESPACE regress_tblspacewith LOCATION '' WITH (some_nonexistent_parameter = true); -- fail
+CREATE TABLESPACE regress_tblspacewith LOCATION '' WITH (random_page_cost = 3.0); -- ok
-- check to see the parameter was used
SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
@@ -14,7 +21,7 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith';
DROP TABLESPACE regress_tblspacewith;
-- create a tablespace we can use
-CREATE TABLESPACE regress_tblspace LOCATION :'testtablespace';
+CREATE TABLESPACE regress_tblspace LOCATION '';
-- try setting and resetting some properties for the new tablespace
ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1);
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index 29086cab51..fc7679e1d1 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -133,7 +133,6 @@ sub installcheck_internal
"--bindir=../../../$Config/psql",
"--schedule=${schedule}_schedule",
"--max-concurrent-tests=20",
- "--make-testtablespace-dir",
"--encoding=SQL_ASCII",
"--no-locale");
push(@args, $maxconn) if $maxconn;
@@ -168,7 +167,6 @@ sub check
"--bindir=",
"--schedule=${schedule}_schedule",
"--max-concurrent-tests=20",
- "--make-testtablespace-dir",
"--encoding=SQL_ASCII",
"--no-locale",
"--temp-instance=./tmp_check");
--
2.30.2
v10-0003-Add-new-simple-TAP-test-for-tablespaces.patch (text/x-patch, 4.8K) download
From 64e768eb3b438684ef5a4e836b1ef73712abd14e Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Fri, 10 Dec 2021 12:22:55 +1300
Subject: [PATCH v10 3/6] Add new simple TAP test for tablespaces.
The tablespace tests in the main regression tests have been changed to
use "in-place" tablespaces, so that they work when streamed to a replica
on the same host. Add a new TAP test that exercises tablespaces with
absolute paths (but couldn't be replicated), for coverage.
Reviewed-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
.../modules/test_misc/t/002_tablespace.pl | 96 +++++++++++++++++++
1 file changed, 96 insertions(+)
create mode 100644 src/test/modules/test_misc/t/002_tablespace.pl
diff --git a/src/test/modules/test_misc/t/002_tablespace.pl b/src/test/modules/test_misc/t/002_tablespace.pl
new file mode 100644
index 0000000000..5f1be368a7
--- /dev/null
+++ b/src/test/modules/test_misc/t/002_tablespace.pl
@@ -0,0 +1,96 @@
+# Simple tablespace tests that can't be replicated on the same host
+# due to the use of absolute paths, so we keep them out of the regular
+# regression tests.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 20;
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->start;
+
+# Create a couple of directories to use as tablespaces.
+my $TS1_LOCATION = $node->basedir() . "/ts1";
+mkdir($TS1_LOCATION);
+my $TS2_LOCATION = $node->basedir() . "/ts2";
+mkdir($TS2_LOCATION);
+
+my $result;
+
+# Create a tablespace with an absolute path
+$result = $node->psql('postgres',
+ "CREATE TABLESPACE regress_ts1 LOCATION '$TS1_LOCATION'");
+ok($result == 0, 'create tablespace with absolute path');
+
+# Can't create a tablespace where there is one already
+$result = $node->psql('postgres',
+ "CREATE TABLESPACE regress_ts1 LOCATION '$TS1_LOCATION'");
+ok($result != 0, 'clobber tablespace with absolute path');
+
+# Create table in it
+$result = $node->psql('postgres',
+ "CREATE TABLE t () TABLESPACE regress_ts1");
+ok($result == 0, 'create table in tablespace with absolute path');
+
+# Can't drop a tablespace that still has a table in it
+$result = $node->psql('postgres',
+ "DROP TABLESPACE regress_ts1");
+ok($result != 0, 'drop tablespace with absolute path');
+
+# Drop the table
+$result = $node->psql('postgres', "DROP TABLE t");
+ok($result == 0, 'drop table in tablespace with absolute path');
+
+# Drop the tablespace
+$result = $node->psql('postgres', "DROP TABLESPACE regress_ts1");
+ok($result == 0, 'drop tablespace with absolute path');
+
+# Create two absolute tablespaces and two in-place tablespaces, so we can
+# testing various kinds of tablespace moves.
+$result = $node->psql('postgres',
+ "CREATE TABLESPACE regress_ts1 LOCATION '$TS1_LOCATION'");
+ok($result == 0, 'create tablespace 1 with absolute path');
+$result = $node->psql('postgres',
+ "CREATE TABLESPACE regress_ts2 LOCATION '$TS2_LOCATION'");
+ok($result == 0, 'create tablespace 2 with absolute path');
+$result = $node->psql('postgres',
+ "SET allow_in_place_tablespaces=on; CREATE TABLESPACE regress_ts3 LOCATION ''");
+ok($result == 0, 'create tablespace 3 with in-place directory');
+$result = $node->psql('postgres',
+ "SET allow_in_place_tablespaces=on; CREATE TABLESPACE regress_ts4 LOCATION ''");
+ok($result == 0, 'create tablespace 4 with in-place directory');
+
+# Create a table and test moving between absolute and in-place tablespaces
+$result = $node->psql('postgres',
+ "CREATE TABLE t () TABLESPACE regress_ts1");
+ok($result == 0, 'create table in tablespace 1');
+$result = $node->psql('postgres',
+ "ALTER TABLE t SET tablespace regress_ts2");
+ok($result == 0, 'move table abs->abs');
+$result = $node->psql('postgres',
+ "ALTER TABLE t SET tablespace regress_ts3");
+ok($result == 0, 'move table abs->in-place');
+$result = $node->psql('postgres',
+ "ALTER TABLE t SET tablespace regress_ts4");
+ok($result == 0, 'move table in-place->in-place');
+$result = $node->psql('postgres',
+ "ALTER TABLE t SET tablespace regress_ts1");
+ok($result == 0, 'move table in-place->abs');
+
+# Drop everything
+$result = $node->psql('postgres',
+ "DROP TABLE t");
+ok($result == 0, 'create table in tablespace 1');
+$result = $node->psql('postgres', "DROP TABLESPACE regress_ts1");
+ok($result == 0, 'drop tablespace 1');
+$result = $node->psql('postgres', "DROP TABLESPACE regress_ts2");
+ok($result == 0, 'drop tablespace 2');
+$result = $node->psql('postgres', "DROP TABLESPACE regress_ts3");
+ok($result == 0, 'drop tablespace 3');
+$result = $node->psql('postgres', "DROP TABLESPACE regress_ts4");
+ok($result == 0, 'drop tablespace 4');
+
+$node->stop;
--
2.30.2
v10-0004-Test-replay-of-regression-tests.patch (text/x-patch, 6.1K) download
From c810b1dece30790f07fe5c0b996e34c78c2b43d1 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 5 Oct 2021 21:01:02 +1300
Subject: [PATCH v10 4/6] Test replay of regression tests.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Add a new TAP test under src/test/recovery to run the standard
regression tests while a streaming replica replays the WAL. This
provides a basic workout for WAL decoding and redo code, and compares
the replicated result.
Optionally, enable (expensive) wal_consistency_checking if listed in
the env variable PG_TEST_EXTRA.
Reviewed-by: 綱川 貴之 (Takayuki Tsunakawa) <tsunakawa.takay@fujitsu.com>
Reviewed-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Anastasia Lubennikova <lubennikovaav@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/CA%2BhUKGKpRWQ9SxdxxDmTBCJoR0YnFpMBe7kyzY8SUQk%2BHeskxg%40mail.gmail.com
---
doc/src/sgml/regress.sgml | 11 ++++
src/test/recovery/Makefile | 6 +-
src/test/recovery/t/027_stream_regress.pl | 80 +++++++++++++++++++++++
src/tools/msvc/vcregress.pl | 2 +
4 files changed, 98 insertions(+), 1 deletion(-)
create mode 100644 src/test/recovery/t/027_stream_regress.pl
diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml
index 724ef566e7..64fc81776d 100644
--- a/doc/src/sgml/regress.sgml
+++ b/doc/src/sgml/regress.sgml
@@ -289,6 +289,17 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl'
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>wal_consistency_checking</literal></term>
+ <listitem>
+ <para>
+ Uses <literal>wal_consistency_checking=all</literal> while running
+ some of the tests under <filename>src/test/recovery</filename>. Not
+ enabled by default because it is resource intensive.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
Tests for features that are not supported by the current build
diff --git a/src/test/recovery/Makefile b/src/test/recovery/Makefile
index 288c04b861..c557018b38 100644
--- a/src/test/recovery/Makefile
+++ b/src/test/recovery/Makefile
@@ -15,10 +15,14 @@ subdir = src/test/recovery
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-# required for 017_shm.pl
+# required for 017_shm.pl and 027_stream_regress.pl
REGRESS_SHLIB=$(abs_top_builddir)/src/test/regress/regress$(DLSUFFIX)
export REGRESS_SHLIB
+# required for 027_stream_regress.pl
+REGRESS_OUTPUTDIR=$(abs_top_builddir)/src/test/recovery
+export REGRESS_OUTPUTDIR
+
check:
$(prove_check)
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
new file mode 100644
index 0000000000..6619298694
--- /dev/null
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -0,0 +1,80 @@
+# Run the standard regression tests with streaming replication
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 4;
+use File::Basename;
+
+# Initialize primary node
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+$node_primary->init(allows_streaming => 1);
+$node_primary->adjust_conf('postgresql.conf', 'max_connections', '25', 1);
+$node_primary->append_conf('postgresql.conf', 'max_prepared_transactions = 10');
+
+# WAL consistency checking is resource intensive so require opt-in with the
+# PG_TEST_EXTRA environment variable.
+if ($ENV{PG_TEST_EXTRA} &&
+ $ENV{PG_TEST_EXTRA} =~ m/\bwal_consistency_checking\b/) {
+ $node_primary->append_conf('postgresql.conf',
+ 'wal_consistency_checking = all');
+}
+
+$node_primary->start;
+is( $node_primary->psql(
+ 'postgres',
+ qq[SELECT pg_create_physical_replication_slot('standby_1');]),
+ 0,
+ 'physical slot created on primary');
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_primary->backup($backup_name);
+
+# Create streaming standby linking to primary
+my $node_standby_1 = PostgreSQL::Test::Cluster->new('standby_1');
+$node_standby_1->init_from_backup($node_primary, $backup_name,
+ has_streaming => 1);
+$node_standby_1->append_conf('postgresql.conf',
+ "primary_slot_name = standby_1");
+$node_standby_1->start;
+
+my $dlpath = PostgreSQL::Test::Utils::perl2host(dirname($ENV{REGRESS_SHLIB}));
+my $outputdir = PostgreSQL::Test::Utils::perl2host($ENV{REGRESS_OUTPUTDIR});
+
+# Run the regression tests against the primary.
+my $extra_opts = $ENV{EXTRA_REGRESS_OPTS} || "";
+system_or_bail($ENV{PG_REGRESS} . " " .
+ "--dlpath=$dlpath " .
+ "--bindir= " .
+ "--port=" . $node_primary->port . " " .
+ "--schedule=../regress/parallel_schedule " .
+ "--max-concurrent-tests=20 " .
+ "--inputdir=../regress " .
+ "--outputdir=$outputdir " .
+ $extra_opts);
+
+# Clobber all sequences with their next value, so that we don't have
+# differences between nodes due to caching.
+$node_primary->psql('regression',
+ "select setval(seqrelid, nextval(seqrelid)) from pg_sequence");
+
+# Wait for standby to catch up
+$node_primary->wait_for_catchup($node_standby_1, 'replay',
+ $node_primary->lsn('insert'));
+
+# Perform a logical dump of primary and standby, and check that they match
+command_ok(
+ [ 'pg_dumpall', '-f', $outputdir . '/primary.dump', '--no-sync',
+ '-p', $node_primary->port ],
+ 'dump primary server');
+command_ok(
+ [ 'pg_dumpall', '-f', $outputdir . '/standby.dump', '--no-sync',
+ '-p', $node_standby_1->port ],
+ 'dump standby server');
+command_ok(
+ [ 'diff', $outputdir . '/primary.dump', $outputdir . '/standby.dump' ],
+ 'compare primary and standby dumps');
+
+$node_standby_1->stop;
+$node_primary->stop;
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index fc7679e1d1..73d411dcf5 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -535,6 +535,8 @@ sub recoverycheck
{
InstallTemp();
+ $ENV{REGRESS_OUTPUTDIR} = "$topdir/src/test/recovery";
+
my $mstat = 0;
my $dir = "$topdir/src/test/recovery";
my $status = tap_check($dir);
--
2.30.2
* Re: A test for replay of regression tests
@ 2022-01-14 11:39 Thomas Munro <thomas.munro@gmail.com>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2022-01-14 11:39 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Andres Freund <andres@anarazel.de>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Wed, Dec 22, 2021 at 11:41 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> Rebased and updated based on feedback. Responses to multiple emails below:
Pushed, but the build farm doesn't like it with a couple of different
ways of failing. I'll collect some results and revert shortly.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-14 13:32 Thomas Munro <thomas.munro@gmail.com>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2022-01-14 13:32 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Andres Freund <andres@anarazel.de>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Sat, Jan 15, 2022 at 12:39 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Wed, Dec 22, 2021 at 11:41 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> > Rebased and updated based on feedback. Responses to multiple emails below:
>
> Pushed, but the build farm doesn't like it with a couple of different
> ways of failing. I'll collect some results and revert shortly.
Problems:
1. The way I invoke pg_regress doesn't seem to work correctly under
the build farm client (though it works fine under make), not sure why
yet, but reproduced here and will figure it out tomorrow.
2. The new test in src/test/modules/t/002_tablespace.pl apparently has
some path-related problem on Windows that I didn't know about, because
CI isn't even running the TAP tests under src/test/module/test_misc
(and various other locations), but the BF is :-/ And I was happy
because modulescheck was passing...
I reverted the two commits responsible for those failures to keep the
build farm green, and I'll try to fix them tomorrow.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-14 23:49 Andres Freund <andres@anarazel.de>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Andres Freund @ 2022-01-14 23:49 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
Hi,
On 2022-01-15 02:32:35 +1300, Thomas Munro wrote:
> 1. The way I invoke pg_regress doesn't seem to work correctly under
> the build farm client (though it works fine under make), not sure why
> yet, but reproduced here and will figure it out tomorrow.
I think it's just a problem of the buildfarm specifying port names in
extra_opts. E.g.
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=eelpout&dt=2022-01-14%2011%3A49%3A36
has
# Checking port 58074
# Found port 58074
Name: primary
...
# Running: /home/tmunro/build-farm/buildroot/HEAD/pgsql.build/src/test/recovery/../../../src/test/regress/pg_regress --dlpath="/home/tmunro/build-farm/buildroot/HEAD/pgsql.build/src/test/regress" --bindir= --port=58074 --schedule=../regress/parallel_schedule --max-concurrent-tests=20 --inputdir=../regress --outputdir="/home/tmunro/build-farm/buildroot/HEAD/pgsql.build/src/test/recovery" --port=5678
(using postmaster on /tmp/1W6qVPVyCv, port 5678)
Note how there's both --port=58074 and --port=5678 in the pg_regress
invocation. The latter comming from EXTRA_REGRESS_OPTS, which the buildfarm
client sets.
The quickest fix would probably be to just move the 027_stream_regress.pl
added --port until after $extra_opts?
> 2. The new test in src/test/modules/t/002_tablespace.pl apparently has
> some path-related problem on Windows
This is the failure you're talking about?
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2022-01-14%2012%3A04%3A55
> that I didn't know about, because CI isn't even running the TAP tests under
> src/test/module/test_misc (and various other locations), but the BF is :-/
> And I was happy because modulescheck was passing...
This we need to fix... But if you're talking about fairywren's failure, it's
more than not running some tests, it's that we do not test windows mingw
outside of cross compilation.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-17 04:25 Thomas Munro <thomas.munro@gmail.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 2 replies; 82+ messages in thread
From: Thomas Munro @ 2022-01-17 04:25 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Michael Paquier <michael@paquier.xyz>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Sat, Jan 15, 2022 at 12:49 PM Andres Freund <andres@anarazel.de> wrote:
> On 2022-01-15 02:32:35 +1300, Thomas Munro wrote:
> > 1. The way I invoke pg_regress doesn't seem to work correctly under
> > the build farm client (though it works fine under make), not sure why
> > yet, but reproduced here and will figure it out tomorrow.
>
> I think it's just a problem of the buildfarm specifying port names in
> extra_opts. E.g.
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=eelpout&dt=2022-01-14%2011%3A49%3A36
> has
>
> # Checking port 58074
> # Found port 58074
> Name: primary
> ...
> # Running: /home/tmunro/build-farm/buildroot/HEAD/pgsql.build/src/test/recovery/../../../src/test/regress/pg_regress --dlpath="/home/tmunro/build-farm/buildroot/HEAD/pgsql.build/src/test/regress" --bindir= --port=58074 --schedule=../regress/parallel_schedule --max-concurrent-tests=20 --inputdir=../regress --outputdir="/home/tmunro/build-farm/buildroot/HEAD/pgsql.build/src/test/recovery" --port=5678
> (using postmaster on /tmp/1W6qVPVyCv, port 5678)
>
> Note how there's both --port=58074 and --port=5678 in the pg_regress
> invocation. The latter comming from EXTRA_REGRESS_OPTS, which the buildfarm
> client sets.
>
> The quickest fix would probably be to just move the 027_stream_regress.pl
> added --port until after $extra_opts?
Thanks, I figured it was an environment variable biting me, and indeed
it was that one. I reordered the arguments, tested locally under the
buildfarm client script, and pushed. I'll keep an eye on the build
farm.
One thing I noticed is that the pg_dump output files should really be
rm'd by the clean target; I may push something for that later.
> > 2. The new test in src/test/modules/t/002_tablespace.pl apparently has
> > some path-related problem on Windows
>
> This is the failure you're talking about?
>
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2022-01-14%2012%3A04%3A55
>
> > that I didn't know about, because CI isn't even running the TAP tests under
> > src/test/module/test_misc (and various other locations), but the BF is :-/
> > And I was happy because modulescheck was passing...
>
> This we need to fix... But if you're talking about fairywren's failure, it's
> more than not running some tests, it's that we do not test windows mingw
> outside of cross compilation.
I'm temporarily stumped by complete ignorance of MSYS. I tried the
test on plain old Windows/MSVC by cherry-picking the reverted commit
d1511fe1 and running .\src\tools\msvc\vcregress.bat taptest
.\src\test\modules\test_misc in my Windows 10 VM, and that passed with
flying colours (so Windows CI would have passed too, if we weren't
ignoring TAP tests in unusual locations, I assume). I'll look into
installing MSYS to work this out if necessary, but it may take me a
few days.
Here's how it failed on fairywren, in case someone knowledgeable of
MSYS path translation etc can spot the problem:
psql:<stdin>:1: ERROR: directory
"/home/pgrunner/bf/root/HEAD/pgsql.build/src/test/modules/test_misc/tmp_check/t_002_tablespace_main_data/ts1"
does not exist
not ok 1 - create tablespace with absolute path
I think that means chmod() failed with ENOENT. That's weird, because
the .pl does:
+my $TS1_LOCATION = $node->basedir() . "/ts1";
+mkdir($TS1_LOCATION);
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-17 05:53 Noah Misch <noah@leadboat.com>
parent: Thomas Munro <thomas.munro@gmail.com>
1 sibling, 1 reply; 82+ messages in thread
From: Noah Misch @ 2022-01-17 05:53 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Michael Paquier <michael@paquier.xyz>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Mon, Jan 17, 2022 at 05:25:19PM +1300, Thomas Munro wrote:
> Here's how it failed on fairywren, in case someone knowledgeable of
> MSYS path translation etc can spot the problem:
>
> psql:<stdin>:1: ERROR: directory
> "/home/pgrunner/bf/root/HEAD/pgsql.build/src/test/modules/test_misc/tmp_check/t_002_tablespace_main_data/ts1"
> does not exist
> not ok 1 - create tablespace with absolute path
>
> I think that means chmod() failed with ENOENT. That's weird, because
> the .pl does:
>
> +my $TS1_LOCATION = $node->basedir() . "/ts1";
> +mkdir($TS1_LOCATION);
You likely need a PostgreSQL::Test::Utils::perl2host() call. MSYS Perl
understands Cygwin-style names like /home/... as well as Windows-style names,
but this PostgreSQL configuration understands only Windows-style names.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-21 02:42 Thomas Munro <thomas.munro@gmail.com>
parent: Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2022-01-21 02:42 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Andres Freund <andres@anarazel.de>; Michael Paquier <michael@paquier.xyz>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Mon, Jan 17, 2022 at 6:53 PM Noah Misch <noah@leadboat.com> wrote:
> On Mon, Jan 17, 2022 at 05:25:19PM +1300, Thomas Munro wrote:
> > Here's how it failed on fairywren, in case someone knowledgeable of
> > MSYS path translation etc can spot the problem:
> You likely need a PostgreSQL::Test::Utils::perl2host() call. MSYS Perl
> understands Cygwin-style names like /home/... as well as Windows-style names,
> but this PostgreSQL configuration understands only Windows-style names.
Thanks. I added that and pushed. Let's see if fairywren likes it
when it comes back online.
I also learned that in the CI environment, node->basedir() is a path
containing an internal "." component (I mean "something/./something").
I added a regex to collapse those, because they're unacceptable in
Windows junction point targets. I'm aware that there is something
happening in another CF entry that might address that sort of
thing[1], so then perhaps I could remove the kludge.
I tested that with a throw-away change to .cirrus.yml, like so. The
CI thread[2] is discussing a proper solution to these Windows CI blind
spots.
test_modules_script:
- perl src/tools/msvc/vcregress.pl modulescheck
+ - perl src/tools/msvc/vcregress.pl taptest ./src/test/modules/test_misc
[1] https://commitfest.postgresql.org/36/3331/
[2] https://www.postgresql.org/message-id/20220114235457.GQ14051%40telsasoft.com
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-21 18:58 Thomas Munro <thomas.munro@gmail.com>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2022-01-21 18:58 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Andres Freund <andres@anarazel.de>; Michael Paquier <michael@paquier.xyz>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Fri, Jan 21, 2022 at 3:42 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> Thanks. I added that and pushed. Let's see if fairywren likes it
> when it comes back online.
A watched pot never boils, but I wonder why Andrew's 4 Windows
configurations jacana, bowerbird, fairywren and drongo have stopped
returning results.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-21 19:48 Andres Freund <andres@anarazel.de>
parent: Thomas Munro <thomas.munro@gmail.com>
1 sibling, 2 replies; 82+ messages in thread
From: Andres Freund @ 2022-01-21 19:48 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
Hi,
On 2022-01-17 17:25:19 +1300, Thomas Munro wrote:
> I reordered the arguments, tested locally under the buildfarm client script,
> and pushed. I'll keep an eye on the build farm.
After the reloptions fix the tests seem much more likely to succeed than
before. Progress!
Unfortunately we don't quite seem there yet:
I saw a couple failures like:
https://api.cirrus-ci.com/v1/artifact/task/5394938773897216/regress_diffs/build/testrun/recovery/t/0...
(from https://cirrus-ci.com/task/5394938773897216?logs=check_world#L183 )
-- Should succeed
DROP TABLESPACE regress_tblspace_renamed;
+ERROR: tablespace "regress_tblspace_renamed" is not empty
I assume the reason we see this semi-regularly when the regression tests run
as part of 027_stream_regress, but not in the main regression test run, is
similar to the reloptions problem, namely that we run with a much smaller
shared buffers.
I assume what happens is that this just makes the known problem of bgwriter or
some other process keeping open a filehandle to an already deleted relation,
preventing the deletion to "fully" take effect, worse.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-21 20:07 Thomas Munro <thomas.munro@gmail.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 82+ messages in thread
From: Thomas Munro @ 2022-01-21 20:07 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Michael Paquier <michael@paquier.xyz>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Sat, Jan 22, 2022 at 8:48 AM Andres Freund <andres@anarazel.de> wrote:
> Unfortunately we don't quite seem there yet:
>
> I saw a couple failures like:
> https://api.cirrus-ci.com/v1/artifact/task/5394938773897216/regress_diffs/build/testrun/recovery/t/0...
> (from https://cirrus-ci.com/task/5394938773897216?logs=check_world#L183 )
>
> -- Should succeed
> DROP TABLESPACE regress_tblspace_renamed;
> +ERROR: tablespace "regress_tblspace_renamed" is not empty
>
>
> I assume the reason we see this semi-regularly when the regression tests run
> as part of 027_stream_regress, but not in the main regression test run, is
> similar to the reloptions problem, namely that we run with a much smaller
> shared buffers.
>
> I assume what happens is that this just makes the known problem of bgwriter or
> some other process keeping open a filehandle to an already deleted relation,
> preventing the deletion to "fully" take effect, worse.
Right, I assume this would be fixed by [1]. I need to re-convince
myself of that patch's correctness and make some changes after
Robert's feedback; I'll look into committing it next week. From a
certain point of view it's now quite good that we hit this case
occasionally in CI.
[1] https://commitfest.postgresql.org/36/2962/
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-21 21:22 Andrew Dunstan <andrew@dunslane.net>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Andrew Dunstan @ 2022-01-21 21:22 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; Noah Misch <noah@leadboat.com>; +Cc: Andres Freund <andres@anarazel.de>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On 1/21/22 13:58, Thomas Munro wrote:
> On Fri, Jan 21, 2022 at 3:42 PM Thomas Munro <thomas.munro@gmail.com> wrote:
>> Thanks. I added that and pushed. Let's see if fairywren likes it
>> when it comes back online.
> A watched pot never boils, but I wonder why Andrew's 4 Windows
> configurations jacana, bowerbird, fairywren and drongo have stopped
> returning results.
I think I have unstuck both machines. I will keep an eye on them and
make sure they don't get stuck again.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-22 05:00 Thomas Munro <thomas.munro@gmail.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2022-01-22 05:00 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Michael Paquier <michael@paquier.xyz>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Sat, Jan 22, 2022 at 8:48 AM Andres Freund <andres@anarazel.de> wrote:
> Unfortunately we don't quite seem there yet:
And another way to fail:
pg_dump: error: query failed: ERROR: canceling statement due to
conflict with recovery
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=dangomushi&dt=2022-01-22%2003%3A06%3A42
Probably needs hot_standby_feedback on. Will adjust this soon.
One more failure seen in today's crop was a "stats" failure on
seawasp, which must be the well known pre-existing problem. (Probably
just needs someone to rewrite the stats subsystem to use shared memory
instead of UDP).
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-27 20:27 Andrew Dunstan <andrew@dunslane.net>
parent: Andrew Dunstan <andrew@dunslane.net>
0 siblings, 2 replies; 82+ messages in thread
From: Andrew Dunstan @ 2022-01-27 20:27 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; Noah Misch <noah@leadboat.com>; +Cc: Andres Freund <andres@anarazel.de>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On 1/21/22 16:22, Andrew Dunstan wrote:
> On 1/21/22 13:58, Thomas Munro wrote:
>> On Fri, Jan 21, 2022 at 3:42 PM Thomas Munro <thomas.munro@gmail.com> wrote:
>>> Thanks. I added that and pushed. Let's see if fairywren likes it
>>> when it comes back online.
>> A watched pot never boils, but I wonder why Andrew's 4 Windows
>> configurations jacana, bowerbird, fairywren and drongo have stopped
>> returning results.
>
>
> I think I have unstuck both machines. I will keep an eye on them and
> make sure they don't get stuck again.
>
>
fairywren is not happy with the recovery tests still.
I have noticed on a different setup that this test adds 11 minutes to
the runtime of the recovery tests, effectively doubling it. The doubling
is roughly true on faster setups, too. At least I would like a simple
way to disable the test.
I'm not very happy about that.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-27 20:47 Andres Freund <andres@anarazel.de>
parent: Andrew Dunstan <andrew@dunslane.net>
1 sibling, 3 replies; 82+ messages in thread
From: Andres Freund @ 2022-01-27 20:47 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Noah Misch <noah@leadboat.com>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
Hi,
On 2022-01-27 15:27:17 -0500, Andrew Dunstan wrote:
> fairywren is not happy with the recovery tests still.
Any more details?
> I have noticed on a different setup that this test adds 11 minutes to the
> runtime of the recovery tests, effectively doubling it. The doubling is
> roughly true on faster setups, too
Does a normal regress run take roughly that long? Or is the problem that the
027_stream_regress.pl ends up defaulting to shared_buffers=1MB, causing lots
of unnecessary IO?
> . At least I would like a simple
> way to disable the test.
One thing we could do to speed up the overall runtime would be to move
027_stream_regress.pl to something numbered earlier. Combined with
PROVE_FLAGS=-j2 that could at least run them in parallel with the rest of the
recovery tests.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-27 21:41 Thomas Munro <thomas.munro@gmail.com>
parent: Andrew Dunstan <andrew@dunslane.net>
1 sibling, 0 replies; 82+ messages in thread
From: Thomas Munro @ 2022-01-27 21:41 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Noah Misch <noah@leadboat.com>; Andres Freund <andres@anarazel.de>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Fri, Jan 28, 2022 at 9:27 AM Andrew Dunstan <andrew@dunslane.net> wrote:
> I have noticed on a different setup that this test adds 11 minutes to
> the runtime of the recovery tests, effectively doubling it. The doubling
> is roughly true on faster setups, too. At least I would like a simple
> way to disable the test.
Ouch, that's ... a lot. Some randomly selected times: ~20 seconds
(dev machines), ~40 seconds (Cirrus CI's Windows image), ~2-3 minutes
(very cheap cloud host accounts), ~3 minutes (my Rapsberry Pi pinned
onto two CPU cores), ~11 minutes (your Windows number). It would be
good to understand why that's such an outlier.
Re skipping, I've also been wondering about an exclusion list to skip
parts of the regression tests that don't really add recovery coverage
but take non-trivial time, like the join tests.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-27 22:03 Andres Freund <andres@anarazel.de>
parent: Andres Freund <andres@anarazel.de>
2 siblings, 2 replies; 82+ messages in thread
From: Andres Freund @ 2022-01-27 22:03 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Noah Misch <noah@leadboat.com>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
Hi,
On 2022-01-27 12:47:08 -0800, Andres Freund wrote:
> > I have noticed on a different setup that this test adds 11 minutes to the
> > runtime of the recovery tests, effectively doubling it. The doubling is
> > roughly true on faster setups, too
>
> Does a normal regress run take roughly that long? Or is the problem that the
> 027_stream_regress.pl ends up defaulting to shared_buffers=1MB, causing lots
> of unnecessary IO?
In my msys install a normal regress run takes 57s, 027_stream_regress.pl takes
194s.
It's *not* shared_buffers. Or any of the other postgresql.conf settings. As
far as I can tell.
< tries a bunch of things >
ARGH. It's the utterly broken handling of refused connections on windows. The
pg_regress invocation doesn't specify the host address, just the port.
Now you might reasonably ask, why does that slow things down so much, rather
than working or not working? The problem is that a tcp connect() on windows
doesn't immediately fail when a connection establishment is rejected, but
instead internally retries several times. Which takes 2s. The reason there
are rejected connections without specifying the host is that Cluster.pm
configures to listen to 127.0.0.1. But the default for libpq/psql is to try
"localhost". Which name resolution returns first as ipv6 (i.e. ::1). Which
takes 2s to fail, upon which libpq goes and tries 127.0.0.1, which works.
That means every single psql started by 027_stream_regress.pl's pg_regress
takes 2s. Which of course adds up...
I'll go and sob in a corner.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-27 22:07 Andres Freund <andres@anarazel.de>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 82+ messages in thread
From: Andres Freund @ 2022-01-27 22:07 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Noah Misch <noah@leadboat.com>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On 2022-01-27 14:03:51 -0800, Andres Freund wrote:
> In my msys install a normal regress run takes 57s, 027_stream_regress.pl takes
> 194s.
>
> That means every single psql started by 027_stream_regress.pl's pg_regress
> takes 2s. Which of course adds up...
Oh, forgot: After adding --host to the pg_regress invocation
027_stream_regress.pl takes 75s (from 194s before).
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-27 22:16 Andrew Dunstan <andrew@dunslane.net>
parent: Andres Freund <andres@anarazel.de>
2 siblings, 1 reply; 82+ messages in thread
From: Andrew Dunstan @ 2022-01-27 22:16 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Noah Misch <noah@leadboat.com>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On 1/27/22 15:47, Andres Freund wrote:
> Hi,
>
> On 2022-01-27 15:27:17 -0500, Andrew Dunstan wrote:
>> fairywren is not happy with the recovery tests still.
> Any more details?
I'll go back and get some.
>
>
>> I have noticed on a different setup that this test adds 11 minutes to the
>> runtime of the recovery tests, effectively doubling it. The doubling is
>> roughly true on faster setups, too
> Does a normal regress run take roughly that long? Or is the problem that the
> 027_stream_regress.pl ends up defaulting to shared_buffers=1MB, causing lots
> of unnecessary IO?
On crake (slowish fedora 34), a normal check run took 95s, and this test
took 114s. On my windows test instance where I noticed this (w10,
msys2/ucrt), check took 516s and this test took 685s.
>
>
>> . At least I would like a simple
>> way to disable the test.
> One thing we could do to speed up the overall runtime would be to move
> 027_stream_regress.pl to something numbered earlier. Combined with
> PROVE_FLAGS=-j2 that could at least run them in parallel with the rest of the
> recovery tests.
>
>
Seems like a bandaid.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-27 22:21 Thomas Munro <thomas.munro@gmail.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 82+ messages in thread
From: Thomas Munro @ 2022-01-27 22:21 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Noah Misch <noah@leadboat.com>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Fri, Jan 28, 2022 at 11:03 AM Andres Freund <andres@anarazel.de> wrote:
> That means every single psql started by 027_stream_regress.pl's pg_regress
> takes 2s. Which of course adds up...
That is very surprising, thanks. Will fix.
I've been experimenting with reusing psql sessions and backends for
queries in TAP tests, since some Windows animals seem to take a
significant fraction of a second *per query* due to forking and
startup costs. ~100ms or whatever is nothing compared to that ~2000ms
silliness, but it still adds up over thousands of queries. I'll post
an experimental patch soon, but this discussion has given me the idea
that pg_regress might ideally be able to reuse processes too, at least
sometimes...
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-27 22:36 Andres Freund <andres@anarazel.de>
parent: Andrew Dunstan <andrew@dunslane.net>
0 siblings, 1 reply; 82+ messages in thread
From: Andres Freund @ 2022-01-27 22:36 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Noah Misch <noah@leadboat.com>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
Hi,
On 2022-01-27 17:16:17 -0500, Andrew Dunstan wrote:
> On crake (slowish fedora 34), a normal check run took 95s, and this test
> took 114s.
That's roughly what I see on msys after the fix.
> On my windows test instance where I noticed this (w10,
> msys2/ucrt), check took 516s and this test took 685s.
Hm. That's both excruciatingly slow. Way way slower than what I see here, also
w10, msys2/ucrt. Any chance the test instance has windows defender running,
without a directory exclusion? I saw that trash performance to a near
standstill.
Does it get better with the attached patch?
I was confused why this didn't fail fatally on CI, which uses
PG_TEST_USE_UNIX_SOCKETS. I think he reason is that pg_regress' use of PGHOST
is busted, btw. It says it'll use PGHOST if --host isn't specified, but it
doesn't work.
* When testing an existing install, we honor existing environment
* variables, except if they're overridden by command line options.
*/
if (hostname != NULL)
{
setenv("PGHOST", hostname, 1);
unsetenv("PGHOSTADDR");
}
but hostname is initialized in the existing-install case:
#if !defined(HAVE_UNIX_SOCKETS)
use_unix_sockets = false;
#elif defined(WIN32)
/*
* We don't use Unix-domain sockets on Windows by default, even if the
* build supports them. (See comment at remove_temp() for a reason.)
* Override at your own risk.
*/
use_unix_sockets = getenv("PG_TEST_USE_UNIX_SOCKETS") ? true : false;
#else
use_unix_sockets = true;
#endif
if (!use_unix_sockets)
hostname = "localhost";
Greetings,
Andres Freund
Attachments:
pghost.diff (text/x-diff, 591B) download
diff --git i/src/test/recovery/t/027_stream_regress.pl w/src/test/recovery/t/027_stream_regress.pl
index 0596980b1a6..782f24b429f 100644
--- i/src/test/recovery/t/027_stream_regress.pl
+++ w/src/test/recovery/t/027_stream_regress.pl
@@ -58,6 +58,7 @@ my $extra_opts = $ENV{EXTRA_REGRESS_OPTS} || "";
system_or_bail($ENV{PG_REGRESS} . " $extra_opts " .
"--dlpath=\"$dlpath\" " .
"--bindir= " .
+ "--host=" . $node_primary->host . " " .
"--port=" . $node_primary->port . " " .
"--schedule=../regress/parallel_schedule " .
"--max-concurrent-tests=20 " .
* Re: A test for replay of regression tests
@ 2022-01-27 22:51 Andrew Dunstan <andrew@dunslane.net>
parent: Andres Freund <andres@anarazel.de>
2 siblings, 1 reply; 82+ messages in thread
From: Andrew Dunstan @ 2022-01-27 22:51 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Noah Misch <noah@leadboat.com>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On 1/27/22 15:47, Andres Freund wrote:
> Hi,
>
> On 2022-01-27 15:27:17 -0500, Andrew Dunstan wrote:
>> fairywren is not happy with the recovery tests still.
> Any more details?
(Not actually fairywren, but equivalent) It's hung at
src/test/recovery/t/009_twophase.pl line 84:
$psql_rc = $cur_primary->psql('postgres', "COMMIT PREPARED
'xact_009_1'");
This is an Amazon EC2 WS2019 instance, of type t3.large i.e. 8Gb of
memory (not the same machine I reported test times from). Perhaps I need
to test on another instance. Note though that when I tested with a
ucrt64 build, including use of the ucrt64 perl/prove, the recovery test
passed on an equivalent instance, so that's probably another reason to
switch fairywren to using the ucrt64 environment.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-27 22:59 Andres Freund <andres@anarazel.de>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 82+ messages in thread
From: Andres Freund @ 2022-01-27 22:59 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Noah Misch <noah@leadboat.com>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
Hi,
On 2022-01-27 14:36:32 -0800, Andres Freund wrote:
> > On my windows test instance where I noticed this (w10,
> > msys2/ucrt), check took 516s and this test took 685s.
>
> Hm. That's both excruciatingly slow. Way way slower than what I see here, also
> w10, msys2/ucrt. Any chance the test instance has windows defender running,
> without a directory exclusion? I saw that trash performance to a near
> standstill.
Could you post the regression test output with the timings? Unless it's AV, I
don't see why a windows VM with a moderate amount of memory should take that
long.
Do the test times get less bad if you use PG_TEST_USE_UNIX_SOCKETS=1
PG_REGRESS_SOCK_DIR: "c:/some-dir/"?
I see there's reports that the connection-timeout problem can be a lot worse
on windows, because several applications, e.g. docker, add additional names
for localhost. Are there any non-commented entries in
C:\Windows\System32\drivers\etc\hosts
> Does it get better with the attached patch?
I pushed something like it now - seemed to be no reason to wait, given it
makes think less slow on my VM.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-27 23:03 Andres Freund <andres@anarazel.de>
parent: Andrew Dunstan <andrew@dunslane.net>
0 siblings, 1 reply; 82+ messages in thread
From: Andres Freund @ 2022-01-27 23:03 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Noah Misch <noah@leadboat.com>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
Hi,
On 2022-01-27 17:51:52 -0500, Andrew Dunstan wrote:
> (Not actually fairywren, but equivalent) It's hung at
> src/test/recovery/t/009_twophase.pl line 84:
>
>
> $psql_rc = $cur_primary->psql('postgres', "COMMIT PREPARED
> 'xact_009_1'");
That very likely is the socket-shutdown bug that lead to:
commit 64b2c6507e5714b5c688b9c5cc551fbedb7b3b58
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date: 2022-01-25 12:17:40 -0500
Revert "graceful shutdown" changes for Windows, in back branches only.
This reverts commits 6051857fc and ed52c3707, but only in the back
branches. Further testing has shown that while those changes do fix
some things, they also break others; in particular, it looks like
walreceivers fail to detect walsender-initiated connection close
reliably if the walsender shuts down this way. We'll keep trying to
improve matters in HEAD, but it now seems unwise to push these changes
into stable releases.
Discussion: https://postgr.es/m/CA+hUKG+OeoETZQ=Qw5Ub5h3tmwQhBmDA=nuNO3KG=zWfUypFAw@mail.gmail.com
If you apply that commit, does the problem go away?
That's why I'd suggested to revert them in
https://postgr.es/m/20220125023609.5ohu3nslxgoygihl%40alap3.anarazel.de
> This is an Amazon EC2 WS2019 instance, of type t3.large i.e. 8Gb of
> memory (not the same machine I reported test times from). Perhaps I need
> to test on another instance. Note though that when I tested with a
> ucrt64 build, including use of the ucrt64 perl/prove, the recovery test
> passed on an equivalent instance, so that's probably another reason to
> switch fairywren to using the ucrt64 environment.
Without the revert I do get through the tests some of the time - imo likely
that the hang isn't related to the specific msys/mingw environment.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-27 23:24 Thomas Munro <thomas.munro@gmail.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2022-01-27 23:24 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Noah Misch <noah@leadboat.com>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Fri, Jan 28, 2022 at 12:03 PM Andres Freund <andres@anarazel.de> wrote:
> Revert "graceful shutdown" changes for Windows, in back branches only.
FTR I'm actively working on a fix for that one for master now (see
that other thread where the POC survived Alexander's torture testing).
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-01-28 14:46 Andrew Dunstan <andrew@dunslane.net>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 0 replies; 82+ messages in thread
From: Andrew Dunstan @ 2022-01-28 14:46 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; Andres Freund <andres@anarazel.de>; +Cc: Noah Misch <noah@leadboat.com>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On 1/27/22 18:24, Thomas Munro wrote:
> On Fri, Jan 28, 2022 at 12:03 PM Andres Freund <andres@anarazel.de> wrote:
>> Revert "graceful shutdown" changes for Windows, in back branches only.
> FTR I'm actively working on a fix for that one for master now (see
> that other thread where the POC survived Alexander's torture testing).
OK, good. A further data point on that: I am not seeing a recovery test
hang or commit_ts test failure on real W10 machines, including jacana. I
am only getting them on WS2019 VMs e.g. drongo/fairywren.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-02-02 00:59 Thomas Munro <thomas.munro@gmail.com>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 2 replies; 82+ messages in thread
From: Thomas Munro @ 2022-02-02 00:59 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Michael Paquier <michael@paquier.xyz>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Sat, Jan 22, 2022 at 6:00 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Sat, Jan 22, 2022 at 8:48 AM Andres Freund <andres@anarazel.de> wrote:
> > Unfortunately we don't quite seem there yet:
>
> And another way to fail:
>
> pg_dump: error: query failed: ERROR: canceling statement due to
> conflict with recovery
>
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=dangomushi&dt=2022-01-22%2003%3A06%3A42
>
> Probably needs hot_standby_feedback on. Will adjust this soon.
Seen again today on prairiedog. Erm, scratch that idea, HS feedback
interferes with test results. I guess max_standby_streaming_delay
should be increased to 'forever', like in the attached, since pg_dump
runs for a very long time on prairiedog:
2022-02-01 04:47:59.294 EST [3670:15] 027_stream_regress.pl LOG:
statement: SET TRANSACTION ISOLATION LEVEL REPEATABLE READ, READ ONLY
...
2022-02-01 04:49:09.881 EST [3683:2585] 027_stream_regress.pl ERROR:
canceling statement due to conflict with recovery
Attachments:
0001-Fix-recovery-conflict-in-027_stream_regress.pl.patch (text/x-patch, 1.8K) download
From 472062f1c38449e700db87db4dbd31b88190bad2 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Wed, 2 Feb 2022 13:40:15 +1300
Subject: [PATCH] Fix recovery conflict in 027_stream_regress.pl.
To avoid "ERROR: canceling statement due to conflict with recovery"
seen on a couple of slower build farm animals, adjust
max_standby_streaming_delay to wait as long as necessary to avoid
conflicts.
In passing, adjust a configuration option that accidentally used a
non-standard format (not a problem, but needlessly inconsistent).
Discussion: https://postgr.es/m/CA%2BhUKGK65xVqNgsSPyrr2LEwtfUN%3DGfEuQ868hTC-mu0bFG42A%40mail.gmail.com
---
src/test/recovery/t/027_stream_regress.pl | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index 782f24b429..ec214daaab 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -20,7 +20,7 @@ else
# Initialize primary node
my $node_primary = PostgreSQL::Test::Cluster->new('primary');
$node_primary->init(allows_streaming => 1);
-$node_primary->adjust_conf('postgresql.conf', 'max_connections', '25', 1);
+$node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
$node_primary->append_conf('postgresql.conf', 'max_prepared_transactions = 10');
# WAL consistency checking is resource intensive so require opt-in with the
@@ -48,6 +48,8 @@ $node_standby_1->init_from_backup($node_primary, $backup_name,
has_streaming => 1);
$node_standby_1->append_conf('postgresql.conf',
"primary_slot_name = standby_1");
+$node_standby_1->append_conf('postgresql.conf',
+ 'max_standby_streaming_delay = -1');
$node_standby_1->start;
my $dlpath = PostgreSQL::Test::Utils::perl2host(dirname($ENV{REGRESS_SHLIB}));
--
2.33.1
* Re: A test for replay of regression tests
@ 2022-02-02 01:14 Andres Freund <andres@anarazel.de>
parent: Thomas Munro <thomas.munro@gmail.com>
1 sibling, 1 reply; 82+ messages in thread
From: Andres Freund @ 2022-02-02 01:14 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
Hi,
On 2022-02-02 13:59:56 +1300, Thomas Munro wrote:
> Seen again today on prairiedog. Erm, scratch that idea, HS feedback
> interferes with test results.
It'd not be sufficient anyway, I think. E.g. autovacuum truncating a table
would not be prevented by hs_f I think?
> I guess max_standby_streaming_delay
> should be increased to 'forever', like in the attached
Seems reasonable.
> , since pg_dump runs for a very long time on prairiedog:
> 2022-02-01 04:47:59.294 EST [3670:15] 027_stream_regress.pl LOG:
> statement: SET TRANSACTION ISOLATION LEVEL REPEATABLE READ, READ ONLY
> ...
> 2022-02-01 04:49:09.881 EST [3683:2585] 027_stream_regress.pl ERROR:
> canceling statement due to conflict with recovery
That, uh, seems slow. Is it perhaps waiting for a lock? Seems
Cluster.pm::init() should add at least log_lock_waits...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-02-02 01:35 Thomas Munro <thomas.munro@gmail.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2022-02-02 01:35 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Michael Paquier <michael@paquier.xyz>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Wed, Feb 2, 2022 at 2:14 PM Andres Freund <andres@anarazel.de> wrote:
> On 2022-02-02 13:59:56 +1300, Thomas Munro wrote:
> > 2022-02-01 04:47:59.294 EST [3670:15] 027_stream_regress.pl LOG:
> > statement: SET TRANSACTION ISOLATION LEVEL REPEATABLE READ, READ ONLY
> > ...
> > 2022-02-01 04:49:09.881 EST [3683:2585] 027_stream_regress.pl ERROR:
> > canceling statement due to conflict with recovery
>
> That, uh, seems slow. Is it perhaps waiting for a lock? Seems
> Cluster.pm::init() should add at least log_lock_waits...
I quoted the wrong lines, let me try that again this time for the same
session, the one with pid 3683:
2022-02-01 04:48:38.352 EST [3683:15] 027_stream_regress.pl LOG:
statement: SET TRANSACTION ISOLATION LEVEL REPEATABLE READ, READ ONLY
...
2022-02-01 04:49:09.881 EST [3683:2585] 027_stream_regress.pl ERROR:
canceling statement due to conflict with recovery
It looks like it's processing statements fairly consistently slowly
through the whole period. Each non-trivial statement takes a bit
under ~10ms, so it would make sense if by the time we've processed
~2.5k lines we've clocked up 30 seconds and a VACUUM replay whacks us.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-02-02 02:11 Tom Lane <tgl@sss.pgh.pa.us>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Tom Lane @ 2022-02-02 02:11 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Michael Paquier <michael@paquier.xyz>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; pgsql-hackers
Thomas Munro <thomas.munro@gmail.com> writes:
> It looks like it's processing statements fairly consistently slowly
> through the whole period. Each non-trivial statement takes a bit
> under ~10ms, so it would make sense if by the time we've processed
> ~2.5k lines we've clocked up 30 seconds and a VACUUM replay whacks us.
This test is set up to time out after 30 seconds? We've long had
an unofficial baseline that no timeouts under 180 seconds should
be used in the buildfarm.
regards, tom lane
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-02-02 02:16 Andres Freund <andres@anarazel.de>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 82+ messages in thread
From: Andres Freund @ 2022-02-02 02:16 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; Thomas Munro <thomas.munro@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; pgsql-hackers
Hi,
On February 1, 2022 6:11:24 PM PST, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>Thomas Munro <thomas.munro@gmail.com> writes:
>> It looks like it's processing statements fairly consistently slowly
>> through the whole period. Each non-trivial statement takes a bit
>> under ~10ms, so it would make sense if by the time we've processed
>> ~2.5k lines we've clocked up 30 seconds and a VACUUM replay whacks us.
>
>This test is set up to time out after 30 seconds? We've long had
>an unofficial baseline that no timeouts under 180 seconds should
>be used in the buildfarm.
30s is the default value of the streaming replay conflict timeout. After that the startup process cancelled the session running pg_dump. So it's not an intentional timeout in the test.
It's not surprising that pg_dump takes 30s on that old a machine. But more than 2min still surprised me. Is that really do be expected?
- Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-02-02 02:33 Tom Lane <tgl@sss.pgh.pa.us>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 82+ messages in thread
From: Tom Lane @ 2022-02-02 02:33 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Michael Paquier <michael@paquier.xyz>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; pgsql-hackers
Andres Freund <andres@anarazel.de> writes:
> It's not surprising that pg_dump takes 30s on that old a machine. But more than 2min still surprised me. Is that really do be expected?
In the previous buildfarm run, that dump took just under 31s:
2022-01-31 14:21:10.358 EST [19325:1] [unknown] LOG: connection received: host=[local]
2022-01-31 14:21:10.367 EST [19325:2] [unknown] LOG: connection authorized: user=buildfarm database=regression application_name=027_stream_regress.pl
...
2022-01-31 14:21:41.139 EST [19325:2663] 027_stream_regress.pl LOG: disconnection: session time: 0:00:30.782 user=buildfarm database=regression host=[local]
In the failing run, we have:
2022-02-01 04:48:37.757 EST [3683:1] [unknown] LOG: connection received: host=[local]
2022-02-01 04:48:37.767 EST [3683:2] [unknown] LOG: connection authorized: user=buildfarm database=regression application_name=027_stream_regress.pl
...
2022-02-01 04:49:09.719 EST [3683:2584] 027_stream_regress.pl LOG: statement: COPY public.tenk1 (unique1, unique2, two, four, ten, twenty, hundred, thousand, twothousand, fivethous, tenthous, odd, even, stringu1, stringu2, string4) TO stdout;
2022-02-01 04:49:09.881 EST [3683:2585] 027_stream_regress.pl ERROR: canceling statement due to conflict with recovery
2022-02-01 04:49:09.881 EST [3683:2586] 027_stream_regress.pl DETAIL: User query might have needed to see row versions that must be removed.
2022-02-01 04:49:09.881 EST [3683:2587] 027_stream_regress.pl STATEMENT: COPY public.tenk1 (unique1, unique2, two, four, ten, twenty, hundred, thousand, twothousand, fivethous, tenthous, odd, even, stringu1, stringu2, string4) TO stdout;
2022-02-01 04:49:09.889 EST [3685:1] [unknown] LOG: connection received: host=[local]
2022-02-01 04:49:09.905 EST [3683:2588] 027_stream_regress.pl LOG: could not send data to client: Broken pipe
2022-02-01 04:49:09.905 EST [3683:2589] 027_stream_regress.pl ERROR: canceling statement due to conflict with recovery
2022-02-01 04:49:09.905 EST [3683:2590] 027_stream_regress.pl DETAIL: User query might have needed to see row versions that must be removed.
2022-02-01 04:49:09.906 EST [3683:2591] 027_stream_regress.pl FATAL: connection to client lost
2022-02-01 04:49:09.935 EST [3683:2592] 027_stream_regress.pl LOG: disconnection: session time: 0:00:32.179 user=buildfarm database=regression host=[local]
That's only a little over 30s. Where are you getting 2m from?
regards, tom lane
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-02-02 02:43 Tom Lane <tgl@sss.pgh.pa.us>
parent: Thomas Munro <thomas.munro@gmail.com>
1 sibling, 1 reply; 82+ messages in thread
From: Tom Lane @ 2022-02-02 02:43 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Michael Paquier <michael@paquier.xyz>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; pgsql-hackers
Thomas Munro <thomas.munro@gmail.com> writes:
> Seen again today on prairiedog. Erm, scratch that idea, HS feedback
> interferes with test results. I guess max_standby_streaming_delay
> should be increased to 'forever', like in the attached, since pg_dump
> runs for a very long time on prairiedog:
FWIW, I'd vote for keeping a finite timeout, but making it say
ten minutes. If the thing gets stuck for some reason, you don't
really want the test waiting forever. (Some buildfarm animals
have overall-test-time limits, but I think it's not the default,
and the behavior when that gets hit is pretty unfriendly anyway
-- you don't get any report of the run at all.)
regards, tom lane
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-02-02 03:14 Thomas Munro <thomas.munro@gmail.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2022-02-02 03:14 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andres Freund <andres@anarazel.de>; Michael Paquier <michael@paquier.xyz>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; pgsql-hackers
On Wed, Feb 2, 2022 at 3:43 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Thomas Munro <thomas.munro@gmail.com> writes:
> > Seen again today on prairiedog. Erm, scratch that idea, HS feedback
> > interferes with test results. I guess max_standby_streaming_delay
> > should be increased to 'forever', like in the attached, since pg_dump
> > runs for a very long time on prairiedog:
>
> FWIW, I'd vote for keeping a finite timeout, but making it say
> ten minutes. If the thing gets stuck for some reason, you don't
> really want the test waiting forever. (Some buildfarm animals
> have overall-test-time limits, but I think it's not the default,
> and the behavior when that gets hit is pretty unfriendly anyway
> -- you don't get any report of the run at all.)
Ok, I've set it to 10 minutes. Thanks.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-03-20 04:20 Thomas Munro <thomas.munro@gmail.com>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2022-03-20 04:20 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andres Freund <andres@anarazel.de>; Michael Paquier <michael@paquier.xyz>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; pgsql-hackers
Another failure under 027_stream_regress.pl:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2022-03-16%2005%3A58%3A05
vacuum ... FAILED 3463 ms
I'll try to come up with the perl needed to see the regression.diffs
next time...
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-03-20 09:36 Thomas Munro <thomas.munro@gmail.com>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2022-03-20 09:36 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andres Freund <andres@anarazel.de>; Michael Paquier <michael@paquier.xyz>; Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; pgsql-hackers
On Sun, Mar 20, 2022 at 5:20 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> Another failure under 027_stream_regress.pl:
>
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2022-03-16%2005%3A58%3A05
>
> vacuum ... FAILED 3463 ms
>
> I'll try to come up with the perl needed to see the regression.diffs
> next time...
Here's my proposed change to achieve that.
Here's an example of where it shows up if it fails (from my
deliberately sabotaged CI run
https://cirrus-ci.com/build/6730380228165632 where I was verifying
that it also works on Windows):
Unix: https://api.cirrus-ci.com/v1/artifact/task/5421419923243008/log/src/test/recovery/tmp_check/log/regr...
Windows: https://api.cirrus-ci.com/v1/artifact/task/4717732481466368/log/src/test/recovery/tmp_check/log/regr...
Attachments:
0001-Log-regression.diffs-in-027_stream_regress.pl.patch (text/x-patch, 2.2K) download
From 5f6257c4b4a44750e85d77b01c20d6cd39c5c795 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Sun, 20 Mar 2022 21:20:42 +1300
Subject: [PATCH] Log regression.diffs in 027_stream_regress.pl.
To help diagnose the reasons for a regression test failure inside this
TAP test, dump the contents of regression.diffs to the log. While the
CI scripts show it automatically, the build farm client does not.
Discussion: https://postgr.es/m/CA%2BhUKGK-q9utvxorA8sCwF3S4SzBmxxDgneP4rLqeHWdZxM4Gg%40mail.gmail.com
---
src/test/recovery/t/027_stream_regress.pl | 30 ++++++++++++++++-------
1 file changed, 21 insertions(+), 9 deletions(-)
diff --git a/src/test/recovery/t/027_stream_regress.pl b/src/test/recovery/t/027_stream_regress.pl
index c40951b7ba..aa972f8958 100644
--- a/src/test/recovery/t/027_stream_regress.pl
+++ b/src/test/recovery/t/027_stream_regress.pl
@@ -53,15 +53,27 @@ my $outputdir = $PostgreSQL::Test::Utils::tmp_check;
# Run the regression tests against the primary.
my $extra_opts = $ENV{EXTRA_REGRESS_OPTS} || "";
-system_or_bail($ENV{PG_REGRESS} . " $extra_opts " .
- "--dlpath=\"$dlpath\" " .
- "--bindir= " .
- "--host=" . $node_primary->host . " " .
- "--port=" . $node_primary->port . " " .
- "--schedule=../regress/parallel_schedule " .
- "--max-concurrent-tests=20 " .
- "--inputdir=../regress " .
- "--outputdir=\"$outputdir\"");
+my $rc = system($ENV{PG_REGRESS} . " $extra_opts " .
+ "--dlpath=\"$dlpath\" " .
+ "--bindir= " .
+ "--host=" . $node_primary->host . " " .
+ "--port=" . $node_primary->port . " " .
+ "--schedule=../regress/parallel_schedule " .
+ "--max-concurrent-tests=20 " .
+ "--inputdir=../regress " .
+ "--outputdir=\"$outputdir\"");
+if ($rc != 0)
+{
+ # Dump out the regression diffs file, if there is one
+ my $diffs = "$outputdir/regression.diffs";
+ if (-e $diffs)
+ {
+ print "=== dumping $diffs ===\n";
+ print slurp_file($diffs);
+ print "=== EOF ===\n";
+ }
+}
+is($rc, 0, 'regression tests pass');
# Clobber all sequences with their next value, so that we don't have
# differences between nodes due to caching.
--
2.30.2
* Re: A test for replay of regression tests
@ 2022-03-20 13:34 Andrew Dunstan <andrew@dunslane.net>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Andrew Dunstan @ 2022-03-20 13:34 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andres Freund <andres@anarazel.de>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; pgsql-hackers
On 3/20/22 05:36, Thomas Munro wrote:
> On Sun, Mar 20, 2022 at 5:20 PM Thomas Munro <thomas.munro@gmail.com> wrote:
>> Another failure under 027_stream_regress.pl:
>>
>> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2022-03-16%2005%3A58%3A05
>>
>> vacuum ... FAILED 3463 ms
>>
>> I'll try to come up with the perl needed to see the regression.diffs
>> next time...
> Here's my proposed change to achieve that.
I think that's OK.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-03-20 20:44 Thomas Munro <thomas.munro@gmail.com>
parent: Andrew Dunstan <andrew@dunslane.net>
0 siblings, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2022-03-20 20:44 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Andres Freund <andres@anarazel.de>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; pgsql-hackers
On Mon, Mar 21, 2022 at 2:34 AM Andrew Dunstan <andrew@dunslane.net> wrote:
> On 3/20/22 05:36, Thomas Munro wrote:
> > On Sun, Mar 20, 2022 at 5:20 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> >> I'll try to come up with the perl needed to see the regression.diffs
> >> next time...
> > Here's my proposed change to achieve that.
>
> I think that's OK.
Thanks for looking! Pushed.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-03-22 03:31 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Masahiko Sawada @ 2022-03-22 03:31 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Tom Lane <tgl@sss.pgh.pa.us>; Andres Freund <andres@anarazel.de>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; pgsql-hackers
i,
On Mon, Mar 21, 2022 at 5:45 AM Thomas Munro <thomas.munro@gmail.com> wrote:
>
> On Mon, Mar 21, 2022 at 2:34 AM Andrew Dunstan <andrew@dunslane.net> wrote:
> > On 3/20/22 05:36, Thomas Munro wrote:
> > > On Sun, Mar 20, 2022 at 5:20 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> > >> I'll try to come up with the perl needed to see the regression.diffs
> > >> next time...
> > > Here's my proposed change to achieve that.
> >
> > I think that's OK.
>
> Thanks for looking! Pushed.
FYI idiacanthus failed 027_stream_regress.pl:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=idiacanthus&dt=2022-03-22%2001%3A58%3A04
The log shows:
=== dumping /home/bf/build/buildfarm-idiacanthus/HEAD/pgsql.build/src/test/recovery/tmp_check/regression.diffs
===
diff -U3 /home/bf/build/buildfarm-idiacanthus/HEAD/pgsql/src/test/regress/expected/vacuum.out
/home/bf/build/buildfarm-idiacanthus/HEAD/pgsql.build/src/test/recovery/tmp_check/results/vacuum.out
--- /home/bf/build/buildfarm-idiacanthus/HEAD/pgsql/src/test/regress/expected/vacuum.out
2021-07-01 19:00:01.936659446 +0200
+++ /home/bf/build/buildfarm-idiacanthus/HEAD/pgsql.build/src/test/recovery/tmp_check/results/vacuum.out
2022-03-22 03:28:09.813377179 +0100
@@ -181,7 +181,7 @@
SELECT pg_relation_size('vac_truncate_test') = 0;
?column?
----------
- t
+ f
(1 row)
VACUUM (TRUNCATE FALSE, FULL TRUE) vac_truncate_test;
=== EOF ===
not ok 2 - regression tests pass
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-03-22 03:58 Thomas Munro <thomas.munro@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2022-03-22 03:58 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Tom Lane <tgl@sss.pgh.pa.us>; Andres Freund <andres@anarazel.de>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; pgsql-hackers
On Tue, Mar 22, 2022 at 4:31 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> SELECT pg_relation_size('vac_truncate_test') = 0;
> ?column?
> ----------
> - t
> + f
Thanks. Ahh, déjà vu... this probably needs the same treatment as
b700f96c and 3414099c provided for the reloptions test. Well, at
least the first one. Here's a patch like that.
Attachments:
0001-Try-to-stabilize-vacuum-test.patch (text/x-patch, 2.7K) download
From ba05f03c202bf66c7692787ec24ece13b193a897 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 22 Mar 2022 16:54:42 +1300
Subject: [PATCH] Try to stabilize vacuum test.
As commits b700f96c and 3414099c did for the reloptions test, take
measures to make sure that vacuum always truncates the table.
Discussion: https://postgr.es/m/CAD21AoCNoWjYkdEtr%2BVDoF9v__V905AedKZ9iF%3DArgCtrbxZqw%40mail.gmail.com
---
src/test/regress/expected/vacuum.out | 6 +++---
src/test/regress/sql/vacuum.sql | 6 +++---
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/src/test/regress/expected/vacuum.out b/src/test/regress/expected/vacuum.out
index 3e70e4c788..c63a157e5f 100644
--- a/src/test/regress/expected/vacuum.out
+++ b/src/test/regress/expected/vacuum.out
@@ -165,19 +165,19 @@ VACUUM (INDEX_CLEANUP FALSE) vaccluster;
VACUUM (INDEX_CLEANUP AUTO) vactst; -- index cleanup option is ignored if no indexes
VACUUM (INDEX_CLEANUP FALSE, FREEZE TRUE) vaccluster;
-- TRUNCATE option
-CREATE TABLE vac_truncate_test(i INT NOT NULL, j text)
+CREATE TEMP TABLE vac_truncate_test(i INT NOT NULL, j text)
WITH (vacuum_truncate=true, autovacuum_enabled=false);
INSERT INTO vac_truncate_test VALUES (1, NULL), (NULL, NULL);
ERROR: null value in column "i" of relation "vac_truncate_test" violates not-null constraint
DETAIL: Failing row contains (null, null).
-VACUUM (TRUNCATE FALSE) vac_truncate_test;
+VACUUM (TRUNCATE FALSE, DISABLE_PAGE_SKIPPING) vac_truncate_test;
SELECT pg_relation_size('vac_truncate_test') > 0;
?column?
----------
t
(1 row)
-VACUUM vac_truncate_test;
+VACUUM (DISABLE_PAGE_SKIPPING) vac_truncate_test;
SELECT pg_relation_size('vac_truncate_test') = 0;
?column?
----------
diff --git a/src/test/regress/sql/vacuum.sql b/src/test/regress/sql/vacuum.sql
index 18cb7fd08a..9faa8a34a6 100644
--- a/src/test/regress/sql/vacuum.sql
+++ b/src/test/regress/sql/vacuum.sql
@@ -147,12 +147,12 @@ VACUUM (INDEX_CLEANUP AUTO) vactst; -- index cleanup option is ignored if no ind
VACUUM (INDEX_CLEANUP FALSE, FREEZE TRUE) vaccluster;
-- TRUNCATE option
-CREATE TABLE vac_truncate_test(i INT NOT NULL, j text)
+CREATE TEMP TABLE vac_truncate_test(i INT NOT NULL, j text)
WITH (vacuum_truncate=true, autovacuum_enabled=false);
INSERT INTO vac_truncate_test VALUES (1, NULL), (NULL, NULL);
-VACUUM (TRUNCATE FALSE) vac_truncate_test;
+VACUUM (TRUNCATE FALSE, DISABLE_PAGE_SKIPPING) vac_truncate_test;
SELECT pg_relation_size('vac_truncate_test') > 0;
-VACUUM vac_truncate_test;
+VACUUM (DISABLE_PAGE_SKIPPING) vac_truncate_test;
SELECT pg_relation_size('vac_truncate_test') = 0;
VACUUM (TRUNCATE FALSE, FULL TRUE) vac_truncate_test;
DROP TABLE vac_truncate_test;
--
2.30.2
* Re: A test for replay of regression tests
@ 2022-03-25 03:02 Peter Geoghegan <pg@bowt.ie>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Peter Geoghegan @ 2022-03-25 03:02 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Tom Lane <tgl@sss.pgh.pa.us>; Andres Freund <andres@anarazel.de>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; pgsql-hackers
On Mon, Mar 21, 2022 at 8:59 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> Thanks. Ahh, déjà vu... this probably needs the same treatment as
> b700f96c and 3414099c provided for the reloptions test. Well, at
> least the first one. Here's a patch like that.
If you want to know whether or not the buildfarm will have problems
due to VACUUM failing to get a cleanup lock randomly, then I suggest
that you use an approach like the one from my patch here:
https://postgr.es/m/CAH2-WzkiB-qcsBmWrpzP0nxvrQExoUts1d7TYShg_DrkOHeg4Q@mail.gmail.com
I recently tried it again myself. With the gizmo in place the tests
fail in exactly the same way you've had problems with on the
buildfarm. On the first try, even.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-03-25 03:55 Thomas Munro <thomas.munro@gmail.com>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2022-03-25 03:55 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Tom Lane <tgl@sss.pgh.pa.us>; Andres Freund <andres@anarazel.de>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; pgsql-hackers
On Fri, Mar 25, 2022 at 4:03 PM Peter Geoghegan <pg@bowt.ie> wrote:
> If you want to know whether or not the buildfarm will have problems
> due to VACUUM failing to get a cleanup lock randomly, then I suggest
> that you use an approach like the one from my patch here:
>
> https://postgr.es/m/CAH2-WzkiB-qcsBmWrpzP0nxvrQExoUts1d7TYShg_DrkOHeg4Q@mail.gmail.com
>
> I recently tried it again myself. With the gizmo in place the tests
> fail in exactly the same way you've had problems with on the
> buildfarm. On the first try, even.
Interesting. IIUC your chaos gizmo shows that particular vacuum test
still failing on master, but that wouldn't happen in real life because
since 383f2221 it's a temp table. Your gizmo should probably detect
temp rels, as your comment says. I was sort of thinking that perhaps
if DISABLE_PAGE_SKIPPING is eventually made to do what its name sounds
like it does, we could remove TEMP from that test and it'd still pass
with the gizmo...
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-03-25 04:06 Peter Geoghegan <pg@bowt.ie>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Peter Geoghegan @ 2022-03-25 04:06 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Tom Lane <tgl@sss.pgh.pa.us>; Andres Freund <andres@anarazel.de>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; pgsql-hackers
On Thu, Mar 24, 2022 at 8:56 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> Interesting. IIUC your chaos gizmo shows that particular vacuum test
> still failing on master, but that wouldn't happen in real life because
> since 383f2221 it's a temp table. Your gizmo should probably detect
> temp rels, as your comment says. I was sort of thinking that perhaps
> if DISABLE_PAGE_SKIPPING is eventually made to do what its name sounds
> like it does, we could remove TEMP from that test and it'd still pass
> with the gizmo...
Why not just use VACUUM FREEZE? That should work, because it won't
settle for a cleanup lock on any page with an XID < OldestXmin. And
even if there were only LP_DEAD items on a page, that wouldn't matter
either, because we don't need a cleanup lock to get rid of those
anymore. And we consistently do all the same steps for rel truncation
in the no-cleanup-lock path (lazy_scan_noprune) now.
I think that DISABLE_PAGE_SKIPPING isn't appropriate for this kind of
thing. It mostly just makes VACUUM not trust the visibility map, which
isn't going to help. While DISABLE_PAGE_SKIPPING also forces
aggressive mode, that isn't going to help either, unless you somehow
also make sure that FreezeLimit is OldestXmin (e.g. by setting
vacuum_freeze_min_age to 0).
VACUUM FREEZE (without DISABLE_PAGE_SKIPPING) seems like it would do
everything you want, without using a temp table. At least on the
master branch.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-03-25 04:16 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 2 replies; 82+ messages in thread
From: Andres Freund @ 2022-03-25 04:16 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Masahiko Sawada <sawada.mshk@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Tom Lane <tgl@sss.pgh.pa.us>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; pgsql-hackers
Hi,
On 2022-03-24 21:06:21 -0700, Peter Geoghegan wrote:
> On Thu, Mar 24, 2022 at 8:56 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> > Interesting. IIUC your chaos gizmo shows that particular vacuum test
> > still failing on master, but that wouldn't happen in real life because
> > since 383f2221 it's a temp table. Your gizmo should probably detect
> > temp rels, as your comment says. I was sort of thinking that perhaps
> > if DISABLE_PAGE_SKIPPING is eventually made to do what its name sounds
> > like it does, we could remove TEMP from that test and it'd still pass
> > with the gizmo...
>
> Why not just use VACUUM FREEZE? That should work, because it won't
> settle for a cleanup lock on any page with an XID < OldestXmin. And
> even if there were only LP_DEAD items on a page, that wouldn't matter
> either, because we don't need a cleanup lock to get rid of those
> anymore. And we consistently do all the same steps for rel truncation
> in the no-cleanup-lock path (lazy_scan_noprune) now.
>
> I think that DISABLE_PAGE_SKIPPING isn't appropriate for this kind of
> thing. It mostly just makes VACUUM not trust the visibility map, which
> isn't going to help. While DISABLE_PAGE_SKIPPING also forces
> aggressive mode, that isn't going to help either, unless you somehow
> also make sure that FreezeLimit is OldestXmin (e.g. by setting
> vacuum_freeze_min_age to 0).
>
> VACUUM FREEZE (without DISABLE_PAGE_SKIPPING) seems like it would do
> everything you want, without using a temp table. At least on the
> master branch.
We tried that in a prior case:
https://postgr.es/m/20220120052404.sonrhq3f3qgplpzj%40alap3.anarazel.de
I don't know if the same danger applies here though.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-03-25 04:25 Thomas Munro <thomas.munro@gmail.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 82+ messages in thread
From: Thomas Munro @ 2022-03-25 04:25 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Peter Geoghegan <pg@bowt.ie>; Masahiko Sawada <sawada.mshk@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Tom Lane <tgl@sss.pgh.pa.us>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; pgsql-hackers
On Fri, Mar 25, 2022 at 5:16 PM Andres Freund <andres@anarazel.de> wrote:
> On 2022-03-24 21:06:21 -0700, Peter Geoghegan wrote:
> > VACUUM FREEZE (without DISABLE_PAGE_SKIPPING) seems like it would do
> > everything you want, without using a temp table. At least on the
> > master branch.
>
> We tried that in a prior case:
> https://postgr.es/m/20220120052404.sonrhq3f3qgplpzj%40alap3.anarazel.de
Yeah, or really, it was Michael that tried that in commit fe246d1c,
and then we tried more things with 3414099c and b700f96c. It's a bit
of a belt-and-braces setup admittedly...
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-03-25 04:26 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 82+ messages in thread
From: Peter Geoghegan @ 2022-03-25 04:26 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Masahiko Sawada <sawada.mshk@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Tom Lane <tgl@sss.pgh.pa.us>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; pgsql-hackers
On Thu, Mar 24, 2022 at 9:16 PM Andres Freund <andres@anarazel.de> wrote:
> > VACUUM FREEZE (without DISABLE_PAGE_SKIPPING) seems like it would do
> > everything you want, without using a temp table. At least on the
> > master branch.
>
> We tried that in a prior case:
> https://postgr.es/m/20220120052404.sonrhq3f3qgplpzj%40alap3.anarazel.de
Oh, yeah. If some other backend is holding back OldestXmin, and you
can't find a way of dealing with that, then you'll need a temp table.
(Mind you, that trick only works on recent versions too.)
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-03-28 03:11 Thomas Munro <thomas.munro@gmail.com>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2022-03-28 03:11 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Andres Freund <andres@anarazel.de>; Masahiko Sawada <sawada.mshk@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Tom Lane <tgl@sss.pgh.pa.us>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; pgsql-hackers
cfbot found another source of nondeterminism in the regression tests,
due to the smaller shared_buffers used in this TAP test:
https://cirrus-ci.com/task/4611828654276608
https://api.cirrus-ci.com/v1/artifact/task/4611828654276608/log/src/test/recovery/tmp_check/regressi...
Turned out that we had already diagnosed that once before, when tiny
build farm animal chipmunk reported the same, but we didn't commit a
fix:
https://www.postgresql.org/message-id/flat/CA%2BhUKGLTK6ZuEkpeJ05-MEmvmgZveCh%2B_w013m7%2ByKWFSmRcDA...
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-04-02 05:10 Tom Lane <tgl@sss.pgh.pa.us>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 82+ messages in thread
From: Tom Lane @ 2022-04-02 05:10 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; Masahiko Sawada <sawada.mshk@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; pgsql-hackers
Thomas Munro <thomas.munro@gmail.com> writes:
> cfbot found another source of nondeterminism in the regression tests,
> due to the smaller shared_buffers used in this TAP test:
This failure seems related but not identical:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=myna&dt=2022-04-02%2004%3A00%3A26
portals.out is expecting that the "foo25ns" cursor will read
starting at the beginning of tenk1, but it's starting somewhere
else, which presumably is a syncscan effect.
I think the fundamental instability here is that this TAP test is
setting shared_buffers small enough to allow the syncscan logic
to kick in where it does not in normal testing. Maybe we should
just disable syncscan in this test script?
regards, tom lane
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-04-04 16:39 Tom Lane <tgl@sss.pgh.pa.us>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 82+ messages in thread
From: Tom Lane @ 2022-04-04 16:39 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; Masahiko Sawada <sawada.mshk@gmail.com>; Andrew Dunstan <andrew@dunslane.net>; Michael Paquier <michael@paquier.xyz>; Anastasia Lubennikova <lubennikovaav@gmail.com>; pgsql-hackers
I wrote:
> I think the fundamental instability here is that this TAP test is
> setting shared_buffers small enough to allow the syncscan logic
> to kick in where it does not in normal testing. Maybe we should
> just disable syncscan in this test script?
Did that, we'll see how much it helps.
regards, tom lane
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-08-03 15:30 Justin Pryzby <pryzby@telsasoft.com>
parent: Thomas Munro <thomas.munro@gmail.com>
2 siblings, 1 reply; 82+ messages in thread
From: Justin Pryzby @ 2022-08-03 15:30 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Thu, Dec 09, 2021 at 12:10:23PM +1300, Thomas Munro wrote:
> This adds 2 whole minutes to the recovery check, when running with the
> Windows serial-only scripts on Cirrus CI (using Andres's CI patches).
> For Linux it adds ~20 seconds to the total of -j8 check-world.
> Hopefully that's time well spent, because it adds test coverage for
> all the redo routines, and hopefully soon we won't have to run 'em in
> series on Windows.
Should 027-stream-regress be renamed to something that starts earlier ?
Off-list earlier this year, Andres referred to 000.
--
Justin
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-08-03 21:24 Thomas Munro <thomas.munro@gmail.com>
parent: Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 82+ messages in thread
From: Thomas Munro @ 2022-08-03 21:24 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Thu, Aug 4, 2022 at 3:30 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
> On Thu, Dec 09, 2021 at 12:10:23PM +1300, Thomas Munro wrote:
> > This adds 2 whole minutes to the recovery check, when running with the
> > Windows serial-only scripts on Cirrus CI (using Andres's CI patches).
> > For Linux it adds ~20 seconds to the total of -j8 check-world.
> > Hopefully that's time well spent, because it adds test coverage for
> > all the redo routines, and hopefully soon we won't have to run 'em in
> > series on Windows.
>
> Should 027-stream-regress be renamed to something that starts earlier ?
> Off-list earlier this year, Andres referred to 000.
Do you have any data on improved times from doing that?
I have wondered about moving it into 001_stream_rep.pl.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: A test for replay of regression tests
@ 2022-08-04 01:13 Justin Pryzby <pryzby@telsasoft.com>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 0 replies; 82+ messages in thread
From: Justin Pryzby @ 2022-08-04 01:13 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Anastasia Lubennikova <lubennikovaav@gmail.com>; Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Thu, Aug 04, 2022 at 09:24:24AM +1200, Thomas Munro wrote:
> On Thu, Aug 4, 2022 at 3:30 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
> > On Thu, Dec 09, 2021 at 12:10:23PM +1300, Thomas Munro wrote:
> > > This adds 2 whole minutes to the recovery check, when running with the
> > > Windows serial-only scripts on Cirrus CI (using Andres's CI patches).
> > > For Linux it adds ~20 seconds to the total of -j8 check-world.
> > > Hopefully that's time well spent, because it adds test coverage for
> > > all the redo routines, and hopefully soon we won't have to run 'em in
> > > series on Windows.
> >
> > Should 027-stream-regress be renamed to something that starts earlier ?
> > Off-list earlier this year, Andres referred to 000.
See also:
https://www.postgresql.org/message-id/20220213220709.vjz5rziuhfdpqxrg@alap3.anarazel.de
> Do you have any data on improved times from doing that?
>
> I have wondered about moving it into 001_stream_rep.pl.
The immediate motive for raising the question is due to working on your cygwin
patch (where I've set PROVE_FLAGS=-j3). The last invocation I have opened ends
like:
[20:46:47.577] [13:46:47] t/026_overwrite_contrecord.pl ........ ok 10264 ms ( 0.02 usr 0.02 sys + 11.25 cusr 35.95 csys = 47.24 CPU)
[20:47:08.087] [13:47:08] t/028_pitr_timelines.pl .............. ok 13153 ms ( 0.00 usr 0.00 sys + 4.03 cusr 14.79 csys = 18.82 CPU)
[20:47:08.999] [13:47:09] t/029_stats_restart.pl ............... ok 12631 ms ( 0.00 usr 0.02 sys + 7.40 cusr 23.30 csys = 30.71 CPU)
[20:47:34.353] [13:47:34] t/031_recovery_conflict.pl ........... ok 11337 ms ( 0.00 usr 0.00 sys + 3.84 cusr 11.82 csys = 15.66 CPU)
[20:47:35.070] [13:47:35] t/030_stats_cleanup_replica.pl ....... ok 14054 ms ( 0.02 usr 0.00 sys + 7.64 cusr 25.02 csys = 32.68 CPU)
[20:48:04.887] [13:48:04] t/032_relfilenode_reuse.pl ........... ok 12755 ms ( 0.00 usr 0.00 sys + 3.36 cusr 11.57 csys = 14.93 CPU)
[20:48:42.055] [13:48:42] t/033_replay_tsp_drops.pl ............ ok 43529 ms ( 0.00 usr 0.00 sys + 12.29 cusr 41.43 csys = 53.71 CPU)
[20:50:02.770] [13:50:02] t/027_stream_regress.pl .............. ok 198408 ms ( 0.02 usr 0.06 sys + 44.92 cusr 142.42 csys = 187.42 CPU)
[20:50:02.771] [13:50:02]
[20:50:02.771] All tests successful.
[20:50:02.771] Files=33, Tests=411, 402 wallclock secs ( 0.16 usr 0.27 sys + 138.03 cusr 441.56 csys = 580.01 CPU)
If 027 had been started sooner, this test might have finished up to 78sec
earlier. If lots of tests are added in the future, maybe it won't matter, but
it seems like it does now.
As I understand, checks are usually parallelized by "make -j" and not by
"prove". In that case, starting a slow test later doesn't matter. But it'd be
better for anyone who runs tap tests manually, and (I think) for meson.
As a one-off test on localhost:
time make check -C src/test/recovery
=> 11m42,790s
time make check -C src/test/recovery PROVE_FLAGS=-j2
=> 7m56,315s
After renaming it to 001:
time make check -C src/test/recovery
=> 11m33,887s (~same)
time make check -C src/test/recovery PROVE_FLAGS=-j2
=> 6m59,969s
I don't know how it affect the buildfarm (but I think that's not optimized
primarily for speed anyway).
--
Justin
^ permalink raw reply [nested|flat] 82+ messages in thread
end of thread, other threads:[~2022-08-04 01:13 UTC | newest]
Thread overview: 82+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-04-23 05:37 A test for replay of regression tests Thomas Munro <thomas.munro@gmail.com>
2021-04-23 06:27 ` tsunakawa.takay@fujitsu.com <tsunakawa.takay@fujitsu.com>
2021-04-23 07:51 ` Thomas Munro <thomas.munro@gmail.com>
2021-04-23 15:20 ` Andres Freund <andres@anarazel.de>
2021-04-23 15:53 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-04-23 17:04 ` Andres Freund <andres@anarazel.de>
2021-04-23 17:13 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-04-23 17:22 ` Andres Freund <andres@anarazel.de>
2021-05-04 11:12 ` Thomas Munro <thomas.munro@gmail.com>
2021-06-07 23:44 ` Anastasia Lubennikova <lubennikovaav@gmail.com>
2021-06-10 07:37 ` Anastasia Lubennikova <lubennikovaav@gmail.com>
2021-06-10 07:47 ` Thomas Munro <thomas.munro@gmail.com>
2021-10-06 06:10 ` Thomas Munro <thomas.munro@gmail.com>
2021-11-23 09:07 ` Thomas Munro <thomas.munro@gmail.com>
2021-11-23 15:47 ` Andrew Dunstan <andrew@dunslane.net>
2021-11-29 14:14 ` Andrew Dunstan <andrew@dunslane.net>
2021-12-04 04:21 ` Thomas Munro <thomas.munro@gmail.com>
2021-12-04 15:16 ` Andrew Dunstan <andrew@dunslane.net>
2021-12-08 23:10 ` Thomas Munro <thomas.munro@gmail.com>
2021-12-09 13:12 ` Andrew Dunstan <andrew@dunslane.net>
2021-12-09 19:38 ` Andres Freund <andres@anarazel.de>
2021-12-09 21:35 ` Thomas Munro <thomas.munro@gmail.com>
2021-12-09 20:15 ` Thomas Munro <thomas.munro@gmail.com>
2021-12-11 00:06 ` Andrew Dunstan <andrew@dunslane.net>
2021-12-09 21:36 ` Andres Freund <andres@anarazel.de>
2021-12-09 23:58 ` Thomas Munro <thomas.munro@gmail.com>
2021-12-10 00:09 ` Thomas Munro <thomas.munro@gmail.com>
2021-12-11 00:17 ` Andres Freund <andres@anarazel.de>
2021-12-15 08:50 ` Michael Paquier <michael@paquier.xyz>
2021-12-16 00:22 ` Michael Paquier <michael@paquier.xyz>
2021-12-21 22:41 ` Thomas Munro <thomas.munro@gmail.com>
2022-01-14 11:39 ` Thomas Munro <thomas.munro@gmail.com>
2022-01-14 13:32 ` Thomas Munro <thomas.munro@gmail.com>
2022-01-14 23:49 ` Andres Freund <andres@anarazel.de>
2022-01-17 04:25 ` Thomas Munro <thomas.munro@gmail.com>
2022-01-17 05:53 ` Noah Misch <noah@leadboat.com>
2022-01-21 02:42 ` Thomas Munro <thomas.munro@gmail.com>
2022-01-21 18:58 ` Thomas Munro <thomas.munro@gmail.com>
2022-01-21 21:22 ` Andrew Dunstan <andrew@dunslane.net>
2022-01-27 20:27 ` Andrew Dunstan <andrew@dunslane.net>
2022-01-27 20:47 ` Andres Freund <andres@anarazel.de>
2022-01-27 22:03 ` Andres Freund <andres@anarazel.de>
2022-01-27 22:07 ` Andres Freund <andres@anarazel.de>
2022-01-27 22:21 ` Thomas Munro <thomas.munro@gmail.com>
2022-01-27 22:16 ` Andrew Dunstan <andrew@dunslane.net>
2022-01-27 22:36 ` Andres Freund <andres@anarazel.de>
2022-01-27 22:59 ` Andres Freund <andres@anarazel.de>
2022-01-27 22:51 ` Andrew Dunstan <andrew@dunslane.net>
2022-01-27 23:03 ` Andres Freund <andres@anarazel.de>
2022-01-27 23:24 ` Thomas Munro <thomas.munro@gmail.com>
2022-01-28 14:46 ` Andrew Dunstan <andrew@dunslane.net>
2022-01-27 21:41 ` Thomas Munro <thomas.munro@gmail.com>
2022-01-21 19:48 ` Andres Freund <andres@anarazel.de>
2022-01-21 20:07 ` Thomas Munro <thomas.munro@gmail.com>
2022-01-22 05:00 ` Thomas Munro <thomas.munro@gmail.com>
2022-02-02 00:59 ` Thomas Munro <thomas.munro@gmail.com>
2022-02-02 01:14 ` Andres Freund <andres@anarazel.de>
2022-02-02 01:35 ` Thomas Munro <thomas.munro@gmail.com>
2022-02-02 02:11 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-02-02 02:16 ` Andres Freund <andres@anarazel.de>
2022-02-02 02:33 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-02-02 02:43 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-02-02 03:14 ` Thomas Munro <thomas.munro@gmail.com>
2022-03-20 04:20 ` Thomas Munro <thomas.munro@gmail.com>
2022-03-20 09:36 ` Thomas Munro <thomas.munro@gmail.com>
2022-03-20 13:34 ` Andrew Dunstan <andrew@dunslane.net>
2022-03-20 20:44 ` Thomas Munro <thomas.munro@gmail.com>
2022-03-22 03:31 ` Masahiko Sawada <sawada.mshk@gmail.com>
2022-03-22 03:58 ` Thomas Munro <thomas.munro@gmail.com>
2022-03-25 03:02 ` Peter Geoghegan <pg@bowt.ie>
2022-03-25 03:55 ` Thomas Munro <thomas.munro@gmail.com>
2022-03-25 04:06 ` Peter Geoghegan <pg@bowt.ie>
2022-03-25 04:16 ` Andres Freund <andres@anarazel.de>
2022-03-25 04:25 ` Thomas Munro <thomas.munro@gmail.com>
2022-03-25 04:26 ` Peter Geoghegan <pg@bowt.ie>
2022-03-28 03:11 ` Thomas Munro <thomas.munro@gmail.com>
2022-04-02 05:10 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-04-04 16:39 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-08-03 15:30 ` Justin Pryzby <pryzby@telsasoft.com>
2022-08-03 21:24 ` Thomas Munro <thomas.munro@gmail.com>
2022-08-04 01:13 ` Justin Pryzby <pryzby@telsasoft.com>
2021-04-23 15:38 ` Andrew Dunstan <andrew@dunslane.net>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox