agora inbox for [email protected]
help / color / mirror / Atom feedpgbench regression test failure
34+ messages / 4 participants
[nested] [flat]
* pgbench regression test failure
@ 2017-09-12 16:07 Tom Lane <[email protected]>
2017-09-12 17:08 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
0 siblings, 1 reply; 34+ messages in thread
From: Tom Lane @ 2017-09-12 16:07 UTC (permalink / raw)
To: Fabien COELHO <[email protected]>; +Cc: pgsql-hackers
francolin just showed a non-reproducing failure in the new pgbench tests:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=francolin&dt=2017-09-12%2014%3A00%3A02
The relevant part of the log is
# Running: pgbench -T 2 -P 1 -l --log-prefix=001_pgbench_log_1 --aggregate-interval=1 -S -b se@2 --rate=20 --latency-limit=1000 -j 2 -c 3 -r
ok 198 - pgbench progress status (got 0 vs expected 0)
ok 199 - pgbench progress stdout /(?^:type: multiple)/
ok 200 - pgbench progress stdout /(?^:clients: 3)/
ok 201 - pgbench progress stdout /(?^:threads: 2)/
ok 202 - pgbench progress stdout /(?^:duration: 2 s)/
ok 203 - pgbench progress stdout /(?^:script 1: .* select only)/
ok 204 - pgbench progress stdout /(?^:script 2: .* select only)/
ok 205 - pgbench progress stdout /(?^:statement latencies in milliseconds)/
ok 206 - pgbench progress stdout /(?^:FROM pgbench_accounts)/
ok 207 - pgbench progress stderr /(?^:vacuum)/
ok 208 - pgbench progress stderr /(?^:progress: 1\b)/
ok 209 - number of log files
ok 210 - file name format
not ok 211 - transaction count for 001_pgbench_log_1.31583 (3)
# Failed test 'transaction count for 001_pgbench_log_1.31583 (3)'
# at t/001_pgbench_with_server.pl line 438.
ok 212 - transaction format for 001_pgbench_log_1
ok 213 - transaction count for 001_pgbench_log_1.31583.1 (2)
ok 214 - transaction format for 001_pgbench_log_1
ok 215 - remove log files
Apparently, one of the threads ran 3 transactions where the test script
expects it to run at most 2. Is this a pgbench bug, or is the test
being overoptimistic about how exact the "-T 2" cutoff is?
regards, tom lane
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pgbench regression test failure
2017-09-12 16:07 pgbench regression test failure Tom Lane <[email protected]>
@ 2017-09-12 17:08 ` Fabien COELHO <[email protected]>
2017-09-12 17:13 ` Re: pgbench regression test failure Tom Lane <[email protected]>
0 siblings, 1 reply; 34+ messages in thread
From: Fabien COELHO @ 2017-09-12 17:08 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
> francolin just showed a non-reproducing failure in the new pgbench tests:
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=francolin&dt=2017-09-12%2014%3A00%3A02
> not ok 211 - transaction count for 001_pgbench_log_1.31583 (3)
> # Failed test 'transaction count for 001_pgbench_log_1.31583 (3)'
> # at t/001_pgbench_with_server.pl line 438.
>
> Apparently, one of the threads ran 3 transactions where the test script
> expects it to run at most 2. Is this a pgbench bug, or is the test
> being overoptimistic about how exact the "-T 2" cutoff is?
Probably both? It seems that cutting off on time is not a precise science,
so I suggest to accept 1, 2 and 3 lines, see attached.
--
Fabien.
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] pgbench-tap-fix-3.patch (610B, ../../alpine.DEB.2.20.1709121850280.30961@lancre/2-pgbench-tap-fix-3.patch)
download | inline diff:
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 3609b9b..1fe3433 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -463,7 +463,8 @@ pgbench(
'pgbench progress');
# $nthreads threads, 2 seconds, sometimes only one aggregated line is written
-check_pgbench_logs('001_pgbench_log_1', $nthreads, 1, 2,
+# and sometimes 3 lines...
+check_pgbench_logs('001_pgbench_log_1', $nthreads, 1, 3,
qr{^\d+ \d{1,2} \d+ \d+ \d+ \d+ \d+ \d+ \d+ \d+ \d+$});
# with sampling rate
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pgbench regression test failure
2017-09-12 16:07 pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 17:08 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
@ 2017-09-12 17:13 ` Tom Lane <[email protected]>
2017-09-12 18:00 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
0 siblings, 1 reply; 34+ messages in thread
From: Tom Lane @ 2017-09-12 17:13 UTC (permalink / raw)
To: Fabien COELHO <[email protected]>; +Cc: pgsql-hackers
Fabien COELHO <[email protected]> writes:
>> Apparently, one of the threads ran 3 transactions where the test script
>> expects it to run at most 2. Is this a pgbench bug, or is the test
>> being overoptimistic about how exact the "-T 2" cutoff is?
> Probably both? It seems that cutting off on time is not a precise science,
> so I suggest to accept 1, 2 and 3 lines, see attached.
Before I'd deciphered the test output fully, I was actually guessing that
the problem was the opposite, namely too few lines. Isn't it possible
that some thread is slow enough to start up that it doesn't get to run
any transactions? IOW, do we need to allow 0 to 3 lines?
regards, tom lane
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pgbench regression test failure
2017-09-12 16:07 pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 17:08 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 17:13 ` Re: pgbench regression test failure Tom Lane <[email protected]>
@ 2017-09-12 18:00 ` Fabien COELHO <[email protected]>
2017-09-12 18:12 ` Re: pgbench regression test failure Tom Lane <[email protected]>
0 siblings, 1 reply; 34+ messages in thread
From: Fabien COELHO @ 2017-09-12 18:00 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
>>> Apparently, one of the threads ran 3 transactions where the test script
>>> expects it to run at most 2. Is this a pgbench bug, or is the test
>>> being overoptimistic about how exact the "-T 2" cutoff is?
>
>> Probably both? It seems that cutting off on time is not a precise science,
>> so I suggest to accept 1, 2 and 3 lines, see attached.
>
> Before I'd deciphered the test output fully, I was actually guessing that
> the problem was the opposite, namely too few lines.
The test was waiting for betwen 1 and 2 lines, so I assumed that the 3
should the number of lines found.
> Isn't it possible that some thread is slow enough to start up that it
> doesn't get to run any transactions? IOW, do we need to allow 0 to 3
> lines?
By definition, parallelism induces non determinism. When I put 2 seconds,
the intention was that I would get a non empty trace with a "every second"
aggregation. I would rather take a longer test rather than allowing an
empty file: the point is to check that something is generated, but
avoiding a longer test is desirable. So I would suggest to stick to
between 1 and 3, and if it fails then maybe add one second...
--
Fabien.
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pgbench regression test failure
2017-09-12 16:07 pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 17:08 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 17:13 ` Re: pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 18:00 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
@ 2017-09-12 18:12 ` Tom Lane <[email protected]>
2017-09-12 19:21 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
0 siblings, 1 reply; 34+ messages in thread
From: Tom Lane @ 2017-09-12 18:12 UTC (permalink / raw)
To: Fabien COELHO <[email protected]>; +Cc: pgsql-hackers
Fabien COELHO <[email protected]> writes:
> By definition, parallelism induces non determinism. When I put 2 seconds,
> the intention was that I would get a non empty trace with a "every second"
> aggregation. I would rather take a longer test rather than allowing an
> empty file: the point is to check that something is generated, but
> avoiding a longer test is desirable. So I would suggest to stick to
> between 1 and 3, and if it fails then maybe add one second...
That's a losing game. You can't ever guarantee that N seconds is
enough for slow, heavily loaded machines, and cranking up N just
penalizes developers who are testing under normal circumstances.
I have a serious, serious dislike for tests that seem to work until
they're run on a heavily loaded machine. So unless there is some
reason why pgbench is *guaranteed* to run at least one transaction
per thread, I'd rather the test not assume that.
I would not necessarily object to doing something in the code that
would guarantee that, though.
regards, tom lane
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pgbench regression test failure
2017-09-12 16:07 pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 17:08 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 17:13 ` Re: pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 18:00 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 18:12 ` Re: pgbench regression test failure Tom Lane <[email protected]>
@ 2017-09-12 19:21 ` Fabien COELHO <[email protected]>
2017-09-12 19:28 ` Re: pgbench regression test failure Tom Lane <[email protected]>
0 siblings, 1 reply; 34+ messages in thread
From: Fabien COELHO @ 2017-09-12 19:21 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
> I have a serious, serious dislike for tests that seem to work until
> they're run on a heavily loaded machine.
I'm not that sure the error message was because of that. ISTM that it was
rather finding 3 seconds in two because it started just at the right time,
or maybe because of slowness induce by load and the order in which the
different checks are performed.
> So unless there is some reason why pgbench is *guaranteed* to run at
> least one transaction per thread, I'd rather the test not assume that.
Well, pgbench is for testing performance... so if the checks allow zero
performance that's quite annoying as well:-) The tests are designed to
require very low performance (eg there are a lot of -t 1 when only one
transaction is enough to check a point), but maybe some test assume a
minimal requirement, maybe 10 tps with 2 threads...
> I would not necessarily object to doing something in the code that
> would guarantee that, though.
Hmmm. Interesting point.
There could be a client-side synchronization barrier, eg something like
"\sync :nclients/nthreads" could be easy enough to implement with pthread,
and quite error prone to use, but probably that could be okay for
validation purposes. Or maybe we could expose something at the SQL level,
eg "SELECT synchro('synchroname', whomanyclientstowait);" which would be
harder to implement server-side but possibly doable as well.
A simpler option may be to introduce a synchronization barrier at thread
start, so that all threads start together and that would set the "zero"
time. Not sure that would solve the potential issue you raise, although
that would help.
Currently the statistics collection and outputs are performed by thread 0
in addition to the client it runs, so that pgbench would work even if
there are no threads, but it also means that under a heavy load some
things may not be done on the target time but a little bit later, if some
thread is stuck somewhere. Although the async protocol try to avoid that.
--
Fabien.
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pgbench regression test failure
2017-09-12 16:07 pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 17:08 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 17:13 ` Re: pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 18:00 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 18:12 ` Re: pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 19:21 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
@ 2017-09-12 19:28 ` Tom Lane <[email protected]>
2017-09-13 09:42 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
0 siblings, 1 reply; 34+ messages in thread
From: Tom Lane @ 2017-09-12 19:28 UTC (permalink / raw)
To: Fabien COELHO <[email protected]>; +Cc: pgsql-hackers
Fabien COELHO <[email protected]> writes:
>> I have a serious, serious dislike for tests that seem to work until
>> they're run on a heavily loaded machine.
> I'm not that sure the error message was because of that.
No, this particular failure (probably) wasn't. But now that I've realized
that this test case is timing-sensitive, I'm worried about what will
happen when it's run on a sufficiently slow or loaded machine.
>> I would not necessarily object to doing something in the code that
>> would guarantee that, though.
> Hmmm. Interesting point.
It could be as simple as putting the check-for-done at the bottom of the
loop not the top, perhaps.
regards, tom lane
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pgbench regression test failure
2017-09-12 16:07 pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 17:08 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 17:13 ` Re: pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 18:00 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 18:12 ` Re: pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 19:21 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 19:28 ` Re: pgbench regression test failure Tom Lane <[email protected]>
@ 2017-09-13 09:42 ` Fabien COELHO <[email protected]>
2017-09-22 17:02 ` Re: pgbench regression test failure Tom Lane <[email protected]>
0 siblings, 1 reply; 34+ messages in thread
From: Fabien COELHO @ 2017-09-13 09:42 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
>>> I have a serious, serious dislike for tests that seem to work until
>>> they're run on a heavily loaded machine.
>
>> I'm not that sure the error message was because of that.
>
> No, this particular failure (probably) wasn't. But now that I've realized
> that this test case is timing-sensitive, I'm worried about what will
> happen when it's run on a sufficiently slow or loaded machine.
>
>>> I would not necessarily object to doing something in the code that
>>> would guarantee that, though.
>
>> Hmmm. Interesting point.
>
> It could be as simple as putting the check-for-done at the bottom of the
> loop not the top, perhaps.
I agree that it is best if tests should work in all reasonable conditions,
including a somehow overloaded host...
I'm going to think about it, but I'm not sure of the best approach. In the
mean time, ISTM that the issue has not been encountered (yet), so this is
not a pressing issue. Maybe under -T > --aggregate-interval pgbench could
go on over the limit if the log file has not been written at all, but that
would be some kind of kludge for this specific test...
Note that to get test coverage for -T and have to assume that maybe a
loaded host would not be able to generate just a one line log every second
during that time is kind of a hard assumption...
Maybe some test could be "warnings", i.e. it could be acceptable to accept
a failure once in a while in specific conditions, if this is rare enough
and documented. ISTM that there is such a test for random output.
--
Fabien.
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pgbench regression test failure
2017-09-12 16:07 pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 17:08 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 17:13 ` Re: pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 18:00 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 18:12 ` Re: pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 19:21 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 19:28 ` Re: pgbench regression test failure Tom Lane <[email protected]>
2017-09-13 09:42 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
@ 2017-09-22 17:02 ` Tom Lane <[email protected]>
2017-09-22 17:55 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
0 siblings, 1 reply; 34+ messages in thread
From: Tom Lane @ 2017-09-22 17:02 UTC (permalink / raw)
To: Fabien COELHO <[email protected]>; +Cc: pgsql-hackers
Fabien COELHO <[email protected]> writes:
>> It could be as simple as putting the check-for-done at the bottom of the
>> loop not the top, perhaps.
> I agree that it is best if tests should work in all reasonable conditions,
> including a somehow overloaded host...
> I'm going to think about it, but I'm not sure of the best approach. In the
> mean time, ISTM that the issue has not been encountered (yet), so this is
> not a pressing issue. Maybe under -T > --aggregate-interval pgbench could
> go on over the limit if the log file has not been written at all, but that
> would be some kind of kludge for this specific test...
After another week of buildfarm runs, we have a few more cases of 3 rows
of output, and none of more than 3 or less than 1. So I went ahead and
pushed your patch. I'm still suspicious of these results, but we might
as well try to make the buildfarm green pending investigation of how
this is happening.
regards, tom lane
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pgbench regression test failure
2017-09-12 16:07 pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 17:08 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 17:13 ` Re: pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 18:00 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 18:12 ` Re: pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 19:21 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 19:28 ` Re: pgbench regression test failure Tom Lane <[email protected]>
2017-09-13 09:42 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-22 17:02 ` Re: pgbench regression test failure Tom Lane <[email protected]>
@ 2017-09-22 17:55 ` Fabien COELHO <[email protected]>
2017-09-23 17:06 ` Re: pgbench regression test failure Tom Lane <[email protected]>
0 siblings, 1 reply; 34+ messages in thread
From: Fabien COELHO @ 2017-09-22 17:55 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
> [...] After another week of buildfarm runs, we have a few more cases of
> 3 rows of output, and none of more than 3 or less than 1. So I went
> ahead and pushed your patch. I'm still suspicious of these results, but
> we might as well try to make the buildfarm green pending investigation
> of how this is happening.
Yep. I keep the issue of pgbench tap test determinism in my todo list,
among other things.
I think that it should be at least clearer under which condition (load ?
luck ? bug ?) the result may be 1 or 3 when 2 are expected, which needs
some thinking.
--
Fabien.
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pgbench regression test failure
2017-09-12 16:07 pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 17:08 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 17:13 ` Re: pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 18:00 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 18:12 ` Re: pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 19:21 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 19:28 ` Re: pgbench regression test failure Tom Lane <[email protected]>
2017-09-13 09:42 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-22 17:02 ` Re: pgbench regression test failure Tom Lane <[email protected]>
2017-09-22 17:55 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
@ 2017-09-23 17:06 ` Tom Lane <[email protected]>
2017-09-24 07:17 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
0 siblings, 1 reply; 34+ messages in thread
From: Tom Lane @ 2017-09-23 17:06 UTC (permalink / raw)
To: Fabien COELHO <[email protected]>; +Cc: pgsql-hackers
Fabien COELHO <[email protected]> writes:
>> [...] After another week of buildfarm runs, we have a few more cases of
>> 3 rows of output, and none of more than 3 or less than 1. So I went
>> ahead and pushed your patch. I'm still suspicious of these results, but
>> we might as well try to make the buildfarm green pending investigation
>> of how this is happening.
> Yep. I keep the issue of pgbench tap test determinism in my todo list,
> among other things.
> I think that it should be at least clearer under which condition (load ?
> luck ? bug ?) the result may be 1 or 3 when 2 are expected, which needs
> some thinking.
skink blew up real good just now:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2017-09-23%2010%3A50%3A01
the critical bit being
# Failed test 'pgbench progress stderr /(?^:progress: 1\b)/'
# at /home/andres/build/buildfarm/HEAD/pgsql.build/../pgsql/src/test/perl/TestLib.pm line 369.
# 'starting vacuum...end.
# progress: 2.6 s, 6.9 tps, lat 0.000 ms stddev 0.000, lag 0.000 ms, 18 skipped
# progress: 3.0 s, 0.0 tps, lat -nan ms stddev -nan, lag -nan ms, 0 skipped
# progress: 4.0 s, 1.0 tps, lat 2682.730 ms stddev 0.000, lag 985.509 ms, 0 skipped
# '
# doesn't match '(?^:progress: 1\b)'
# Failed test 'transaction count for 001_pgbench_log_1.15981 (5)'
# at t/001_pgbench_with_server.pl line 438.
# Failed test 'transaction count for 001_pgbench_log_1.15981.1 (4)'
# at t/001_pgbench_with_server.pl line 438.
# Looks like you failed 3 tests of 233.
That's exceeded my patience with this test case, so I've removed it
for the moment. We can put it back as soon as we figure some way
to make it more robust. (BTW, the "-nan" bits suggest an actual
pgbench bug, independently of anything else.)
Possibly you can duplicate skink's issue by running things under
valgrind.
regards, tom lane
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: pgbench regression test failure
2017-09-12 16:07 pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 17:08 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 17:13 ` Re: pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 18:00 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 18:12 ` Re: pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 19:21 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-12 19:28 ` Re: pgbench regression test failure Tom Lane <[email protected]>
2017-09-13 09:42 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-22 17:02 ` Re: pgbench regression test failure Tom Lane <[email protected]>
2017-09-22 17:55 ` Re: pgbench regression test failure Fabien COELHO <[email protected]>
2017-09-23 17:06 ` Re: pgbench regression test failure Tom Lane <[email protected]>
@ 2017-09-24 07:17 ` Fabien COELHO <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Fabien COELHO @ 2017-09-24 07:17 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
This message is in MIME format. The first part should be readable text,
while the remaining parts are likely unreadable without MIME-aware tools.
--8323329-1587912164-1506237438=:4999
Content-Type: text/plain; charset=US-ASCII; format=flowed
Hello Tom,
> # progress: 2.6 s, 6.9 tps, lat 0.000 ms stddev 0.000, lag 0.000 ms, 18 skipped
> # progress: 3.0 s, 0.0 tps, lat -nan ms stddev -nan, lag -nan ms, 0 skipped
> # progress: 4.0 s, 1.0 tps, lat 2682.730 ms stddev 0.000, lag 985.509 ms, 0 skipped
> (BTW, the "-nan" bits suggest an actual pgbench bug, independently of
> anything else.)
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour
@ 2021-01-13 11:05 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:05 UTC (permalink / raw)
Currently, in logical replication, publisher/walsender publishes
the tables even though they aren't part of the publication i.e
they are dropped from the publication. Because of this ALTER
PUBLICATION...DROP TABLE doesn't work as expected.
---
src/backend/replication/pgoutput/pgoutput.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2f01137b42..118faf6ce3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1179,5 +1179,18 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
*/
hash_seq_init(&status, RelationSyncCache);
while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+ {
entry->replicate_valid = false;
+
+ /*
+ * There might some relations dropped from the publication, we do
+ * not need to publish the changes for them. However, we cannot get
+ * the dropped relations (see above), so we reset pubactions for all
+ * entries.
+ */
+ entry->pubactions.pubinsert = false;
+ entry->pubactions.pubupdate = false;
+ entry->pubactions.pubdelete = false;
+ entry->pubactions.pubtruncate = false;
+ }
}
--
2.30.0
--=-=-=
Content-Type: text/x-patch
Content-Disposition: attachment;
filename=v3-0002-Invalidate-relation-map-cache-in-subscriber-sysca.patch
^ permalink raw reply [nested|flat] 34+ messages in thread
* Re: Converting pqsignal to void return
@ 2025-01-22 16:22 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 34+ messages in thread
From: Nathan Bossart @ 2025-01-22 16:22 UTC (permalink / raw)
To: Paul Ramsey <[email protected]>; +Cc: pgsql-hackers
On Wed, Jan 22, 2025 at 07:57:52AM -0800, Paul Ramsey wrote:
> I just ran across this change when a bunch of CI´s I run barfed.
Oops, sorry.
> The problem we are having in extension land is that we often run
> functions in external libraries that take a long time to return, and we
> would like to ensure that PgSQL users can cancel their queries, even when
> control has passed into those functions.
>
> The way we have done it, historically, has been to take the return value
> of pqsignal(SIGINT, extension_signint_handler) and remember it, and then,
> inside extension_signint_handler, call the pgsql handler once we have
> done our own business..
I see a couple other projects that might be doing something similar [0].
> It is possible we have been Doing It Wrong all this time, and would love
> some pointers on the right way to do this.
>
> Alternatively, if we have a valid use case, it would be nice to have the
> return value from pqsignal() back.
I don't think you were doing it wrong before, but commit 3b00fdb did add
some weird race conditions to the return value, but only in rare cases that
I don't think apply to the case you shared.
I was tempted to suggest that you try using sigaction() with the second
"act" argument set to NULL to retrieve the original signal handler, but I
think there's a good chance you'd get our wrapper_handler() function (we
store the actual handlers in a separate array), which is no good. So, I
should probably just revert d4a43b2. An alternative approach could be to
provide our own way to retrieve the current handler, but that comes with
its own race conditions, although perhaps those are more obvious than the
pqsignal() ones. WDYT?
[0] https://codesearch.debian.net/search?q=%3D+pqsignal&literal=1
--
nathan
^ permalink raw reply [nested|flat] 34+ messages in thread
end of thread, other threads:[~2025-01-22 16:22 UTC | newest]
Thread overview: 34+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2017-09-12 16:07 pgbench regression test failure Tom Lane <[email protected]>
2017-09-12 17:08 ` Fabien COELHO <[email protected]>
2017-09-12 17:13 ` Tom Lane <[email protected]>
2017-09-12 18:00 ` Fabien COELHO <[email protected]>
2017-09-12 18:12 ` Tom Lane <[email protected]>
2017-09-12 19:21 ` Fabien COELHO <[email protected]>
2017-09-12 19:28 ` Tom Lane <[email protected]>
2017-09-13 09:42 ` Fabien COELHO <[email protected]>
2017-09-22 17:02 ` Tom Lane <[email protected]>
2017-09-22 17:55 ` Fabien COELHO <[email protected]>
2017-09-23 17:06 ` Tom Lane <[email protected]>
2017-09-24 07:17 ` Fabien COELHO <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2021-01-13 11:05 [PATCH v3 1/2] Fix ALTER PUBLICATION...DROP TABLE behaviour Bharath Rupireddy <[email protected]>
2025-01-22 16:22 Re: Converting pqsignal to void return Nathan Bossart <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox