pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedconchuela timeouts since 2021-10-09 system upgrade
40+ messages / 5 participants
[nested] [flat]
Commits from this thread:
2021-10-28 7f580aa5d8 141cd0ef0b 5a4b8a8a72 b1f943d2aa Improve contrib/amcheck's tests for CREATE INDEX CONCURRENTLY.
* conchuela timeouts since 2021-10-09 system upgrade
@ 2021-10-24 16:19 Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 40+ messages in thread
From: Noah Misch @ 2021-10-24 16:19 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Sun, Oct 24, 2021 at 02:45:38PM +0300, Andrey Borodin wrote:
> > 24 окт. 2021 г., в 08:00, Noah Misch <noah@leadboat.com> написал(а):
> > Buildfarm member conchuela (DragonFly BSD 6.0) has gotten multiple
> > "IPC::Run: timeout on timer" in the new tests. No other animal has.
> > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=conchuela&dt=2021-10-24%2003%3A05%3A09
> > is an example run. The pgbench queries finished quickly, but the
> > $pgbench_h->finish() apparently timed out after 180s. I guess this would be
> > consistent with pgbench blocking in write(), waiting for something to empty a
> > pipe buffer so it can write more. I thought finish() will drain any incoming
> > I/O, though. This phenomenon has been appearing regularly via
> > src/test/recovery/t/017_shm.pl[1], so this thread doesn't have a duty to
> > resolve it. A stack trace of the stuck pgbench should be informative, though.
>
> Some thoughts:
> 0. I doubt that psql\pgbench is stuck in these failures.
Got it. If pgbench is a zombie, the fault does lie in IPC::Run or the kernel.
> 1. All observed similar failures seem to be related to finish() sub of IPC::Run harness
> 2. Finish must pump any pending data from process [0]. But it can hang if process is waiting for something.
> 3. There is reported bug of finish [1]. But the description is slightly different.
Since that report is about a Perl-process child on Linux, I think we can treat
it as unrelated.
These failures started on 2021-10-09, the day conchuela updated from DragonFly
v4.4.3-RELEASE to DragonFly v6.0.0-RELEASE. It smells like a kernel bug.
Since the theorized kernel bug seems not to affect
src/test/subscription/t/015_stream.pl, I wonder if we can borrow a workaround
from other tests. One thing in common with src/test/recovery/t/017_shm.pl and
the newest failure sites is that they don't write anything to the child stdin.
Does writing e.g. a single byte (that the child doesn't use) work around the
problem? If not, does passing the script via stdin, like "pgbench -f-
<script.sql", work around the problem?
> [0] https://metacpan.org/dist/IPC-Run/source/lib/IPC/Run.pm#L3481
> [1] https://github.com/toddr/IPC-Run/issues/57
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
@ 2021-10-24 18:03 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-24 23:53 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
0 siblings, 2 replies; 40+ messages in thread
From: Andrey Borodin @ 2021-10-24 18:03 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
> 24 окт. 2021 г., в 19:19, Noah Misch <noah@leadboat.com> написал(а):
>
> These failures started on 2021-10-09, the day conchuela updated from DragonFly
> v4.4.3-RELEASE to DragonFly v6.0.0-RELEASE. It smells like a kernel bug.
> Since the theorized kernel bug seems not to affect
> src/test/subscription/t/015_stream.pl, I wonder if we can borrow a workaround
> from other tests. One thing in common with src/test/recovery/t/017_shm.pl and
> the newest failure sites is that they don't write anything to the child stdin.
> Does writing e.g. a single byte (that the child doesn't use) work around the
> problem?
Following diff did not solve the problem on my VM, finish() still hangs sometimes.
inline patch 1: src/test/perl/PostgreSQL/Test/Cluster.pm (+1/-0 lines) download inline-1.patch
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 86eb920ea1..0c550ff0dc 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -2112,6 +2112,7 @@ sub background_pgbench
my $harness = IPC::Run::start \@cmd, '<', \$stdin, '>', $stdout, '2>&1',
$timer;
+ $stdin .= "some bytes";
return $harness;
}
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-10-24 23:53 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-10-25 08:30 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 1 reply; 40+ messages in thread
From: Tom Lane @ 2021-10-24 23:53 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Noah Misch <noah@leadboat.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
Andrey Borodin <x4mmm@yandex-team.ru> writes:
>> 24 окт. 2021 г., в 19:19, Noah Misch <noah@leadboat.com> написал(а):
>> These failures started on 2021-10-09, the day conchuela updated from DragonFly
>> v4.4.3-RELEASE to DragonFly v6.0.0-RELEASE. It smells like a kernel bug.
prairiedog just reported the identical symptom:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prairiedog&dt=2021-10-24%2016%3A05%3A58
I'm thinking "timing problem" more than "kernel bug". (I'd have also
believed "ancient IPC::Run bug", except that I assume conchuela is
running a pretty modern IPC::Run.)
I can poke into it tomorrow.
regards, tom lane
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-24 23:53 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
@ 2021-10-25 08:30 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-25 20:59 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 40+ messages in thread
From: Andrey Borodin @ 2021-10-25 08:30 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Noah Misch <noah@leadboat.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
> 25 окт. 2021 г., в 02:53, Tom Lane <tgl@sss.pgh.pa.us> написал(а):
>
> I can poke into it tomorrow.
FWIW it's easy to make the issue reproduce faster with following diff
inline patch 1: contrib/amcheck/t/003_cic_2pc.pl (+1/-1 lines) download inline-1.patch
diff --git a/contrib/amcheck/t/003_cic_2pc.pl b/contrib/amcheck/t/003_cic_2pc.pl
index f4255c1fb8..0d68c53387 100644
--- a/contrib/amcheck/t/003_cic_2pc.pl
+++ b/contrib/amcheck/t/003_cic_2pc.pl
@@ -141,7 +141,7 @@ $node->safe_psql('postgres', q(REINDEX TABLE tbl;));
my $pgbench_out = '';
my $pgbench_timer = IPC::Run::timeout(180);
my $pgbench_h = $node->background_pgbench(
- '--no-vacuum --client=1 --transactions=100',
+ '--no-vacuum --client=1 --transactions=1',
{
'002_pgbench_concurrent_cic' => q(
DROP INDEX CONCURRENTLY idx;
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-24 23:53 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-25 08:30 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-10-25 20:59 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 01:51 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 40+ messages in thread
From: Tom Lane @ 2021-10-25 20:59 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Noah Misch <noah@leadboat.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
Andrey Borodin <x4mmm@yandex-team.ru> writes:
> FWIW it's easy to make the issue reproduce faster with following diff
> - '--no-vacuum --client=1 --transactions=100',
> + '--no-vacuum --client=1 --transactions=1',
Hmm, didn't help here. It seems that even though prairiedog managed to
fail on its first attempt, it's not terribly reproducible there; I've
seen only one failure in about 30 manual attempts. In the one failure,
the non-background pgbench completed fine (as determined by counting
statements in the postmaster's log); but the background one had only
finished about 90 transactions before seemingly getting stuck. No new
SQL commands had been issued after about 10 seconds.
Nonetheless, I have a theory and a proposal. This coding pattern
seems pretty silly:
$pgbench_h->pump_nb;
$pgbench_h->finish();
ISTM that if you need to call pump at all, you need a loop not just
one call. So I'm guessing that when it fails, it's for lack of
pumping.
The other thing I noticed is that at least on prairiedog's host, the
number of invocations of the DROP/CREATE/bt_index_check transaction
is ridiculously out of proportion to the number of invocations of the
other transactions. It can only get through seven or eight iterations
of the index transaction before the other transactions are all done,
which means the last 190 iterations of that transaction are a complete
waste of cycles.
What I think we should do in these two tests is nuke the use of
background_pgbench entirely; that looks like a solution in search
of a problem, and it seems unnecessary here. Why not run
the DROP/CREATE/bt_index_check transaction as one of three script
options in the main pgbench run? Aside from dodging this
maybe-its-a-bug-or-maybe-not behavior in IPC::Run, this would make the
test automatically scale the number of iterations of the different
transactions to appropriate values, so that we'd not be wasting cycles.
regards, tom lane
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-24 23:53 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-25 08:30 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-25 20:59 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
@ 2021-10-26 01:51 ` Noah Misch <noah@leadboat.com>
2021-10-26 06:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 40+ messages in thread
From: Noah Misch @ 2021-10-26 01:51 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Mon, Oct 25, 2021 at 04:59:42PM -0400, Tom Lane wrote:
> Andrey Borodin <x4mmm@yandex-team.ru> writes:
> > FWIW it's easy to make the issue reproduce faster with following diff
> > - '--no-vacuum --client=1 --transactions=100',
> > + '--no-vacuum --client=1 --transactions=1',
>
> Hmm, didn't help here. It seems that even though prairiedog managed to
> fail on its first attempt, it's not terribly reproducible there; I've
> seen only one failure in about 30 manual attempts. In the one failure,
> the non-background pgbench completed fine (as determined by counting
> statements in the postmaster's log); but the background one had only
> finished about 90 transactions before seemingly getting stuck. No new
> SQL commands had been issued after about 10 seconds.
Interesting.
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prairiedog&dt=2021-10-24%2016%3A05%3A58
also shows a short command count, just 131/200 completed. However,
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=conchuela&dt=2021-10-25%2000%3A35%3A27
shows the full 200/200. I'm starting to think the prairiedog failures have
only superficial similarity to the conchuela failures.
> Nonetheless, I have a theory and a proposal. This coding pattern
> seems pretty silly:
>
> $pgbench_h->pump_nb;
> $pgbench_h->finish();
>
> ISTM that if you need to call pump at all, you need a loop not just
> one call. So I'm guessing that when it fails, it's for lack of
> pumping.
The pump_nb() is just unnecessary. We've not added anything destined for
stdin, and finish() takes care of pumping outputs.
> The other thing I noticed is that at least on prairiedog's host, the
> number of invocations of the DROP/CREATE/bt_index_check transaction
> is ridiculously out of proportion to the number of invocations of the
> other transactions. It can only get through seven or eight iterations
> of the index transaction before the other transactions are all done,
> which means the last 190 iterations of that transaction are a complete
> waste of cycles.
That makes sense.
> What I think we should do in these two tests is nuke the use of
> background_pgbench entirely; that looks like a solution in search
> of a problem, and it seems unnecessary here. Why not run
> the DROP/CREATE/bt_index_check transaction as one of three script
> options in the main pgbench run?
The author tried that and got deadlocks:
https://postgr.es/m/5E041A70-4946-489C-9B6D-764DF627A92D@yandex-team.ru
On prairiedog, the proximate trouble is pgbench getting stuck. IPC::Run is
behaving normally given a stuck pgbench. When pgbench stops sending queries,
does pg_stat_activity show anything at all running? If so, are those backends
waiting on locks? If not, what's the pgbench stack trace at that time?
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-24 23:53 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-25 08:30 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-25 20:59 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 01:51 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
@ 2021-10-26 06:03 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 13:45 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 40+ messages in thread
From: Tom Lane @ 2021-10-26 06:03 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
Noah Misch <noah@leadboat.com> writes:
> On Mon, Oct 25, 2021 at 04:59:42PM -0400, Tom Lane wrote:
>> What I think we should do in these two tests is nuke the use of
>> background_pgbench entirely; that looks like a solution in search
>> of a problem, and it seems unnecessary here. Why not run
>> the DROP/CREATE/bt_index_check transaction as one of three script
>> options in the main pgbench run?
> The author tried that and got deadlocks:
> https://postgr.es/m/5E041A70-4946-489C-9B6D-764DF627A92D@yandex-team.ru
Hmm, I guess that's because two concurrent CICs can deadlock against each
other. I wonder if we could fix that ... or maybe we could teach pgbench
that it mustn't launch more than one instance of that script? Or more
practically, use advisory locks in that script to enforce that only one
runs at once.
> On prairiedog, the proximate trouble is pgbench getting stuck. IPC::Run is
> behaving normally given a stuck pgbench. When pgbench stops sending queries,
> does pg_stat_activity show anything at all running? If so, are those backends
> waiting on locks? If not, what's the pgbench stack trace at that time?
The connected backend is idle, waiting for client input:
#0 0x9002ec88 in kevent ()
#1 0x003e3fa8 in WaitEventSetWait (set=0x3802258, timeout=-1, occurred_events=0xbfffe098, nevents=1, wait_event_info=0) at latch.c:1601
#2 0x0028d6a0 in secure_read (port=0x2501f40, ptr=0x7ad1d0, len=8192) at be-secure.c:186
#3 0x002958a4 in pq_recvbuf () at pqcomm.c:957
#4 0x00295994 in pq_getbyte () at pqcomm.c:1000
#5 0x004196a0 in PostgresMain
The last few entries in the postmaster log are:
2021-10-26 01:19:31.122 EDT [1013] 002_cic.pl LOG: statement: DROP INDEX CONCURRENTLY idx;
2021-10-26 01:19:31.136 EDT [1013] 002_cic.pl LOG: statement: CREATE INDEX CONCURRENTLY idx ON tbl(i);
2021-10-26 01:19:31.167 EDT [1013] 002_cic.pl LOG: statement: SELECT bt_index_check('idx',true);
which is consistent with pg_stat_activity:
datid | datname | pid | leader_pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | xact_start | query_start | state_change
| wait_event_type | wait_event | state | backend_xid | backend_xmin | query_id | query | backend_type
-------+----------+------+------------+----------+---------+------------------+-------------+-----------------+-------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------
+-----------------+---------------------+--------+-------------+--------------+----------+------------------------------------+------------------------------
13019 | postgres | 1013 | | 10 | tgl | 002_cic.pl | | | -1 | 2021-10-26 01:19:18.49282-04 | | 2021-10-26 01:19:31.167508-04 | 2021-10-26 01:19:31.190256-04
| Client | ClientRead | idle | | | | SELECT bt_index_check('idx',true); | client backend
The pgbench process is sitting here:
#0 0x9001f488 in select ()
#1 0x0000758c in wait_on_socket_set (sa=0x300bb0, usecs=0) at pgbench.c:7098
#2 0x0000d514 in threadRun (arg=0x300420) at pgbench.c:6776
#3 0x00010c44 in main (argc=-1635084506, argv=0x0) at pgbench.c:6540
which means it thinks it's waiting for input. The PGconn object is in
this state:
status = CONNECTION_OK,
asyncStatus = PGASYNC_BUSY,
xactStatus = PQTRANS_IDLE,
...
inBuffer = 0x6a000 "T",
inBufSize = 16384,
inStart = 0,
inCursor = 0,
inEnd = 0,
outBuffer = 0x6e000 "Q",
outBufSize = 16384,
outCount = 0,
outMsgStart = 1,
outMsgEnd = 34,
The inBuffer is logically empty, but its contents look like the last input
was the result of the "SELECT bt_index_check" command, consistent with
the postmaster log. The outBuffer is likewise logically empty, but what
it contains appears to be the next DROP INDEX command:
(gdb) x/64c state->con->outBuffer
0x6e000: 81 'Q' 0 '\0' 0 '\0' 0 '\0' 33 '!' 68 'D' 82 'R' 79 'O'
0x6e008: 80 'P' 32 ' ' 73 'I' 78 'N' 68 'D' 69 'E' 88 'X' 32 ' '
0x6e010: 67 'C' 79 'O' 78 'N' 67 'C' 85 'U' 82 'R' 82 'R' 69 'E'
0x6e018: 78 'N' 84 'T' 76 'L' 89 'Y' 32 ' ' 105 'i' 100 'd' 120 'x'
0x6e020: 59 ';' 0 '\0' 114 'r' 117 'u' 101 'e' 41 ')' 59 ';' 0 '\0'
0x6e028: 108 'l' 40 '(' 105 'i' 41 ')' 59 ';' 0 '\0' 95 '_' 110 'n'
0x6e030: 97 'a' 109 'm' 101 'e' 0 '\0' 48 '0' 48 '0' 50 '2' 95 '_'
0x6e038: 99 'c' 105 'i' 99 'c' 46 '.' 112 'p' 108 'l' 0 '\0' 0 '\0'
So what we have is that libpq thinks it's sent the next DROP INDEX,
but the backend hasn't seen it.
It's fairly hard to blame that state of affairs on the IPC::Run harness.
I'm wondering if we might be looking at some timing-dependent corner-case
bug in the new libpq pipelining code. Pipelining isn't enabled:
pipelineStatus = PQ_PIPELINE_OFF,
but that doesn't mean that the pipelining code hasn't been anywhere
near this command. I can see
cmd_queue_head = 0x300d40,
cmd_queue_tail = 0x300d40,
cmd_queue_recycle = 0x0,
(gdb) p *state->con->cmd_queue_head
$4 = {
queryclass = PGQUERY_SIMPLE,
query = 0x3004e0 "DROP INDEX CONCURRENTLY idx;",
next = 0x0
}
The trouble with this theory, of course, is "if libpq is busted, why is
only this test case showing it?". But AFAICS it would take some pretty
spooky action-at-a-distance for the Perl harness to have caused this.
regards, tom lane
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-24 23:53 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-25 08:30 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-25 20:59 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 01:51 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-26 06:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
@ 2021-10-26 13:45 ` Noah Misch <noah@leadboat.com>
2021-10-26 14:29 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 19:41 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 40+ messages in thread
From: Noah Misch @ 2021-10-26 13:45 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Tue, Oct 26, 2021 at 02:03:54AM -0400, Tom Lane wrote:
> Noah Misch <noah@leadboat.com> writes:
> > On Mon, Oct 25, 2021 at 04:59:42PM -0400, Tom Lane wrote:
> >> What I think we should do in these two tests is nuke the use of
> >> background_pgbench entirely; that looks like a solution in search
> >> of a problem, and it seems unnecessary here. Why not run
> >> the DROP/CREATE/bt_index_check transaction as one of three script
> >> options in the main pgbench run?
>
> > The author tried that and got deadlocks:
> > https://postgr.es/m/5E041A70-4946-489C-9B6D-764DF627A92D@yandex-team.ru
>
> Hmm, I guess that's because two concurrent CICs can deadlock against each
> other. I wonder if we could fix that ... or maybe we could teach pgbench
> that it mustn't launch more than one instance of that script?
Both sound doable, but I don't expect either to fix prairiedog's trouble.
> Or more
> practically, use advisory locks in that script to enforce that only one
> runs at once.
The author did try that.
> So what we have is that libpq thinks it's sent the next DROP INDEX,
> but the backend hasn't seen it.
Thanks for isolating that.
> It's fairly hard to blame that state of affairs on the IPC::Run harness.
> I'm wondering if we might be looking at some timing-dependent corner-case
> bug in the new libpq pipelining code. Pipelining isn't enabled:
>
> pipelineStatus = PQ_PIPELINE_OFF,
>
> but that doesn't mean that the pipelining code hasn't been anywhere
> near this command. I can see
>
> cmd_queue_head = 0x300d40,
> cmd_queue_tail = 0x300d40,
> cmd_queue_recycle = 0x0,
>
> (gdb) p *state->con->cmd_queue_head
> $4 = {
> queryclass = PGQUERY_SIMPLE,
> query = 0x3004e0 "DROP INDEX CONCURRENTLY idx;",
> next = 0x0
> }
>
> The trouble with this theory, of course, is "if libpq is busted, why is
> only this test case showing it?".
Agreed, it's not clear how the new tests would reveal a libpq bug that
src/bin/pgbench/t/001_pgbench_with_server.pl has been unable to reveal. Does
the problem reproduce on v13?
Grasping at straws, background_pgbench does differ by specifying stdin as a
ref to an empty scalar. I think that makes IPC::Run open a pipe and never
write to it. The older pgbench tests don't override stdin, so I think that
makes pgbench inherit the original stdin. Given your pgbench stack trace,
this seems awfully unlikely to be the relevant difference. If we run out of
ideas, you could try some runs with that difference removed:
inline patch 1: src/test/perl/PostgreSQL/Test/Cluster.pm (+1/-1 lines) download inline-1.patch
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -2110,7 +2110,7 @@ sub background_pgbench
# IPC::Run would otherwise append to existing contents:
$$stdout = "" if ref($stdout);
- my $harness = IPC::Run::start \@cmd, '<', \$stdin, '>', $stdout, '2>&1',
+ my $harness = IPC::Run::start \@cmd, '>', $stdout, '2>&1',
$timer;
return $harness;
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-24 23:53 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-25 08:30 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-25 20:59 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 01:51 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-26 06:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 13:45 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
@ 2021-10-26 14:29 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 20:45 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
1 sibling, 1 reply; 40+ messages in thread
From: Tom Lane @ 2021-10-26 14:29 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
Noah Misch <noah@leadboat.com> writes:
> On Tue, Oct 26, 2021 at 02:03:54AM -0400, Tom Lane wrote:
>> Or more
>> practically, use advisory locks in that script to enforce that only one
>> runs at once.
> The author did try that.
Hmm ... that ought to have done the trick, I'd think. However:
> Both sound doable, but I don't expect either to fix prairiedog's trouble.
Yeah :-(. I think this test is somehow stumbling over a pre-existing bug.
>> So what we have is that libpq thinks it's sent the next DROP INDEX,
>> but the backend hasn't seen it.
> Thanks for isolating that.
The plot thickens. When I went back to look at that machine this morning,
I found this in the postmaster log:
2021-10-26 02:52:09.324 EDT [1013] 002_cic.pl LOG: statement: DROP INDEX CONCURRENTLY idx;
2021-10-26 02:52:09.352 EDT [1013] 002_cic.pl LOG: could not send data to client: Broken pipe
2021-10-26 02:52:09.352 EDT [1013] 002_cic.pl FATAL: connection to client lost
The timestamps correspond (more or less anyway) to when I killed off the
stuck test run and went to bed. So the DROP command *was* sent, and it
was eventually received by the backend, but it seems to have taken killing
the pgbench process to do it.
I think this probably exonerates the pgbench/libpq side of things, and
instead we have to wonder about a backend or kernel bug. A kernel bug
could possibly explain the unexplainable connection to what's happening on
some other file descriptor. I'd be prepared to believe that prairiedog's
ancient macOS version has some weird bug preventing kevent() from noticing
available data ... but (a) surely conchuela wouldn't share such a bug,
and (b) we've been using kevent() for a couple years now, so how come
we didn't see this before?
Still baffled. I'm currently experimenting to see if the bug reproduces
when latch.c is made to use poll() instead of kevent(). But the failure
rate was low enough that it'll be hours before I can say confidently
that it doesn't (unless, of course, it does).
regards, tom lane
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-24 23:53 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-25 08:30 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-25 20:59 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 01:51 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-26 06:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 13:45 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-26 14:29 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
@ 2021-10-26 20:45 ` Thomas Munro <thomas.munro@gmail.com>
2021-10-27 00:53 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 40+ messages in thread
From: Thomas Munro @ 2021-10-26 20:45 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Noah Misch <noah@leadboat.com>; Andrey Borodin <x4mmm@yandex-team.ru>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Wed, Oct 27, 2021 at 3:29 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> [...] I'd be prepared to believe that prairiedog's
> ancient macOS version has some weird bug preventing kevent() from noticing
> available data ... but (a) surely conchuela wouldn't share such a bug,
> and (b) we've been using kevent() for a couple years now, so how come
> we didn't see this before?
There was this case soon after our kqueue support landed:
https://www.postgresql.org/message-id/CA%2BhUKGLzaR5cV0EmZWoVXJDO_XwZpmpQX_sYwCBRE1qLBEcGPQ%40mail.g...
There are a few discussions on the 'net about the flakiness of both
kevent() and poll() around that vintage of macOS (both were new and
shared infrastructure, separate from select()); for example in libcurl
and libevent talked about this and blocked version ranges.
I don't have any ideas about conchuela. For the next person who
manages to reproduce this, just to sanity-check what we're passing in
to kevent(), what do *port and waitfor look like when secure_read()
blocks in WaitEventSetWait? It's good news that Andrey could
reproduce this on a VM. I may look into setting one of those up too.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-24 23:53 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-25 08:30 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-25 20:59 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 01:51 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-26 06:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 13:45 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-26 14:29 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 20:45 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
@ 2021-10-27 00:53 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 40+ messages in thread
From: Tom Lane @ 2021-10-27 00:53 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; Andrey Borodin <x4mmm@yandex-team.ru>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
Thomas Munro <thomas.munro@gmail.com> writes:
> On Wed, Oct 27, 2021 at 3:29 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> [...] I'd be prepared to believe that prairiedog's
>> ancient macOS version has some weird bug preventing kevent() from noticing
>> available data ... but (a) surely conchuela wouldn't share such a bug,
>> and (b) we've been using kevent() for a couple years now, so how come
>> we didn't see this before?
> There was this case soon after our kqueue support landed:
> https://www.postgresql.org/message-id/CA%2BhUKGLzaR5cV0EmZWoVXJDO_XwZpmpQX_sYwCBRE1qLBEcGPQ%40mail.g...
Oh yeah ... that looks like the exact same thing, doesn't it?
I've just finished doing 500 cycles of amcheck's 002_cic.pl without a
failure after recompiling latch.c with -DWAIT_USE_POLL. With the kevent
code path, it is hard to reproduce but not that hard. So unless we can
learn something from the DragonFly case, I'm inclined to write this off
as ancient-macOS-bug and start using -DWAIT_USE_POLL on prairiedog.
> I don't have any ideas about conchuela. For the next person who
> manages to reproduce this, just to sanity-check what we're passing in
> to kevent(), what do *port and waitfor look like when secure_read()
> blocks in WaitEventSetWait?
Not sure which variables you're referring to there, but here's a more
complete gdb dump from the stuck backend:
(gdb) bt
#0 0x9002ec88 in kevent ()
#1 0x003e3fa8 in WaitEventSetWait (set=0x3802258, timeout=-1, occurred_events=0xbfffe098, nevents=1, wait_event_info=0) at latch.c:1601
#2 0x0028d6a0 in secure_read (port=0x2501f40, ptr=0x7ad1d0, len=8192) at be-secure.c:186
#3 0x002958a4 in pq_recvbuf () at pqcomm.c:957
#4 0x00295994 in pq_getbyte () at pqcomm.c:1000
#5 0x004196a0 in PostgresMain (dbname=0x3805894 "", username=0x789020 "...") at postgres.c:353
#6 0x00363df8 in PostmasterMain (argc=8059652, argv=0x61789299) at postmaster.c:4560
#7 0x00298b6c in main (argc=4, argv=0x2500a30) at main.c:198
(gdb) f 1
#1 0x003e3fa8 in WaitEventSetWait (set=0x3802258, timeout=-1, occurred_events=0xbfffe098, nevents=1, wait_event_info=0) at latch.c:1601
1601 rc = kevent(set->kqueue_fd, NULL, 0,
(gdb) i locals
cur_event = (WaitEvent *) 0x78ed40
cur_kqueue_event = (struct kevent *) 0x0
timeout = {
tv_sec = 13,
tv_nsec = 0
}
rc = 4
timeout_p = (struct timespec *) 0x0
rc = 4
returned_events = 4
start_time = {
tv_sec = -1879041144,
tv_usec = 58742932
}
cur_time = {
tv_sec = 59028992,
tv_usec = 2
}
cur_timeout = -1
(gdb) p *set
$1 = {
nevents = 3,
nevents_space = 3,
events = 0x380227c,
latch = 0xc954ebc,
latch_pos = 1,
exit_on_postmaster_death = 0 '\0',
kqueue_fd = 4,
kqueue_ret_events = 0x38022ac,
report_postmaster_not_running = 0 '\0'
}
(gdb) p set->events[0]
$2 = {
pos = 0,
events = 2,
fd = 7, # matches MyProcPort->sock
user_data = 0x0
}
(gdb) p set->events[1]
$3 = {
pos = 1,
events = 1,
fd = -1,
user_data = 0x0
}
(gdb) p set->events[2]
$4 = {
pos = 2,
events = 16,
fd = 3,
user_data = 0x0
}
(gdb) p set->kqueue_ret_events[0]
$6 = {
ident = 7, # matches MyProcPort->sock
filter = -1, # EVFILT_READ
flags = 1, # EV_ADD
fflags = 0,
data = 40,
udata = 0x380227c
}
So AFAICS, the data we're passing to kevent() is all sane.
It then occurred to me to replicate the case I saw this morning
(control-C'ing the test script) and ktrace all the involved processes.
The perl test script and the pgbench process both just immediately
die on SIGINT:
25500 pgbench RET select -1 errno 4 Interrupted system call
25500 pgbench PSIG SIGINT SIG_DFL
so they are clearly not doing anything to cause the DROP command
to get delivered. Nonetheless, what the backend sees post-control-C is
25502 postgres RET kevent 1
25502 postgres CALL recvfrom(0x7,0x7ad1d0,0x2000,0,0,0)
25502 postgres GIO fd 7 wrote 34 bytes
"Q\0\0\0!DROP INDEX CONCURRENTLY idx;\0"
25502 postgres RET recvfrom 34/0x22
... blah, blah, executing the command ...
25502 postgres CALL sendto(0x7,0x241d020,0x16,0,0,0)
25502 postgres RET sendto -1 errno 32 Broken pipe
... blah, blah, shutting down ...
It's really hard to look at this and not call it a kernel bug.
regards, tom lane
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-24 23:53 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-25 08:30 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-25 20:59 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 01:51 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-26 06:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 13:45 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
@ 2021-10-26 19:41 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-10-27 03:01 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-27 11:08 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 2 replies; 40+ messages in thread
From: Tom Lane @ 2021-10-26 19:41 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
Noah Misch <noah@leadboat.com> writes:
> On Tue, Oct 26, 2021 at 02:03:54AM -0400, Tom Lane wrote:
>> Or more
>> practically, use advisory locks in that script to enforce that only one
>> runs at once.
> The author did try that.
Ah, I see: if the other pgbench thread is waiting in pg_advisory_lock,
then it is inside a transaction, so it blocks CIC from completing.
We can get around that though, by using pg_try_advisory_lock and not
proceeding if it fails. The attached POC does this for the 002 test;
it looks like the same thing could be done to 003.
Now the problem with this is it will only work back to v12, because
pgbench lacks the necessary features before that. However, I think
it's worth doing it like this in versions where we can do so, because
of the load-balancing aspect: this won't waste cycles running CICs
after the inserts have stopped, nor vice versa.
Thoughts?
regards, tom lane
Attachments:
CIC-test-with-one-pgbench.patch (text/x-diff, 2.1K) download
diff --git a/contrib/amcheck/t/002_cic.pl b/contrib/amcheck/t/002_cic.pl
index 0b14e66270..1f258dd59d 100644
--- a/contrib/amcheck/t/002_cic.pl
+++ b/contrib/amcheck/t/002_cic.pl
@@ -9,7 +9,7 @@ use Config;
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
-use Test::More tests => 4;
+use Test::More tests => 3;
my ($node, $result);
@@ -25,32 +25,18 @@ $node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
$node->safe_psql('postgres', q(CREATE INDEX idx ON tbl(i)));
#
-# Stress CIC with pgbench
+# Stress CIC with pgbench.
+#
+# pgbench might try to launch more than one instance of the CIC
+# transaction concurrently. That would deadlock, so use an advisory
+# lock to ensure only one CIC runs at a time.
#
-
-# Run background pgbench with CIC. We cannot mix-in this script into single
-# pgbench: CIC will deadlock with itself occasionally.
-my $pgbench_out = '';
-my $pgbench_timer = IPC::Run::timeout(180);
-my $pgbench_h = $node->background_pgbench(
- '--no-vacuum --client=1 --transactions=200',
- {
- '002_pgbench_concurrent_cic' => q(
- DROP INDEX CONCURRENTLY idx;
- CREATE INDEX CONCURRENTLY idx ON tbl(i);
- SELECT bt_index_check('idx',true);
- )
- },
- \$pgbench_out,
- $pgbench_timer);
-
-# Run pgbench.
$node->pgbench(
'--no-vacuum --client=5 --transactions=200',
0,
[qr{actually processed}],
[qr{^$}],
- 'concurrent INSERTs',
+ 'concurrent INSERTs and CIC',
{
'002_pgbench_concurrent_transaction' => q(
BEGIN;
@@ -62,17 +48,17 @@ $node->pgbench(
SAVEPOINT s1;
INSERT INTO tbl VALUES(0);
COMMIT;
+ ),
+ '002_pgbench_concurrent_cic' => q(
+ SELECT pg_try_advisory_lock(42)::integer AS gotlock \gset
+ \if :gotlock
+ DROP INDEX CONCURRENTLY idx;
+ CREATE INDEX CONCURRENTLY idx ON tbl(i);
+ SELECT bt_index_check('idx',true);
+ SELECT pg_advisory_unlock(42);
+ \endif
)
});
-$pgbench_h->pump_nb;
-$pgbench_h->finish();
-$result =
- ($Config{osname} eq "MSWin32")
- ? ($pgbench_h->full_results)[0]
- : $pgbench_h->result(0);
-is($result, 0, "pgbench with CIC works");
-
-# done
$node->stop;
done_testing();
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-24 23:53 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-25 08:30 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-25 20:59 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 01:51 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-26 06:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 13:45 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-26 19:41 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
@ 2021-10-27 03:01 ` Noah Misch <noah@leadboat.com>
2021-10-27 03:20 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 40+ messages in thread
From: Noah Misch @ 2021-10-27 03:01 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Tue, Oct 26, 2021 at 03:41:58PM -0400, Tom Lane wrote:
> Noah Misch <noah@leadboat.com> writes:
> > On Tue, Oct 26, 2021 at 02:03:54AM -0400, Tom Lane wrote:
> >> Or more
> >> practically, use advisory locks in that script to enforce that only one
> >> runs at once.
>
> > The author did try that.
>
> Ah, I see: if the other pgbench thread is waiting in pg_advisory_lock,
> then it is inside a transaction, so it blocks CIC from completing.
> We can get around that though, by using pg_try_advisory_lock and not
> proceeding if it fails.
Good thought.
> The attached POC does this for the 002 test;
> it looks like the same thing could be done to 003.
>
> Now the problem with this is it will only work back to v12, because
> pgbench lacks the necessary features before that. However, I think
> it's worth doing it like this in versions where we can do so, because
> of the load-balancing aspect: this won't waste cycles running CICs
> after the inserts have stopped, nor vice versa.
>
> Thoughts?
I tried it out with the fix removed (git checkout fdd965d; git checkout HEAD^
src/include src/backend; git apply CIC-test-with-one-pgbench.patch).
Sensitivity (~25%) and runtime (~900ms) were in the same ballpark. Still,
even if it doesn't buy back notable cycles, the test code seems easier to read
and understand with your change.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-24 23:53 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-25 08:30 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-25 20:59 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 01:51 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-26 06:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 13:45 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-26 19:41 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-27 03:01 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
@ 2021-10-27 03:20 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 40+ messages in thread
From: Tom Lane @ 2021-10-27 03:20 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
Noah Misch <noah@leadboat.com> writes:
> On Tue, Oct 26, 2021 at 03:41:58PM -0400, Tom Lane wrote:
>> Now the problem with this is it will only work back to v12, because
>> pgbench lacks the necessary features before that. However, I think
>> it's worth doing it like this in versions where we can do so, because
>> of the load-balancing aspect: this won't waste cycles running CICs
>> after the inserts have stopped, nor vice versa.
> I tried it out with the fix removed (git checkout fdd965d; git checkout HEAD^
> src/include src/backend; git apply CIC-test-with-one-pgbench.patch).
> Sensitivity (~25%) and runtime (~900ms) were in the same ballpark. Still,
> even if it doesn't buy back notable cycles, the test code seems easier to read
> and understand with your change.
The point wasn't to reduce the runtime: it was to get more useful test
iterations in the same runtime. A test iteration that runs CIC while
there are no concurrent inserts isn't providing any useful coverage.
It's possible that with this patch, we could dial it back to less
than 1000 (200x5) total transactions and still get the same effective
number of test iterations as we have in HEAD. I didn't really
look into that. I did notice that even on a much faster machine
than prairiedog, HEAD wastes at least three-quarters of the CIC
iterations.
regards, tom lane
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-24 23:53 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-25 08:30 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-25 20:59 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 01:51 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-26 06:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 13:45 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-26 19:41 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
@ 2021-10-27 11:08 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-28 15:47 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 40+ messages in thread
From: Andrey Borodin @ 2021-10-27 11:08 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Noah Misch <noah@leadboat.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
> 26 окт. 2021 г., в 22:41, Tom Lane <tgl@sss.pgh.pa.us> написал(а):
>
> We can get around that though, by using pg_try_advisory_lock and not
> proceeding if it fails. The attached POC does this for the 002 test;
> it looks like the same thing could be done to 003.
That's a neat idea. PFA patch with copy of this changes to 003.
Also I've added a script to stress not only CREATE INDEX CONCURRENTLY, but also REINDEX CONCURRENTLY.
Because it was easy and checks slightly more code paths.
I do not observe failures on Dragonfly 6.0 with the patch, but I didn't run for a long yet time.
Best regards, Andrey Borodin.
Attachments:
v2-0001-CIC-test-without-background-pgbench.patch (application/octet-stream, 5.3K) download
From b4487f79b91632c22bb0acdf12616406bf342e08 Mon Sep 17 00:00:00 2001
From: Andrey Borodin <amborodin@acm.org>
Date: Wed, 27 Oct 2021 13:41:15 +0300
Subject: [PATCH v2] CIC test without background pgbench
Background pgbench showed to be unstabe on some buildfarm members.
Also usage of background pgbench was wasting some test time because
of uneven pgbench running times.
---
contrib/amcheck/t/002_cic.pl | 46 ++++++++++------------------
contrib/amcheck/t/003_cic_2pc.pl | 52 +++++++++++++++-----------------
2 files changed, 40 insertions(+), 58 deletions(-)
diff --git a/contrib/amcheck/t/002_cic.pl b/contrib/amcheck/t/002_cic.pl
index 0b14e66270..1f258dd59d 100644
--- a/contrib/amcheck/t/002_cic.pl
+++ b/contrib/amcheck/t/002_cic.pl
@@ -9,7 +9,7 @@ use Config;
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
-use Test::More tests => 4;
+use Test::More tests => 3;
my ($node, $result);
@@ -25,32 +25,18 @@ $node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
$node->safe_psql('postgres', q(CREATE INDEX idx ON tbl(i)));
#
-# Stress CIC with pgbench
+# Stress CIC with pgbench.
+#
+# pgbench might try to launch more than one instance of the CIC
+# transaction concurrently. That would deadlock, so use an advisory
+# lock to ensure only one CIC runs at a time.
#
-
-# Run background pgbench with CIC. We cannot mix-in this script into single
-# pgbench: CIC will deadlock with itself occasionally.
-my $pgbench_out = '';
-my $pgbench_timer = IPC::Run::timeout(180);
-my $pgbench_h = $node->background_pgbench(
- '--no-vacuum --client=1 --transactions=200',
- {
- '002_pgbench_concurrent_cic' => q(
- DROP INDEX CONCURRENTLY idx;
- CREATE INDEX CONCURRENTLY idx ON tbl(i);
- SELECT bt_index_check('idx',true);
- )
- },
- \$pgbench_out,
- $pgbench_timer);
-
-# Run pgbench.
$node->pgbench(
'--no-vacuum --client=5 --transactions=200',
0,
[qr{actually processed}],
[qr{^$}],
- 'concurrent INSERTs',
+ 'concurrent INSERTs and CIC',
{
'002_pgbench_concurrent_transaction' => q(
BEGIN;
@@ -62,17 +48,17 @@ $node->pgbench(
SAVEPOINT s1;
INSERT INTO tbl VALUES(0);
COMMIT;
+ ),
+ '002_pgbench_concurrent_cic' => q(
+ SELECT pg_try_advisory_lock(42)::integer AS gotlock \gset
+ \if :gotlock
+ DROP INDEX CONCURRENTLY idx;
+ CREATE INDEX CONCURRENTLY idx ON tbl(i);
+ SELECT bt_index_check('idx',true);
+ SELECT pg_advisory_unlock(42);
+ \endif
)
});
-$pgbench_h->pump_nb;
-$pgbench_h->finish();
-$result =
- ($Config{osname} eq "MSWin32")
- ? ($pgbench_h->full_results)[0]
- : $pgbench_h->result(0);
-is($result, 0, "pgbench with CIC works");
-
-# done
$node->stop;
done_testing();
diff --git a/contrib/amcheck/t/003_cic_2pc.pl b/contrib/amcheck/t/003_cic_2pc.pl
index f4255c1fb8..e6b0c5ba20 100644
--- a/contrib/amcheck/t/003_cic_2pc.pl
+++ b/contrib/amcheck/t/003_cic_2pc.pl
@@ -9,7 +9,7 @@ use Config;
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
-use Test::More tests => 6;
+use Test::More tests => 5;
my ($node, $result);
@@ -132,57 +132,53 @@ is($result, '0', 'bt_index_check after 2PC and restart');
#
# Stress CIC+2PC with pgbench
#
+# pgbench might try to launch more than one instance of the CIC
+# transaction concurrently. That would deadlock, so use an advisory
+# lock to ensure only one CIC runs at a time.
# Fix broken index first
$node->safe_psql('postgres', q(REINDEX TABLE tbl;));
-# Run background pgbench with CIC. We cannot mix-in this script into single
-# pgbench: CIC will deadlock with itself occasionally.
-my $pgbench_out = '';
-my $pgbench_timer = IPC::Run::timeout(180);
-my $pgbench_h = $node->background_pgbench(
- '--no-vacuum --client=1 --transactions=100',
- {
- '002_pgbench_concurrent_cic' => q(
- DROP INDEX CONCURRENTLY idx;
- CREATE INDEX CONCURRENTLY idx ON tbl(i);
- SELECT bt_index_check('idx',true);
- )
- },
- \$pgbench_out,
- $pgbench_timer);
-
# Run pgbench.
$node->pgbench(
'--no-vacuum --client=5 --transactions=100',
0,
[qr{actually processed}],
[qr{^$}],
- 'concurrent INSERTs w/ 2PC',
+ 'concurrent INSERTs w/ 2PC and CIC',
{
- '002_pgbench_concurrent_2pc' => q(
+ '003_pgbench_concurrent_2pc' => q(
BEGIN;
INSERT INTO tbl VALUES(0);
PREPARE TRANSACTION 'c:client_id';
COMMIT PREPARED 'c:client_id';
),
- '002_pgbench_concurrent_2pc_savepoint' => q(
+ '003_pgbench_concurrent_2pc_savepoint' => q(
BEGIN;
SAVEPOINT s1;
INSERT INTO tbl VALUES(0);
PREPARE TRANSACTION 'c:client_id';
COMMIT PREPARED 'c:client_id';
+ ),
+ '003_pgbench_concurrent_cic' => q(
+ SELECT pg_try_advisory_lock(42)::integer AS gotlock \gset
+ \if :gotlock
+ DROP INDEX CONCURRENTLY idx;
+ CREATE INDEX CONCURRENTLY idx ON tbl(i);
+ SELECT bt_index_check('idx',true);
+ SELECT pg_advisory_unlock(42);
+ \endif
+ ),
+ '004_pgbench_concurrent_ric' => q(
+ SELECT pg_try_advisory_lock(42)::integer AS gotlock \gset
+ \if :gotlock
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT bt_index_check('idx',true);
+ SELECT pg_advisory_unlock(42);
+ \endif
)
});
-$pgbench_h->pump_nb;
-$pgbench_h->finish();
-$result =
- ($Config{osname} eq "MSWin32")
- ? ($pgbench_h->full_results)[0]
- : $pgbench_h->result(0);
-is($result, 0, "pgbench with CIC works");
-
# done
$node->stop;
done_testing();
--
2.24.3 (Apple Git-128)
=
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-24 23:53 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-25 08:30 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-25 20:59 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 01:51 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-26 06:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 13:45 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-26 19:41 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-27 11:08 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-10-28 15:47 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 40+ messages in thread
From: Tom Lane @ 2021-10-28 15:47 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Noah Misch <noah@leadboat.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
Andrey Borodin <x4mmm@yandex-team.ru> writes:
>> 26 окт. 2021 г., в 22:41, Tom Lane <tgl@sss.pgh.pa.us> написал(а):
>> We can get around that though, by using pg_try_advisory_lock and not
>> proceeding if it fails. The attached POC does this for the 002 test;
>> it looks like the same thing could be done to 003.
> That's a neat idea. PFA patch with copy of this changes to 003.
Pushed. It'll be interesting to see if conchuela's behavior changes.
regards, tom lane
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-10-29 02:56 ` Noah Misch <noah@leadboat.com>
2021-10-29 03:19 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-11-10 04:15 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
1 sibling, 2 replies; 40+ messages in thread
From: Noah Misch @ 2021-10-29 02:56 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Sun, Oct 24, 2021 at 09:03:27PM +0300, Andrey Borodin wrote:
> > 24 окт. 2021 г., в 19:19, Noah Misch <noah@leadboat.com> написал(а):
> > These failures started on 2021-10-09, the day conchuela updated from DragonFly
> > v4.4.3-RELEASE to DragonFly v6.0.0-RELEASE. It smells like a kernel bug.
> > Since the theorized kernel bug seems not to affect
> > src/test/subscription/t/015_stream.pl, I wonder if we can borrow a workaround
> > from other tests. One thing in common with src/test/recovery/t/017_shm.pl and
> > the newest failure sites is that they don't write anything to the child stdin.
> > If not, does passing the script via stdin, like "pgbench -f-
> > <script.sql", work around the problem?
>
> I'll test it tomorrow, the refactoring does not seem trivial given we use many simultaneous scripts.
Did that work? Commit 7f580aa should make this unnecessary for v12+
contrib/amcheck tests, but older branches still need a fix, and 017_shm.pl
needs a fix in all branches. A backup plan is just to skip affected tests on
dragonfly 6+. Since the breakage has been limited to so few tests, I'm
optimistic that a better workaround will present itself.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
@ 2021-10-29 03:19 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-10-29 03:42 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
1 sibling, 1 reply; 40+ messages in thread
From: Tom Lane @ 2021-10-29 03:19 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
Noah Misch <noah@leadboat.com> writes:
> Did that work? Commit 7f580aa should make this unnecessary for v12+
> contrib/amcheck tests, but older branches still need a fix, and 017_shm.pl
> needs a fix in all branches. A backup plan is just to skip affected tests on
> dragonfly 6+. Since the breakage has been limited to so few tests, I'm
> optimistic that a better workaround will present itself.
It indeed is looking like 7f580aa made the problem go away on conchuela,
but do we understand why? The only theory I can think of is "kernel bug",
but while that's plausible for prairiedog it seems hard to credit for a
late-model BSD kernel.
regards, tom lane
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-29 03:19 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
@ 2021-10-29 03:42 ` Thomas Munro <thomas.munro@gmail.com>
2021-10-29 06:39 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 11:57 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
0 siblings, 2 replies; 40+ messages in thread
From: Thomas Munro @ 2021-10-29 03:42 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Noah Misch <noah@leadboat.com>; Andrey Borodin <x4mmm@yandex-team.ru>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Fri, Oct 29, 2021 at 4:20 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> It indeed is looking like 7f580aa made the problem go away on conchuela,
> but do we understand why? The only theory I can think of is "kernel bug",
> but while that's plausible for prairiedog it seems hard to credit for a
> late-model BSD kernel.
I have yet to even log into a DBSD system (my attempt to install the
6.0.1 ISO on bhyve failed for lack of a driver, or something), but I
do intend to get it working at some point. But I can offer a poorly
researched wildly speculative hypothesis: DBSD forked from FBSD in
2003. macOS 10.3 took FBSD's kqueue code in... 2003. So maybe a bug
was fixed later that they both inherited? Or perhaps that makes no
sense, I dunno. It'd be nice to try to write a repro and send them a
report, if we can.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-29 03:19 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-29 03:42 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
@ 2021-10-29 06:39 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 03:57 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
1 sibling, 1 reply; 40+ messages in thread
From: Andrey Borodin @ 2021-10-29 06:39 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Noah Misch <noah@leadboat.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
> 29 окт. 2021 г., в 08:42, Thomas Munro <thomas.munro@gmail.com> написал(а):
>
> On Fri, Oct 29, 2021 at 4:20 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> It indeed is looking like 7f580aa made the problem go away on conchuela,
>> but do we understand why? The only theory I can think of is "kernel bug",
>> but while that's plausible for prairiedog it seems hard to credit for a
>> late-model BSD kernel.
>
> I have yet to even log into a DBSD system (my attempt to install the
> 6.0.1 ISO on bhyve failed for lack of a driver, or something), but I
> do intend to get it working at some point.
Here's an exported VM https://storage.yandexcloud.net/freebsd/dffb.ova if it's of any use.
root password is P@ssw0rd
To reproduce
cd postgres/contrib/amcheck
su x4m
/tesh.sh
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-29 03:19 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-29 03:42 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-10-29 06:39 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-10-30 03:57 ` Thomas Munro <thomas.munro@gmail.com>
0 siblings, 0 replies; 40+ messages in thread
From: Thomas Munro @ 2021-10-30 03:57 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Noah Misch <noah@leadboat.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Fri, Oct 29, 2021 at 7:39 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> Here's an exported VM https://storage.yandexcloud.net/freebsd/dffb.ova if it's of any use.
Thanks! I may yet need to use that because I haven't seen the problem
yet, but I finally managed to get the OS running with a reusable
Vagrant file after solving a couple of stupid problems[1].
[1] https://github.com/macdice/postgresql-dev-vagrant/blob/master/dragonfly6/Vagrantfile
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-29 03:19 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-10-29 03:42 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
@ 2021-10-29 11:57 ` Noah Misch <noah@leadboat.com>
1 sibling, 0 replies; 40+ messages in thread
From: Noah Misch @ 2021-10-29 11:57 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Andrey Borodin <x4mmm@yandex-team.ru>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Fri, Oct 29, 2021 at 04:42:31PM +1300, Thomas Munro wrote:
> On Fri, Oct 29, 2021 at 4:20 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > It indeed is looking like 7f580aa made the problem go away on conchuela,
> > but do we understand why?
I don't.
> > The only theory I can think of is "kernel bug",
> > but while that's plausible for prairiedog it seems hard to credit for a
> > late-model BSD kernel.
DragonFly BSD is a niche OS, so I'm more willing than usual to conclude that.
Could be a bug in IPC::Run or in the port of Perl to DragonFly, but those feel
less likely than the kernel. The upgrade from DragonFly v4.4.3 to DragonFly
v6.0.0, which introduced this form of PostgreSQL test breakage, also updated
Perl from v5.20.3 to 5.32.1.
> I have yet to even log into a DBSD system (my attempt to install the
> 6.0.1 ISO on bhyve failed for lack of a driver, or something), but I
> do intend to get it working at some point. But I can offer a poorly
> researched wildly speculative hypothesis: DBSD forked from FBSD in
> 2003. macOS 10.3 took FBSD's kqueue code in... 2003. So maybe a bug
> was fixed later that they both inherited? Or perhaps that makes no
> sense, I dunno. It'd be nice to try to write a repro and send them a
> report, if we can.
The conchuela bug and the prairiedog bug both present with a timeout in
IPC::Run::finish, but the similarity ends there. On prairiedog, the
postmaster was stuck when it should have been reading a query from pgbench.
On conchuela, pgbench ran to completion and became a zombie, and IPC::Run got
stuck when it should have been reaping that zombie. Good thought, however.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
@ 2021-11-10 04:15 ` Noah Misch <noah@leadboat.com>
2021-11-10 05:49 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 1 reply; 40+ messages in thread
From: Noah Misch @ 2021-11-10 04:15 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Thu, Oct 28, 2021 at 07:56:00PM -0700, Noah Misch wrote:
> On Sun, Oct 24, 2021 at 09:03:27PM +0300, Andrey Borodin wrote:
> > > 24 окт. 2021 г., в 19:19, Noah Misch <noah@leadboat.com> написал(а):
> > > These failures started on 2021-10-09, the day conchuela updated from DragonFly
> > > v4.4.3-RELEASE to DragonFly v6.0.0-RELEASE. It smells like a kernel bug.
> > > Since the theorized kernel bug seems not to affect
> > > src/test/subscription/t/015_stream.pl, I wonder if we can borrow a workaround
> > > from other tests. One thing in common with src/test/recovery/t/017_shm.pl and
> > > the newest failure sites is that they don't write anything to the child stdin.
>
> > > If not, does passing the script via stdin, like "pgbench -f-
> > > <script.sql", work around the problem?
> >
> > I'll test it tomorrow, the refactoring does not seem trivial given we use many simultaneous scripts.
>
> Did that work? Commit 7f580aa should make this unnecessary for v12+
> contrib/amcheck tests, but older branches still need a fix, and 017_shm.pl
> needs a fix in all branches. A backup plan is just to skip affected tests on
> dragonfly 6+. Since the breakage has been limited to so few tests, I'm
> optimistic that a better workaround will present itself.
Is this still in your queue, or not? The conchuela breakage in
src/test/recovery and v11+v10 src/bin/pgbench is a large source of buildfarm
noise right now.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 04:15 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
@ 2021-11-10 05:49 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-13 18:47 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 40+ messages in thread
From: Andrey Borodin @ 2021-11-10 05:49 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
> 10 нояб. 2021 г., в 09:15, Noah Misch <noah@leadboat.com> написал(а):
>
> On Thu, Oct 28, 2021 at 07:56:00PM -0700, Noah Misch wrote:
>> On Sun, Oct 24, 2021 at 09:03:27PM +0300, Andrey Borodin wrote:
>>>> 24 окт. 2021 г., в 19:19, Noah Misch <noah@leadboat.com> написал(а):
>>>> These failures started on 2021-10-09, the day conchuela updated from DragonFly
>>>> v4.4.3-RELEASE to DragonFly v6.0.0-RELEASE. It smells like a kernel bug.
>>>> Since the theorized kernel bug seems not to affect
>>>> src/test/subscription/t/015_stream.pl, I wonder if we can borrow a workaround
>>>> from other tests. One thing in common with src/test/recovery/t/017_shm.pl and
>>>> the newest failure sites is that they don't write anything to the child stdin.
>>
>>>> If not, does passing the script via stdin, like "pgbench -f-
>>>> <script.sql", work around the problem?
>>>
>>> I'll test it tomorrow, the refactoring does not seem trivial given we use many simultaneous scripts.
>>
>> Did that work? Commit 7f580aa should make this unnecessary for v12+
>> contrib/amcheck tests, but older branches still need a fix, and 017_shm.pl
>> needs a fix in all branches. A backup plan is just to skip affected tests on
>> dragonfly 6+. Since the breakage has been limited to so few tests, I'm
>> optimistic that a better workaround will present itself.
>
> Is this still in your queue, or not? The conchuela breakage in
> src/test/recovery and v11+v10 src/bin/pgbench is a large source of buildfarm
> noise right now.
Uh, sorry, this problem fell out of my attention somehow. I'll try to do something with 10 and 11 this or next week.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 04:15 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 05:49 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-11-13 18:47 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-14 00:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 40+ messages in thread
From: Andrey Borodin @ 2021-11-13 18:47 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>; Noah Misch <noah@leadboat.com>
>> 10 нояб. 2021 г., в 09:15, Noah Misch <noah@leadboat.com> написал(а):
>>
> Uh, sorry, this problem fell out of my attention somehow. I'll try to do something with 10 and 11 this or next week.
>
I've adapted 7f580aa to functionality of REL_11 using "\if 0 = :client_id" metacommand.
I really do not like idea of supporting background_pgbench() in older branches without counterpart in newer branches.
But so far I didn't come up with some clever mutex idea for REL_10.
Best regards, Andrey Borodin.
Attachments:
0001-Get-rid-of-background_pgbench-in-CIC-tests.patch (text/x-diff, 6.4K) download
From 8cc72e4455b724c9e891b2c71924102e192657cd Mon Sep 17 00:00:00 2001
From: Andrey Borodin <x4m@flight.local>
Date: Sat, 13 Nov 2021 23:40:30 +0500
Subject: [PATCH] Get rid of background_pgbench() in CIC tests
This is an adaptation of commit 7f580aa for the REL_11_STABLE
that lacks \gset pgbench command. To emulate conditional REINDEX
depending on result of advisory lock attemp we just do REINDEX
only when :client_id = 0.
---
src/bin/pgbench/t/022_cic.pl | 40 ++++++++++----------------
src/bin/pgbench/t/023_cic_2pc.pl | 46 ++++++++++--------------------
src/test/perl/PostgresNode.pm | 49 --------------------------------
3 files changed, 30 insertions(+), 105 deletions(-)
diff --git a/src/bin/pgbench/t/022_cic.pl b/src/bin/pgbench/t/022_cic.pl
index a1cda5b47ae..55288f0058f 100644
--- a/src/bin/pgbench/t/022_cic.pl
+++ b/src/bin/pgbench/t/022_cic.pl
@@ -9,7 +9,7 @@ use Config;
use PostgresNode;
use TestLib;
-use Test::More tests => 4;
+use Test::More tests => 3;
my ($node, $result);
@@ -42,26 +42,11 @@ $node->safe_psql(
));
#
-# Stress CIC with pgbench
+# Stress CIC with pgbench.
#
-
-# Run background pgbench with CIC. We cannot mix-in this script into single
-# pgbench: CIC will deadlock with itself occasionally.
-my $pgbench_out = '';
-my $pgbench_timer = IPC::Run::timeout(180);
-my $pgbench_h = $node->background_pgbench(
- '--no-vacuum --client=1 --transactions=200',
- {
- '002_pgbench_concurrent_cic' => q(
- DROP INDEX CONCURRENTLY idx;
- CREATE INDEX CONCURRENTLY idx ON tbl(i);
- BEGIN ISOLATION LEVEL REPEATABLE READ;
- SELECT heapallindexed();
- ROLLBACK;
- )
- },
- \$pgbench_out,
- $pgbench_timer);
+# pgbench might try to launch more than one instance of the CIC
+# transaction concurrently. That would deadlock, so use 0 = :client_id
+# to ensure only one CIC runs at a time.
# Run pgbench.
$node->pgbench(
@@ -71,7 +56,7 @@ $node->pgbench(
[qr{^$}],
'concurrent INSERTs',
{
- '002_pgbench_concurrent_transaction' => q(
+ '001_pgbench_concurrent_transaction' => q(
BEGIN;
INSERT INTO tbl VALUES(0);
COMMIT;
@@ -81,13 +66,18 @@ $node->pgbench(
SAVEPOINT s1;
INSERT INTO tbl VALUES(0);
COMMIT;
+ ),
+ '003_pgbench_concurrent_cic' => q(
+ \if 0 = :client_id
+ DROP INDEX CONCURRENTLY idx;
+ CREATE INDEX CONCURRENTLY idx ON tbl(i);
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT heapallindexed();
+ ROLLBACK;
+ \endif
)
});
-$pgbench_h->pump_nb;
-$pgbench_h->finish();
-unlike($pgbench_out, qr/aborted in command/, "pgbench with CIC works");
-
# done
$node->stop;
done_testing();
diff --git a/src/bin/pgbench/t/023_cic_2pc.pl b/src/bin/pgbench/t/023_cic_2pc.pl
index e29c2564b77..81f0f75349f 100644
--- a/src/bin/pgbench/t/023_cic_2pc.pl
+++ b/src/bin/pgbench/t/023_cic_2pc.pl
@@ -9,7 +9,7 @@ use Config;
use PostgresNode;
use TestLib;
-use Test::More tests => 6;
+use Test::More tests => 5;
my ($node, $result);
@@ -149,29 +149,15 @@ is($result, '0', 'all indexed after 2PC and restart');
#
-# Stress CIC+2PC with pgbench
+# Stress CIC+2PC with pgbench.
#
+# pgbench might try to launch more than one instance of the CIC
+# transaction concurrently. That would deadlock, so use 0 = :client_id
+# to ensure only one CIC runs at a time.
# Fix broken index first
$node->safe_psql('postgres', q(REINDEX TABLE tbl;));
-# Run background pgbench with CIC. We cannot mix-in this script into single
-# pgbench: CIC will deadlock with itself occasionally.
-my $pgbench_out = '';
-my $pgbench_timer = IPC::Run::timeout(180);
-my $pgbench_h = $node->background_pgbench(
- '--no-vacuum --client=1 --transactions=100',
- {
- '002_pgbench_concurrent_cic' => q(
- DROP INDEX CONCURRENTLY idx;
- CREATE INDEX CONCURRENTLY idx ON tbl(i);
- BEGIN ISOLATION LEVEL REPEATABLE READ;
- SELECT heapallindexed();
- ROLLBACK;
- )
- },
- \$pgbench_out,
- $pgbench_timer);
# Run pgbench.
$node->pgbench(
@@ -179,27 +165,25 @@ $node->pgbench(
0,
[qr{actually processed}],
[qr{^$}],
- 'concurrent INSERTs w/ 2PC',
+ 'concurrent INSERTs w/ 2PC and CIC',
{
- '002_pgbench_concurrent_2pc' => q(
+ '001_pgbench_concurrent_2pc' => q(
BEGIN;
INSERT INTO tbl VALUES(0);
PREPARE TRANSACTION 'c:client_id';
COMMIT PREPARED 'c:client_id';
),
- '002_pgbench_concurrent_2pc_savepoint' => q(
- BEGIN;
- SAVEPOINT s1;
- INSERT INTO tbl VALUES(0);
- PREPARE TRANSACTION 'c:client_id';
- COMMIT PREPARED 'c:client_id';
+ '002_pgbench_concurrent_cic' => q(
+ \if 0 = :client_id
+ DROP INDEX CONCURRENTLY idx;
+ CREATE INDEX CONCURRENTLY idx ON tbl(i);
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT heapallindexed();
+ ROLLBACK;
+ \endif
)
});
-$pgbench_h->pump_nb;
-$pgbench_h->finish();
-unlike($pgbench_out, qr/aborted in command/, "pgbench with CIC works");
-
# done
$node->stop;
done_testing();
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 76ef3826d15..bf4b9f96905 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -1659,55 +1659,6 @@ sub pgbench
=pod
-=item $node->background_pgbench($opts, $files, \$stdout, $timer) => harness
-
-Invoke B<pgbench> and return an IPC::Run harness object. The process's stdin
-is empty, and its stdout and stderr go to the $stdout scalar reference. This
-allows the caller to act on other parts of the system while B<pgbench> is
-running. Errors from B<pgbench> are the caller's problem.
-
-The specified timer object is attached to the harness, as well. It's caller's
-responsibility to select the timeout length, and to restart the timer after
-each command if the timeout is per-command.
-
-Be sure to "finish" the harness when done with it.
-
-=over
-
-=item $opts
-
-Options as a string to be split on spaces.
-
-=item $files
-
-Reference to filename/contents dictionary.
-
-=back
-
-=cut
-
-sub background_pgbench
-{
- my ($self, $opts, $files, $stdout, $timer) = @_;
-
- my @cmd =
- ('pgbench', split(/\s+/, $opts), $self->_pgbench_make_files($files));
-
- local $ENV{PGHOST} = $self->host;
- local $ENV{PGPORT} = $self->port;
-
- my $stdin = "";
- # IPC::Run would otherwise append to existing contents:
- $$stdout = "" if ref($stdout);
-
- my $harness = IPC::Run::start \@cmd, '<', \$stdin, '>', $stdout, '2>&1',
- $timer;
-
- return $harness;
-}
-
-=pod
-
=item $node->poll_query_until($dbname, $query [, $expected ])
Run B<$query> repeatedly, until it returns the B<$expected> result
--
2.33.1
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 04:15 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 05:49 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-13 18:47 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-11-14 00:09 ` Noah Misch <noah@leadboat.com>
2021-11-14 01:24 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-15 07:19 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 2 replies; 40+ messages in thread
From: Noah Misch @ 2021-11-14 00:09 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Sat, Nov 13, 2021 at 11:47:43PM +0500, Andrey Borodin wrote:
> >> 10 нояб. 2021 г., в 09:15, Noah Misch <noah@leadboat.com> написал(а):
> > Uh, sorry, this problem fell out of my attention somehow. I'll try to do something with 10 and 11 this or next week.
>
> I've adapted 7f580aa to functionality of REL_11 using "\if 0 = :client_id" metacommand.
> I really do not like idea of supporting background_pgbench() in older branches without counterpart in newer branches.
> But so far I didn't come up with some clever mutex idea for REL_10.
That's a reasonable sentiment, but removing background_pgbench() isn't going
to fix 017_shm.pl. I'm not enthusiastic about any fix that repairs
src/bin/pgbench without repairing 017_shm.pl. I'm okay with skipping affected
test files on dragonfly >= 6 if you decide to cease figuring out how to make
them pass like the others do.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 04:15 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 05:49 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-13 18:47 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-14 00:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
@ 2021-11-14 01:24 ` Thomas Munro <thomas.munro@gmail.com>
1 sibling, 0 replies; 40+ messages in thread
From: Thomas Munro @ 2021-11-14 01:24 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Sun, Nov 14, 2021 at 1:17 PM Noah Misch <noah@leadboat.com> wrote:
> On Sat, Nov 13, 2021 at 11:47:43PM +0500, Andrey Borodin wrote:
> > I've adapted 7f580aa to functionality of REL_11 using "\if 0 = :client_id" metacommand.
> > I really do not like idea of supporting background_pgbench() in older branches without counterpart in newer branches.
> > But so far I didn't come up with some clever mutex idea for REL_10.
>
> That's a reasonable sentiment, but removing background_pgbench() isn't going
> to fix 017_shm.pl. I'm not enthusiastic about any fix that repairs
> src/bin/pgbench without repairing 017_shm.pl. I'm okay with skipping affected
> test files on dragonfly >= 6 if you decide to cease figuring out how to make
> them pass like the others do.
Hmm, so if "IPC::Run got stuck when it should have been reaping that
zombie", what's it stuck in, I guess select() or waitpid()? Maybe
there' s a kernel bug but it seems hard to believe that a Unix system
would have bugs in such fundamental facilities and still be able to
build itself and ship a release... Otherwise I guess Perl, or perl
scripts, would need to be confusing fds or pids or something? But
that's hard to believe on its own, too, given the lack of problems on
other systems that are pretty similar. If Andrey can still reproduce
this, it'd be interesting to see a gdb backtrace, and also "ps O
wchan" or perhaps kill -INFO $pid, and lsof for the process (or
according to old pages found with google, perhaps the equivalent tool
is "fstat" on that system).
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 04:15 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 05:49 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-13 18:47 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-14 00:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
@ 2021-11-15 07:19 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-15 21:23 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
1 sibling, 1 reply; 40+ messages in thread
From: Andrey Borodin @ 2021-11-15 07:19 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
> 14 нояб. 2021 г., в 05:09, Noah Misch <noah@leadboat.com> написал(а):
>
> On Sat, Nov 13, 2021 at 11:47:43PM +0500, Andrey Borodin wrote:
>>>> 10 нояб. 2021 г., в 09:15, Noah Misch <noah@leadboat.com> написал(а):
>>> Uh, sorry, this problem fell out of my attention somehow. I'll try to do something with 10 and 11 this or next week.
>>
>> I've adapted 7f580aa to functionality of REL_11 using "\if 0 = :client_id" metacommand.
>> I really do not like idea of supporting background_pgbench() in older branches without counterpart in newer branches.
>> But so far I didn't come up with some clever mutex idea for REL_10.
>
> That's a reasonable sentiment, but removing background_pgbench() isn't going
> to fix 017_shm.pl. I'm not enthusiastic about any fix that repairs
> src/bin/pgbench without repairing 017_shm.pl. I'm okay with skipping affected
> test files on dragonfly >= 6 if you decide to cease figuring out how to make
> them pass like the others do.
Let's skip these tests. How this can be accomplished?
Should we mute only 022_cic.pl, 023_cic_2pc.pl, and 017_shm.pl? Or everything that calls harness->finish?
I've sent some diagnostics info to Thomas offlist, but I do not understand how it can be used...
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 04:15 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 05:49 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-13 18:47 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-14 00:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-15 07:19 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-11-15 21:23 ` Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Thomas Munro @ 2021-11-15 21:23 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Noah Misch <noah@leadboat.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Mon, Nov 15, 2021 at 8:20 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> I've sent some diagnostics info to Thomas offlist, but I do not understand how it can be used...
Summary for the record: defunct child, and gdb showed perl blocked in
select(), and ps revealed that Dragonfly internally maps select() onto
kevent() (wchan = kqread), which is interesting, but apparently not
new. (Hmmm, /me eyes the recent changes to events reported on
half-closed pipes[1].)
[1] https://github.com/DragonFlyBSD/DragonFlyBSD/commits/master/sys/kern/sys_generic.c
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 04:15 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 05:49 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-13 18:47 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-14 00:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-15 07:19 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-15 21:23 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
@ 2021-11-19 04:09 ` Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:52 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Thomas Munro @ 2021-11-19 04:09 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Noah Misch <noah@leadboat.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
I managed to reproduce the 017_shm.pl hang using my Vagrant file (see
earlier). It happens for "gmake -s check-world", but not "gmake -s -C
src/test/recovery check", which might have confused me last time. The
symptoms are as described already, though this time I learned with
truss that it's in a retry loop waiting 1 second at a time.
I don't know how to get debug symbols for libc on this operating
system (well, probably needed to select debug option while installing
the OS, but I didn't install the OS, I'm using a lazy bones pre-rolled
Vagrant image). So in order to be able to inspect the arguments to
select, I made my own LD_PRELOAD wrapper for select (see end), and I
observed that it was calling select() with an empty fd_set:
(gdb) bt
#0 0x00000008014c1e6c in select () from /lib/libc.so.8
#1 0x00000008009f0fe6 in select () from /usr/lib/libpthread.so.0
#2 0x000000080044c622 in select (nfds=16, readfds=0x8017f6de0,
writefds=0x8017f6d50, exceptfds=0x0,
timeout=0x7fffffdfd4a0) at select.c:13
#3 0x00000008007bfaf9 in Perl_pp_sselect () from
/usr/local/lib/perl5/5.32/mach/CORE/libperl.so.5.32
#4 0x000000080076b036 in Perl_runops_standard ()
from /usr/local/lib/perl5/5.32/mach/CORE/libperl.so.5.32
#5 0x00000008006da3b7 in perl_run () from
/usr/local/lib/perl5/5.32/mach/CORE/libperl.so.5.32
#6 0x0000000000400de4 in main ()
(gdb) f 2
#2 0x000000080044c622 in select (nfds=16, readfds=0x8017f6de0,
writefds=0x8017f6d50, exceptfds=0x0,
timeout=0x7fffffdfd4a0) at select.c:13
13 return real_function(nfds, readfds, writefds, exceptfds, timeout);
(gdb) print nfds
$1 = 16
(gdb) print *readfds
$2 = {fds_bits = {0 <repeats 16 times>}}
(gdb) print *writefds
$3 = {fds_bits = {0 <repeats 16 times>}}
So it looks a lot like something on the perl side has lost track of
the pipe it's supposed to be selecting on. If I understand correctly,
it's supposed to be waiting for one of the following pipes to appear
as readable, whichever is the one that the zombie psql process
previously held the other end of:
$ fstat -p 73032
USER CMD PID FD PATH INUM MODE
SZ|DV R/W
vagrant perl 73032 root / 1 drwxr-xr-x
offset:0 r
vagrant perl 73032 wd
/home/vagrant/postgres/src/test/recovery 4434524058 drwxr-xr-x
offset:0 r
vagrant perl 73032 text /pfs/@@-1:00001/local/bin/perl
4329919842 -rwxr-xr-x offset:0 r
vagrant perl 73032 0 /dev/pts/0
1335 crw--w---- pts/0:155171 rw
vagrant perl 73032 1
/home/vagrant/postgres/src/test/recovery/tmp_check/log/regress_log_017_shm
4477427235 -rw-r--r-- offset:9457 w
vagrant perl 73032 2
/home/vagrant/postgres/src/test/recovery/tmp_check/log/regress_log_017_shm
4477427235 -rw-r--r-- offset:9457 w
vagrant perl 73032 3
/home/vagrant/postgres/src/test/recovery/tmp_check/log/regress_log_017_shm
4477427235 -rw-r--r-- offset:9457 w
vagrant perl 73032 4* pipe fffff800aac1dca0 (B<->A) ravail 0
wavail 0 rw
vagrant perl 73032 5* pipe fffff800ab6e9020 (B<->A) ravail 0
wavail 0 rw
vagrant perl 73032 6* pipe fffff800aac1dca0 (B<->A) ravail 0
wavail 0 rw
vagrant perl 73032 7* pipe fffff800ab6e9020 (B<->A) ravail 0
wavail 0 rw
vagrant perl 73032 8* pipe fffff800aac1dca0 (B<->A) ravail 0
wavail 0 rw
vagrant perl 73032 9* pipe fffff800ab6e9020 (B<->A) ravail 0
wavail 0 rw
vagrant perl 73032 13* pipe fffff800ab157560 (B<->A) ravail 0
wavail 0 rw
=== select wrapper ===
$ cat select.c
#include <dlfcn.h>
#include <sys/select.h>
int
select(int nfds,fd_set *readfds, fd_set *writefds, fd_set *exceptfds,
struct timeval *timeout)
{
static int (*real_function)(int, fd_set *, fd_set *, fd_set *,
struct timeval *);
if (!real_function)
real_function = dlsym(RTLD_NEXT, "select");
return real_function(nfds, readfds, writefds, exceptfds, timeout);
}
$ cc -Wall -fPIC -shared -g -o myselect.so select.c -ldl
$ cd postgres
$ LD_PRELOAD=/home/vagrant/myselect.so gmake check-world -s
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 04:15 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 05:49 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-13 18:47 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-14 00:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-15 07:19 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-15 21:23 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
@ 2021-11-19 04:52 ` Thomas Munro <thomas.munro@gmail.com>
2021-11-19 06:13 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-19 16:22 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 40+ messages in thread
From: Thomas Munro @ 2021-11-19 04:52 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Noah Misch <noah@leadboat.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Fri, Nov 19, 2021 at 5:09 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> (gdb) print *writefds
> $3 = {fds_bits = {0 <repeats 16 times>}}
Oops, that was after it had been cleared already by the OS; duh. On
entry to my wrapper, writefds does in fact contain the bit pattern for
fd 13. That led me to try a very simply C program which runs to
completion on Linux and FreeBSD, but hangs forever on Dragonfly.
Attachments:
test.c (text/x-csrc, 944B) download
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/select.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
int
main()
{
int fd[2];
pid_t pid;
if (pipe(fd) < 0) {
perror("pipe");
return EXIT_FAILURE;
}
pid = fork();
if (pid < 0) {
perror("fork");
return EXIT_FAILURE;
} else if (pid == 0) {
close(fd[1]);
printf("child hello\n");
sleep(1);
printf("child goodbye\n");
} else {
char buf;
/* parent waits until the pipe is writable */
close(fd[0]); /* close read end */
sleep(2);
printf("parent select...\n");
for (;;) {
fd_set writefds;
FD_ZERO(&writefds);
FD_SET(fd[1], &writefds);
if (select(fd[1] + 1, NULL, &writefds, NULL, NULL) < 0) {
if (errno != EINTR) {
perror("select");
return EXIT_FAILURE;
}
continue;
}
if (FD_ISSET(fd[1], &writefds))
break;
}
printf("parent sees writable\n");
}
}
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 04:15 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 05:49 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-13 18:47 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-14 00:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-15 07:19 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-15 21:23 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:52 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
@ 2021-11-19 06:13 ` Noah Misch <noah@leadboat.com>
2021-11-19 08:43 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-19 09:07 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
1 sibling, 2 replies; 40+ messages in thread
From: Noah Misch @ 2021-11-19 06:13 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Fri, Nov 19, 2021 at 05:52:59PM +1300, Thomas Munro wrote:
> led me to try a very simply C program which runs to
> completion on Linux and FreeBSD, but hangs forever on Dragonfly.
> [pipe FD considered non-writable, but writing would give EPIPE]
Looks like optimal material for a kernel bug report. Excellent discovery.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 04:15 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 05:49 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-13 18:47 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-14 00:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-15 07:19 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-15 21:23 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:52 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 06:13 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
@ 2021-11-19 08:43 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-19 14:57 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 40+ messages in thread
From: Andrey Borodin @ 2021-11-19 08:43 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
> 19 нояб. 2021 г., в 11:13, Noah Misch <noah@leadboat.com> написал(а):
>
> Looks like optimal material for a kernel bug report. Excellent discovery.
Are we going to wait for the fix or disable tests partially on conchuela?
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 04:15 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 05:49 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-13 18:47 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-14 00:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-15 07:19 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-15 21:23 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:52 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 06:13 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-19 08:43 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
@ 2021-11-19 14:57 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-11-19 19:35 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Tom Lane @ 2021-11-19 14:57 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Noah Misch <noah@leadboat.com>; Thomas Munro <thomas.munro@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
Andrey Borodin <x4mmm@yandex-team.ru> writes:
> Are we going to wait for the fix or disable tests partially on conchuela?
Let's wait a bit and see if a fix will be forthcoming promptly.
regards, tom lane
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 04:15 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 05:49 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-13 18:47 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-14 00:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-15 07:19 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-15 21:23 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:52 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 06:13 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-19 08:43 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-19 14:57 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
@ 2021-11-19 19:35 ` Thomas Munro <thomas.munro@gmail.com>
2021-12-29 19:53 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Thomas Munro @ 2021-11-19 19:35 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Noah Misch <noah@leadboat.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Sat, Nov 20, 2021 at 3:57 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Andrey Borodin <x4mmm@yandex-team.ru> writes:
> > Are we going to wait for the fix or disable tests partially on conchuela?
>
> Let's wait a bit and see if a fix will be forthcoming promptly.
The fix came forth. I'll wait for the back-patch and then ask Mikael
to upgrade conchuela.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 04:15 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 05:49 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-13 18:47 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-14 00:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-15 07:19 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-15 21:23 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:52 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 06:13 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-19 08:43 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-19 14:57 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-11-19 19:35 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
@ 2021-12-29 19:53 ` Thomas Munro <thomas.munro@gmail.com>
2021-12-29 22:58 ` Re: conchuela timeouts since 2021-10-09 system upgrade Mikael Kjellström <mikael.kjellstrom@mksoft.nu>
0 siblings, 1 reply; 40+ messages in thread
From: Thomas Munro @ 2021-12-29 19:53 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Noah Misch <noah@leadboat.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Sat, Nov 20, 2021 at 8:35 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> The fix came forth. I'll wait for the back-patch and then ask Mikael
> to upgrade conchuela.
Done; thanks Mikael.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 04:15 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 05:49 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-13 18:47 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-14 00:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-15 07:19 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-15 21:23 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:52 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 06:13 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-19 08:43 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-19 14:57 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
2021-11-19 19:35 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-12-29 19:53 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
@ 2021-12-29 22:58 ` Mikael Kjellström <mikael.kjellstrom@mksoft.nu>
0 siblings, 0 replies; 40+ messages in thread
From: Mikael Kjellström @ 2021-12-29 22:58 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Noah Misch <noah@leadboat.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On 2021-12-29 20:53, Thomas Munro wrote:
> On Sat, Nov 20, 2021 at 8:35 AM Thomas Munro <thomas.munro@gmail.com> wrote:
>> The fix came forth. I'll wait for the back-patch and then ask Mikael
>> to upgrade conchuela.
>
> Done; thanks Mikael.
It has only had 1 pass through the buildfarm so far but all branches passed.
So looking promising.
Let's see how it looks after a few more passes.
/Mikael
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 04:15 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 05:49 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-13 18:47 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-14 00:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-15 07:19 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-15 21:23 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:52 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 06:13 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
@ 2021-11-19 09:07 ` Thomas Munro <thomas.munro@gmail.com>
2021-11-19 14:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 40+ messages in thread
From: Thomas Munro @ 2021-11-19 09:07 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Tom Lane <tgl@sss.pgh.pa.us>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
On Fri, Nov 19, 2021 at 7:13 PM Noah Misch <noah@leadboat.com> wrote:
> On Fri, Nov 19, 2021 at 05:52:59PM +1300, Thomas Munro wrote:
> > led me to try a very simply C program which runs to
> > completion on Linux and FreeBSD, but hangs forever on Dragonfly.
>
> > [pipe FD considered non-writable, but writing would give EPIPE]
>
> Looks like optimal material for a kernel bug report. Excellent discovery.
https://bugs.dragonflybsd.org/issues/3307
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 04:15 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 05:49 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-13 18:47 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-14 00:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-15 07:19 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-15 21:23 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:52 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 06:13 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-19 09:07 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
@ 2021-11-19 14:56 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 40+ messages in thread
From: Tom Lane @ 2021-11-19 14:56 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; Andrey Borodin <x4mmm@yandex-team.ru>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
Thomas Munro <thomas.munro@gmail.com> writes:
> On Fri, Nov 19, 2021 at 7:13 PM Noah Misch <noah@leadboat.com> wrote:
>> Looks like optimal material for a kernel bug report. Excellent discovery.
> https://bugs.dragonflybsd.org/issues/3307
I see they're pushing back on whether this is a bug. I failed to find a
way to quickly comment on the bug report, but I suggest you quote POSIX
select(2) at them:
A descriptor shall be considered ready for reading when a call to an
input function with O_NONBLOCK clear would not block, whether or not
the function would transfer data successfully. (The function might
return data, an end-of-file indication, or an error other than one
indicating that it is blocked, and in each of these cases the
descriptor shall be considered ready for reading.)
A descriptor shall be considered ready for writing when a call to an
output function with O_NONBLOCK clear would not block, whether or not
the function would transfer data successfully.
I don't know whether it'd help to point out that the test program works
as expected on other BSDen. (I'm planning to go try it on a few more
platforms in a bit, but in any case the standards-compliance question
looks pretty open-and-shut to me.)
regards, tom lane
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: conchuela timeouts since 2021-10-09 system upgrade
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-29 02:56 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 04:15 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-10 05:49 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-13 18:47 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-14 00:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-11-15 07:19 ` Re: conchuela timeouts since 2021-10-09 system upgrade Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-15 21:23 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:09 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:52 ` Re: conchuela timeouts since 2021-10-09 system upgrade Thomas Munro <thomas.munro@gmail.com>
@ 2021-11-19 16:22 ` Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 40+ messages in thread
From: Tom Lane @ 2021-11-19 16:22 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Noah Misch <noah@leadboat.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Michael Paquier <michael@paquier.xyz>; Peter Geoghegan <pg@bowt.ie>; Andres Freund <andres@anarazel.de>
Thomas Munro <thomas.munro@gmail.com> writes:
> Oops, that was after it had been cleared already by the OS; duh. On
> entry to my wrapper, writefds does in fact contain the bit pattern for
> fd 13. That led me to try a very simply C program which runs to
> completion on Linux and FreeBSD, but hangs forever on Dragonfly.
For completeness, I checked this on macOS, OpenBSD, and NetBSD,
and they all give the expected results.
regards, tom lane
^ permalink raw reply [nested|flat] 40+ messages in thread
end of thread, other threads:[~2021-12-29 22:58 UTC | newest]
Thread overview: 40+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-10-24 16:19 conchuela timeouts since 2021-10-09 system upgrade Noah Misch <noah@leadboat.com>
2021-10-24 18:03 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-24 23:53 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-10-25 08:30 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-25 20:59 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 01:51 ` Noah Misch <noah@leadboat.com>
2021-10-26 06:03 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 13:45 ` Noah Misch <noah@leadboat.com>
2021-10-26 14:29 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 20:45 ` Thomas Munro <thomas.munro@gmail.com>
2021-10-27 00:53 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-10-26 19:41 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-10-27 03:01 ` Noah Misch <noah@leadboat.com>
2021-10-27 03:20 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-10-27 11:08 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-28 15:47 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-10-29 02:56 ` Noah Misch <noah@leadboat.com>
2021-10-29 03:19 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-10-29 03:42 ` Thomas Munro <thomas.munro@gmail.com>
2021-10-29 06:39 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-10-30 03:57 ` Thomas Munro <thomas.munro@gmail.com>
2021-10-29 11:57 ` Noah Misch <noah@leadboat.com>
2021-11-10 04:15 ` Noah Misch <noah@leadboat.com>
2021-11-10 05:49 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-13 18:47 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-14 00:09 ` Noah Misch <noah@leadboat.com>
2021-11-14 01:24 ` Thomas Munro <thomas.munro@gmail.com>
2021-11-15 07:19 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-15 21:23 ` Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:09 ` Thomas Munro <thomas.munro@gmail.com>
2021-11-19 04:52 ` Thomas Munro <thomas.munro@gmail.com>
2021-11-19 06:13 ` Noah Misch <noah@leadboat.com>
2021-11-19 08:43 ` Andrey Borodin <x4mmm@yandex-team.ru>
2021-11-19 14:57 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-11-19 19:35 ` Thomas Munro <thomas.munro@gmail.com>
2021-12-29 19:53 ` Thomas Munro <thomas.munro@gmail.com>
2021-12-29 22:58 ` Mikael Kjellström <mikael.kjellstrom@mksoft.nu>
2021-11-19 09:07 ` Thomas Munro <thomas.munro@gmail.com>
2021-11-19 14:56 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-11-19 16:22 ` Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox