Subject: Re: New Object Access Type hooks

From: Thomas Munro <[email-masked]>
Date: Tue, 22 Mar 2022 18:03:31 +1300

On Fri, Mar 18, 2022 at 4:22 PM Mark Dilger
<[email-masked]> wrote:
> (FYI, I got a test failure from src/test/recovery/t/013_crash_restart.pl when testing v1-0001.  I'm not sure yet what that is about.)

Doesn't look like 0001 has anything to do with that...   Are you on a
Mac?  Did it look like this recent failure from CI?

https://cirrus-ci.com/task/4686108033286144
https://api.cirrus-ci.com/v1/artifact/task/4686108033286144/log/src/test/recovery/tmp_check/log/regress_log_013_crash_restart
https://api.cirrus-ci.com/v1/artifact/task/4686108033286144/log/src/test/recovery/tmp_check/log/013_crash_restart_primary.log

I have no idea what is going on there, but searching for discussion
brought me here...

---

From: Andrew Dunstan <[email-masked]>
Date: Tue, 22 Mar 2022 09:25:18 -0400

> Mac?  Did it look like this recent failure from CI?

Probably not connected. It's working fine for me on Ubuntu/

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

---

From: Mark Dilger <[email-masked]>
Date: Mon, 4 Apr 2022 09:16:39 -0700

> On Mar 21, 2022, at 10:03 PM, Thomas Munro <[email-masked]> wrote:
> 

I just got a crash in this test again.  Are you still interested?  I still have the logs.  No core file appears to have been generated.

The test failure is

not ok 5 - psql query died successfully after SIGQUIT

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

---

From: Tom Lane <[email-masked]>
Date: Mon, 04 Apr 2022 13:03:10 -0400

> not ok 5 - psql query died successfully after SIGQUIT

Hmm ... I can see one problem with that test:

ok( pump_until(
		$killme,
		$psql_timeout,
		\$killme_stderr,
		qr/WARNING:  terminating connection because of crash of another server process|server closed the connection unexpectedly|connection to server was lost/m
	),
	"psql query died successfully after SIGQUIT");

It's been a little while since that message looked like that.
Nowadays you get

WARNING:  terminating connection because of unexpected SIGQUIT signal
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

Usually the test would succeed anyway because of matching the
second or third regex alternative, but I wonder if there is
some other spelling of libpq's complaint that shows up
occasionally.  It'd be nice if we could see the contents of
$killme_stderr upon failure.

			regards, tom lane

---

From: Tom Lane <[email-masked]>
Date: Mon, 04 Apr 2022 13:41:17 -0400

> $killme_stderr upon failure.

OK, now I'm confused, because pump_until is very clearly
*trying* to report exactly that:

        if (not $proc->pumpable())
        {
            diag("pump_until: process terminated unexpectedly when searching for \"$until\" with stream: \"$$stream\"");
            return 0;
        }

and if I intentionally break the regex then I do see this
output when running the test by hand:

# Running: pg_ctl kill QUIT 1922645
ok 4 - killed process with SIGQUIT
# pump_until: process terminated unexpectedly when searching for "(?^m:WARNING:  terminating connection because of crash of another server process|server closed the connection foounexpectedly|connection to server was lost)" with stream: "psql:<stdin>:9: WARNING:  terminating connection because of unexpected SIGQUIT signal
# psql:<stdin>:9: server closed the connection unexpectedly
#       This probably means the server terminated abnormally
#       before or while processing the request.
# "
not ok 5 - psql query died successfully after SIGQUIT

Is our CI setup failing to capture stderr from TAP tests??

			regards, tom lane

---

From: Mark Dilger <[email-masked]>
Date: Mon, 4 Apr 2022 10:44:25 -0700

> On Apr 4, 2022, at 10:41 AM, Tom Lane <[email-masked]> wrote:
> 

I'm looking into the way our TAP test infrastructure assigns port numbers to nodes, and whether that is reasonable during parallel test runs with nodes stopping and starting again.  On casual inspection, that doesn't seem ok, because the Cluster.pm logic to make sure nodes get unique ports doesn't seem to be thinking about other parallel tests running.  It will notice if another node is already bound to the port, but if another node has been killed and not yet restarted, won't things go off the rails?

I'm writing a parallel test just for this.  Will get back to you.

—
Mark Dilger

---

From: Mark Dilger <[email-masked]>
Date: Mon, 4 Apr 2022 11:32:40 -0700

> On Apr 4, 2022, at 10:44 AM, Mark Dilger <[email-masked]> wrote:
> 

Ok, that experiment didn't accomplish anything, beyond refreshing my memory regarding Cluster.pm preferring sockets over ports.

—
Mark Dilger