Subject: Re: [HACKERS] psql casts aspersions on server reliability

From: Bruce Momjian <[email-masked]>
Date: Wed, 22 Nov 2023 19:11:41 -0500

On Wed, Sep 28, 2016 at 09:14:41AM -0400, Tom Lane wrote:
> Robert Haas <[email-masked]> writes:
> > psql tends to do things like this:
> > rhaas=# select * from pg_stat_activity;
> > FATAL:  terminating connection due to administrator command
> > server closed the connection unexpectedly
> >     This probably means the server terminated abnormally
> >     before or while processing the request.
> 
> > Basically everything psql has to say about this is a lie:
> 
> I cannot get terribly excited about this.  What you seem to be proposing
> is that psql try to intuit the reason for connection closure from the
> last error message it got, but that seems likely to lead to worse lies
> than printing a boilerplate message.
> 
> I could go along with just dropping the last sentence ("This probably...")
> if the last error we got was FATAL level.  I don't find "unexpectedly"
> to be problematic here: from the point of view of psql, and probably
> of its user, the shutdown *was* unexpected.

I looked at this thread from 2016 and I think the problem is the
"abnormally" word, since if the server was shutdown by the administrator
(most likely), it isn't abnormal.  Here is a patch to remove
"abnormally".

-- 
  Bruce Momjian  <[email-masked]>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Only you can decide what is important to you.
diff --git a/src/interfaces/libpq/fe-misc.c b/src/interfaces/libpq/fe-misc.c
index 660cdec93c..634708d716 100644
--- a/src/interfaces/libpq/fe-misc.c
+++ b/src/interfaces/libpq/fe-misc.c
@@ -749,7 +749,7 @@ retry4:
 	 */
 definitelyEOF:
 	libpq_append_conn_error(conn, "server closed the connection unexpectedly\n"
-							"\tThis probably means the server terminated abnormally\n"
+							"\tThis probably means the server terminated\n"
 							"\tbefore or while processing the request.");
 
 	/* Come here if lower-level code already set a suitable errorMessage */
diff --git a/src/interfaces/libpq/fe-secure-openssl.c b/src/interfaces/libpq/fe-secure-openssl.c
index f1192d28f2..115776ce6c 100644
--- a/src/interfaces/libpq/fe-secure-openssl.c
+++ b/src/interfaces/libpq/fe-secure-openssl.c
@@ -206,7 +206,7 @@ rloop:
 				if (result_errno == EPIPE ||
 					result_errno == ECONNRESET)
 					libpq_append_conn_error(conn, "server closed the connection unexpectedly\n"
-											"\tThis probably means the server terminated abnormally\n"
+											"\tThis probably means the server terminated\n"
 											"\tbefore or while processing the request.");
 				else
 					libpq_append_conn_error(conn, "SSL SYSCALL error: %s",
@@ -306,7 +306,7 @@ pgtls_write(PGconn *conn, const void *ptr, size_t len)
 				result_errno = SOCK_ERRNO;
 				if (result_errno == EPIPE || result_errno == ECONNRESET)
 					libpq_append_conn_error(conn, "server closed the connection unexpectedly\n"
-											"\tThis probably means the server terminated abnormally\n"
+											"\tThis probably means the server terminated\n"
 											"\tbefore or while processing the request.");
 				else
 					libpq_append_conn_error(conn, "SSL SYSCALL error: %s",
diff --git a/src/interfaces/libpq/fe-secure.c b/src/interfaces/libpq/fe-secure.c
index bd72a87bbb..b972bd3ced 100644
--- a/src/interfaces/libpq/fe-secure.c
+++ b/src/interfaces/libpq/fe-secure.c
@@ -233,7 +233,7 @@ pqsecure_raw_read(PGconn *conn, void *ptr, size_t len)
 			case EPIPE:
 			case ECONNRESET:
 				libpq_append_conn_error(conn, "server closed the connection unexpectedly\n"
-										"\tThis probably means the server terminated abnormally\n"
+										"\tThis probably means the server terminated\n"
 										"\tbefore or while processing the request.");
 				break;
 
@@ -395,7 +395,7 @@ retry_masked:
 				/* (strdup failure is OK, we'll cope later) */
 				snprintf(msgbuf, sizeof(msgbuf),
 						 libpq_gettext("server closed the connection unexpectedly\n"
-									   "\tThis probably means the server terminated abnormally\n"
+									   "\tThis probably means the server terminated\n"
 									   "\tbefore or while processing the request."));
 				/* keep newline out of translated string */
 				strlcat(msgbuf, "\n", sizeof(msgbuf));

---

From: Tom Lane <[email-masked]>
Date: Wed, 22 Nov 2023 19:38:52 -0500

> (most likely), it isn't abnormal.  Here is a patch to remove

I do not think this is an improvement.  The places you are changing
are reacting to a connection closure.  *If* we had previously gotten a
"FATAL: terminating connection due to administrator command" message,
then yeah the connection closure is expected; but if not, it isn't.
Your patch adds no check for that.  (As I remarked in 2016, we could
probably condition this on the elevel being FATAL, rather than
checking for specific error messages.)

			regards, tom lane

---

From: Bruce Momjian <[email-masked]>
Date: Wed, 22 Nov 2023 22:25:14 -0500

> checking for specific error messages.)

Yes, you are correct.  Here is a patch that implements the FATAL test,
though I am not sure I have the logic correct or backwards, and I don't
know how to test this.  Thanks.

  Only you can decide what is important to you.
diff --git a/src/interfaces/libpq/fe-misc.c b/src/interfaces/libpq/fe-misc.c
index 660cdec93c..c541fd8b02 100644
--- a/src/interfaces/libpq/fe-misc.c
+++ b/src/interfaces/libpq/fe-misc.c
@@ -749,8 +749,10 @@ retry4:
 	 */
 definitelyEOF:
 	libpq_append_conn_error(conn, "server closed the connection unexpectedly\n"
-							"\tThis probably means the server terminated abnormally\n"
-							"\tbefore or while processing the request.");
+							"\tThis probably means the server terminated%s\n"
+							"\tbefore or while processing the request.",
+							conn->result->resultStatus == PGRES_FATAL_ERROR ?
+							"" : "abnormally");
 
 	/* Come here if lower-level code already set a suitable errorMessage */
 definitelyFailed:
diff --git a/src/interfaces/libpq/fe-secure-openssl.c b/src/interfaces/libpq/fe-secure-openssl.c
index f1192d28f2..6c21f91817 100644
--- a/src/interfaces/libpq/fe-secure-openssl.c
+++ b/src/interfaces/libpq/fe-secure-openssl.c
@@ -206,8 +206,10 @@ rloop:
 				if (result_errno == EPIPE ||
 					result_errno == ECONNRESET)
 					libpq_append_conn_error(conn, "server closed the connection unexpectedly\n"
-											"\tThis probably means the server terminated abnormally\n"
-											"\tbefore or while processing the request.");
+											  "\tThis probably means the server terminated%s\n"
+											  "\tbefore or while processing the request.",
+											  conn->result->resultStatus == PGRES_FATAL_ERROR ?
+											  "" : "abnormally");
 				else
 					libpq_append_conn_error(conn, "SSL SYSCALL error: %s",
 											SOCK_STRERROR(result_errno,
@@ -306,8 +308,10 @@ pgtls_write(PGconn *conn, const void *ptr, size_t len)
 				result_errno = SOCK_ERRNO;
 				if (result_errno == EPIPE || result_errno == ECONNRESET)
 					libpq_append_conn_error(conn, "server closed the connection unexpectedly\n"
-											"\tThis probably means the server terminated abnormally\n"
-											"\tbefore or while processing the request.");
+											  "\tThis probably means the server terminated%s\n"
+											  "\tbefore or while processing the request.",
+											  conn->result->resultStatus == PGRES_FATAL_ERROR ?
+											  "" : "abnormally");
 				else
 					libpq_append_conn_error(conn, "SSL SYSCALL error: %s",
 											SOCK_STRERROR(result_errno,
diff --git a/src/interfaces/libpq/fe-secure.c b/src/interfaces/libpq/fe-secure.c
index bd72a87bbb..5e7136195a 100644
--- a/src/interfaces/libpq/fe-secure.c
+++ b/src/interfaces/libpq/fe-secure.c
@@ -233,8 +233,10 @@ pqsecure_raw_read(PGconn *conn, void *ptr, size_t len)
 			case EPIPE:
 			case ECONNRESET:
 				libpq_append_conn_error(conn, "server closed the connection unexpectedly\n"
-										"\tThis probably means the server terminated abnormally\n"
-										"\tbefore or while processing the request.");
+										"\tThis probably means the server terminated%s\n"
+										"\tbefore or while processing the request.",
+										conn->result->resultStatus == PGRES_FATAL_ERROR ?
+										"" : "abnormally");
 				break;
 
 			default:
@@ -395,8 +397,11 @@ retry_masked:
 				/* (strdup failure is OK, we'll cope later) */
 				snprintf(msgbuf, sizeof(msgbuf),
 						 libpq_gettext("server closed the connection unexpectedly\n"
-									   "\tThis probably means the server terminated abnormally\n"
-									   "\tbefore or while processing the request."));
+									   "\tThis probably means the server terminated%s\n"
+									   "\tbefore or while processing the request."),
+										conn->result->resultStatus == PGRES_FATAL_ERROR ?
+										"" : "abnormally");
+
 				/* keep newline out of translated string */
 				strlcat(msgbuf, "\n", sizeof(msgbuf));
 				conn->write_err_msg = strdup(msgbuf);

---

From: Bruce Momjian <[email-masked]>
Date: Thu, 23 Nov 2023 11:12:30 -0500

> know how to test this.  Thanks.

I developed the attached patch which seems to work better.  In testing
kill -3 on a backend or calling elog(FATAL) in the server for a
session, libpq's 'res' is NULL, meaning we don't have any status to
check for PGRES_FATAL_ERROR.  It is very possible that libpq just isn't
stuctured to have the PGRES_FATAL_ERROR at the point where we issue this
message, and this is not worth improving.

	test=> select pg_sleep(100);
-->	FATAL:  FATAL called
	
	server closed the connection unexpectedly
-->	        This probably means the server terminated null
	        before or while processing the request.
	The connection to the server was lost. Attempting reset: Succeeded.

  Only you can decide what is important to you.
diff --git a/src/interfaces/libpq/fe-misc.c b/src/interfaces/libpq/fe-misc.c
index 660cdec93c..64faad19df 100644
--- a/src/interfaces/libpq/fe-misc.c
+++ b/src/interfaces/libpq/fe-misc.c
@@ -749,8 +749,11 @@ retry4:
 	 */
 definitelyEOF:
 	libpq_append_conn_error(conn, "server closed the connection unexpectedly\n"
-							"\tThis probably means the server terminated abnormally\n"
-							"\tbefore or while processing the request.");
+							"\tThis probably means the server terminated%s\n"
+							"\tbefore or while processing the request.",
+							(conn->result == NULL) ? " null" :
+							(conn->result->resultStatus == PGRES_FATAL_ERROR) ?
+							"" : " abnormally");
 
 	/* Come here if lower-level code already set a suitable errorMessage */
 definitelyFailed:
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 5613c56b14..03914b97fc 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -2158,6 +2158,7 @@ pqFunctionCall3(PGconn *conn, Oid fnid,
 				if (pqGetErrorNotice3(conn, true))
 					continue;
 				status = PGRES_FATAL_ERROR;
+				fprintf(stderr, "Got 'E'\n");
 				break;
 			case 'A':			/* notify message */
 				/* handle notify and go back to processing return values */
diff --git a/src/interfaces/libpq/fe-secure-openssl.c b/src/interfaces/libpq/fe-secure-openssl.c
index f1192d28f2..f4c7f51b0a 100644
--- a/src/interfaces/libpq/fe-secure-openssl.c
+++ b/src/interfaces/libpq/fe-secure-openssl.c
@@ -206,8 +206,11 @@ rloop:
 				if (result_errno == EPIPE ||
 					result_errno == ECONNRESET)
 					libpq_append_conn_error(conn, "server closed the connection unexpectedly\n"
-											"\tThis probably means the server terminated abnormally\n"
-											"\tbefore or while processing the request.");
+											  "\tThis probably means the server terminated%s\n"
+											  "\tbefore or while processing the request.",
+											  (conn->result == NULL) ? " null" :
+											  (conn->result->resultStatus == PGRES_FATAL_ERROR) ?
+											  "" : " abnormally");
 				else
 					libpq_append_conn_error(conn, "SSL SYSCALL error: %s",
 											SOCK_STRERROR(result_errno,
@@ -306,8 +309,11 @@ pgtls_write(PGconn *conn, const void *ptr, size_t len)
 				result_errno = SOCK_ERRNO;
 				if (result_errno == EPIPE || result_errno == ECONNRESET)
 					libpq_append_conn_error(conn, "server closed the connection unexpectedly\n"
-											"\tThis probably means the server terminated abnormally\n"
-											"\tbefore or while processing the request.");
+											  "\tThis probably means the server terminated%s\n"
+											  "\tbefore or while processing the request.",
+											  (conn->result == NULL) ? " null" :
+											  (conn->result->resultStatus == PGRES_FATAL_ERROR) ?
+											  "" : " abnormally");
 				else
 					libpq_append_conn_error(conn, "SSL SYSCALL error: %s",
 											SOCK_STRERROR(result_errno,
diff --git a/src/interfaces/libpq/fe-secure.c b/src/interfaces/libpq/fe-secure.c
index bd72a87bbb..be93c2c0f9 100644
--- a/src/interfaces/libpq/fe-secure.c
+++ b/src/interfaces/libpq/fe-secure.c
@@ -233,8 +233,11 @@ pqsecure_raw_read(PGconn *conn, void *ptr, size_t len)
 			case EPIPE:
 			case ECONNRESET:
 				libpq_append_conn_error(conn, "server closed the connection unexpectedly\n"
-										"\tThis probably means the server terminated abnormally\n"
-										"\tbefore or while processing the request.");
+										"\tThis probably means the server terminated%s\n"
+										"\tbefore or while processing the request.",
+										(conn->result == NULL) ? " null" :
+										(conn->result->resultStatus == PGRES_FATAL_ERROR) ?
+										"" : " abnormally");
 				break;
 
 			default:
@@ -395,8 +398,12 @@ retry_masked:
 				/* (strdup failure is OK, we'll cope later) */
 				snprintf(msgbuf, sizeof(msgbuf),
 						 libpq_gettext("server closed the connection unexpectedly\n"
-									   "\tThis probably means the server terminated abnormally\n"
-									   "\tbefore or while processing the request."));
+									   "\tThis probably means the server terminated%s\n"
+									   "\tbefore or while processing the request."),
+									   (conn->result == NULL) ? " null" :
+									   (conn->result->resultStatus == PGRES_FATAL_ERROR) ?
+									   "" : " abnormally");
+
 				/* keep newline out of translated string */
 				strlcat(msgbuf, "\n", sizeof(msgbuf));
 				conn->write_err_msg = strdup(msgbuf);

---

From: Laurenz Albe <[email-masked]>
Date: Fri, 24 Nov 2023 04:06:22 +0100

> 	The connection to the server was lost. Attempting reset: Succeeded.

I don't thing "terminated null" is a meaningful message.

> +							(conn->result->resultStatus == PGRES_FATAL_ERROR) ?

Apart from the weird "null", will that work well for translation?

>  				/* handle notify and go back to processing return values */

That looks like a leftover debugging message.

Yours,
Laurenz Albe