public inbox for [email protected]
help / color / mirror / Atom feedRe: Why is src/test/modules/committs/t/002_standby.pl flaky?
38+ messages / 7 participants
[nested] [flat]
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
@ 2022-01-08 23:41 Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Tom Lane @ 2022-01-08 23:41 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Thomas Munro <[email protected]>; pgsql-hackers
Andrew Dunstan <[email protected]> writes:
> On 12/30/21 15:01, Thomas Munro wrote:
>> There's a wait for replay that is open coded (instead of using the
>> wait_for_catchup() routine), and sometimes the second of two such
>> waits at line 51 (in master) times out after 3 minutes with "standby
>> never caught up". It's happening on three particular Windows boxes,
>> but once also happened on the AIX box "tern".
> FYI, drongo and fairywren are run on the same AWS/EC2 Windows Server
> 2019 instance. Nothing else runs on it.
I spent a little time looking into this just now. There are similar
failures in both 002_standby.pl and 003_standby_2.pl, which is
unsurprising because there are essentially-identical test sequences
in both. What I've realized is that the issue is triggered by
this sequence:
$standby->start;
...
$primary->restart;
$primary->safe_psql('postgres', 'checkpoint');
my $primary_lsn =
$primary->safe_psql('postgres', 'select pg_current_wal_lsn()');
$standby->poll_query_until('postgres',
qq{SELECT '$primary_lsn'::pg_lsn <= pg_last_wal_replay_lsn()})
or die "standby never caught up";
(the failing poll_query_until is at line 51 in 002_standby.pl, or
line 37 in 003_standby_2.pl). That is, we have forced a primary
restart since the standby first connected to the primary, and
now we have to wait for the standby to reconnect and catch up.
*These two tests seem to be the only TAP tests that do that*.
So I think there's not really anything specific to commit_ts testing
involved, it's just a dearth of primary restarts elsewhere.
Looking at the logs in the failing cases, there's no evidence
that the standby has even detected the primary's disconnection,
which explains why it hasn't attempted to reconnect. For
example, in the most recent HEAD failure,
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2022-01-03%2018%3A04%3A41
the standby reports successful connection:
2022-01-03 18:58:04.920 UTC [179700:1] LOG: started streaming WAL from primary at 0/3000000 on timeline 1
(which we can also see in the primary's log), but after that
there's no log traffic at all except the test script's vain
checks of pg_last_wal_replay_lsn(). In the same animal's
immediately preceding successful run,
https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fairywren&dt=2022-01-03%2015%3A04%...
we see:
2022-01-03 15:59:24.186 UTC [176664:1] LOG: started streaming WAL from primary at 0/3000000 on timeline 1
2022-01-03 15:59:25.003 UTC [176664:2] LOG: replication terminated by primary server
2022-01-03 15:59:25.003 UTC [176664:3] DETAIL: End of WAL reached on timeline 1 at 0/3030CB8.
2022-01-03 15:59:25.003 UTC [176664:4] FATAL: could not send end-of-streaming message to primary: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
no COPY in progress
2022-01-03 15:59:25.005 UTC [177092:5] LOG: invalid record length at 0/3030CB8: wanted 24, got 0
...
2022-01-03 15:59:25.564 UTC [177580:1] LOG: started streaming WAL from primary at 0/3000000 on timeline 1
So for some reason, on these machines detection of walsender-initiated
connection close is unreliable ... or maybe, the walsender didn't close
the connection, but is somehow still hanging around? Don't have much idea
where to dig beyond that, but maybe someone else will. I wonder in
particular if this could be related to our recent discussions about
whether to use shutdown(2) on Windows --- could we need to do the
equivalent of 6051857fc/ed52c3707 on walsender connections?
regards, tom lane
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
@ 2022-01-09 01:17 ` Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Tom Lane @ 2022-01-09 01:17 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; Alexander Lakhin <[email protected]>; pgsql-hackers
I wrote:
> So for some reason, on these machines detection of walsender-initiated
> connection close is unreliable ... or maybe, the walsender didn't close
> the connection, but is somehow still hanging around? Don't have much idea
> where to dig beyond that, but maybe someone else will. I wonder in
> particular if this could be related to our recent discussions about
> whether to use shutdown(2) on Windows --- could we need to do the
> equivalent of 6051857fc/ed52c3707 on walsender connections?
... wait a minute. After some more study of the buildfarm logs,
it was brought home to me that these failures started happening
just after 6051857fc went in:
https://buildfarm.postgresql.org/cgi-bin/show_failures.pl?max_days=90&branch=&member=&st...
The oldest matching failure is jacana's on 2021-12-03.
(The above sweep finds an unrelated-looking failure on 2021-11-11,
but no others before 6051857fc went in on 2021-12-02. Also, it
looks likely that ed52c3707 on 2021-12-07 made the failure more
probable, because jacana's is the only matching failure before 12-07.)
So I'm now thinking it's highly likely that those commits are
causing it somehow, but how?
regards, tom lane
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
@ 2022-01-09 11:00 ` Alexander Lakhin <[email protected]>
2022-01-09 16:49 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
0 siblings, 2 replies; 38+ messages in thread
From: Alexander Lakhin @ 2022-01-09 11:00 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Thomas Munro <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; pgsql-hackers
Hello Tom,
09.01.2022 04:17, Tom Lane wrote:
>> So for some reason, on these machines detection of walsender-initiated
>> connection close is unreliable ... or maybe, the walsender didn't close
>> the connection, but is somehow still hanging around? Don't have much idea
>> where to dig beyond that, but maybe someone else will. I wonder in
>> particular if this could be related to our recent discussions about
>> whether to use shutdown(2) on Windows --- could we need to do the
>> equivalent of 6051857fc/ed52c3707 on walsender connections?
> ... wait a minute. After some more study of the buildfarm logs,
> it was brought home to me that these failures started happening
> just after 6051857fc went in:
>
> https://buildfarm.postgresql.org/cgi-bin/show_failures.pl?max_days=90&branch=&member=&st...
>
> The oldest matching failure is jacana's on 2021-12-03.
> (The above sweep finds an unrelated-looking failure on 2021-11-11,
> but no others before 6051857fc went in on 2021-12-02. Also, it
> looks likely that ed52c3707 on 2021-12-07 made the failure more
> probable, because jacana's is the only matching failure before 12-07.)
>
> So I'm now thinking it's highly likely that those commits are
> causing it somehow, but how?
>
I've managed to reproduce this failure too.
Removing "shutdown(MyProcPort->sock, SD_SEND);" doesn't help here, so
the culprit is exactly "closesocket(MyProcPort->sock);".
I've added `system("netstat -ano");` before die() in 002_standby.pl and see:
# Postmaster PID for node "primary" is 944
Proto Local Address Foreign Address State PID
...
TCP 127.0.0.1:58545 127.0.0.1:61995 FIN_WAIT_2 944
...
TCP 127.0.0.1:61995 127.0.0.1:58545 CLOSE_WAIT 1352
(Replacing SD_SEND with SD_BOTH doesn't change the behaviour.)
Looking at the libpqwalreceiver.c:
/* Now that we've consumed some input, try again */
rawlen = PQgetCopyData(conn->streamConn, &conn->recvBuf, 1);
here we get -1 on the primary disconnection.
Then we get COMMAND_OK here:
res = libpqrcv_PQgetResult(conn->streamConn);
if (PQresultStatus(res) == PGRES_COMMAND_OK)
and finally just hang at:
/* Verify that there are no more results. */
res = libpqrcv_PQgetResult(conn->streamConn);
until the standby gets interrupted by the TAP test. (That call can also
return NULL and then the test completes successfully.)
Going down through the call chain, I see that at the end of it
WaitForMultipleObjects() hangs while waiting for the primary connection
socket event. So it looks like the socket, that is closed by the
primary, can get into a state unsuitable for WaitForMultipleObjects().
I tried to check the socket state with the WSAPoll() function and
discovered that it returns POLLHUP for the "problematic" socket.
The following draft addition in latch.c:
int
WaitLatchOrSocket(Latch *latch, int wakeEvents, pgsocket sock,
long timeout, uint32 wait_event_info)
{
int ret = 0;
int rc;
WaitEvent event;
#ifdef WIN32
if (wakeEvents & WL_SOCKET_MASK) {
WSAPOLLFD pollfd;
pollfd.fd = sock;
pollfd.events = POLLRDNORM | POLLWRNORM;
pollfd.revents = 0;
int rc = WSAPoll(&pollfd, 1, 0);
if ((rc == 1) && (pollfd.revents & POLLHUP)) {
elog(WARNING, "WaitLatchOrSocket: A stream-oriented
connection was either disconnected or aborted.");
return WL_SOCKET_MASK;
}
}
#endif
makes the test 002_stanby.pl pass (100 of 100 iterations, while without
the fix I get failures roughly on each third). I'm not sure where to
place this check, maybe it's better to move it up to
libpqrcv_PQgetResult() to minimize it's footprint or to find less
Windows-specific approach, but I'd prefer a client-side fix anyway, as
graceful closing a socket by a server seems a legitimate action.
Best regards,
Alexander
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
@ 2022-01-09 16:49 ` Tom Lane <[email protected]>
1 sibling, 0 replies; 38+ messages in thread
From: Tom Lane @ 2022-01-09 16:49 UTC (permalink / raw)
To: Alexander Lakhin <[email protected]>; +Cc: Thomas Munro <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
Alexander Lakhin <[email protected]> writes:
> 09.01.2022 04:17, Tom Lane wrote:
>> ... wait a minute. After some more study of the buildfarm logs,
>> it was brought home to me that these failures started happening
>> just after 6051857fc went in:
> I've managed to reproduce this failure too.
> Removing "shutdown(MyProcPort->sock, SD_SEND);" doesn't help here, so
> the culprit is exactly "closesocket(MyProcPort->sock);".
Ugh. Did you try removing the closesocket and keeping shutdown?
I don't recall if we tried that combination before.
> ... I'm not sure where to
> place this check, maybe it's better to move it up to
> libpqrcv_PQgetResult() to minimize it's footprint or to find less
> Windows-specific approach, but I'd prefer a client-side fix anyway, as
> graceful closing a socket by a server seems a legitimate action.
What concerns me here is whether this implies that other clients
(libpq, jdbc, etc) are going to need changes as well. Maybe
libpq is okay, because we've not seen failures of the isolation
tests that use pg_cancel_backend(), but still it's worrisome.
I'm not entirely sure whether the isolationtester would notice
that a connection that should have died didn't.
regards, tom lane
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
@ 2022-01-09 19:06 ` Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
1 sibling, 1 reply; 38+ messages in thread
From: Thomas Munro @ 2022-01-09 19:06 UTC (permalink / raw)
To: Alexander Lakhin <[email protected]>; +Cc: Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
On Mon, Jan 10, 2022 at 12:00 AM Alexander Lakhin <[email protected]> wrote:
> Going down through the call chain, I see that at the end of it
> WaitForMultipleObjects() hangs while waiting for the primary connection
> socket event. So it looks like the socket, that is closed by the
> primary, can get into a state unsuitable for WaitForMultipleObjects().
I wonder if FD_CLOSE is edge-triggered, and it's already told us once.
I think that's what these Python Twisted guys are saying:
https://stackoverflow.com/questions/7598936/how-can-a-disconnected-tcp-socket-be-reliably-detected-u...
> I tried to check the socket state with the WSAPoll() function and
> discovered that it returns POLLHUP for the "problematic" socket.
Good discovery. I guess if the above theory is right, there's a
memory somewhere that makes this level-triggered as expected by users
of poll().
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
@ 2022-01-10 02:00 ` Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Thomas Munro @ 2022-01-10 02:00 UTC (permalink / raw)
To: Alexander Lakhin <[email protected]>; +Cc: Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
On Mon, Jan 10, 2022 at 8:06 AM Thomas Munro <[email protected]> wrote:
> On Mon, Jan 10, 2022 at 12:00 AM Alexander Lakhin <[email protected]> wrote:
> > Going down through the call chain, I see that at the end of it
> > WaitForMultipleObjects() hangs while waiting for the primary connection
> > socket event. So it looks like the socket, that is closed by the
> > primary, can get into a state unsuitable for WaitForMultipleObjects().
>
> I wonder if FD_CLOSE is edge-triggered, and it's already told us once.
Can you reproduce it with this patch?
Attachments:
[text/x-patch] 0001-Make-Windows-FD_CLOSE-reporting-sticky.patch (2.0K, ../../CA+hUKG+9kOBvurZoouprt0it=7XutQR17NMpfTeyYJNyc6V_5w@mail.gmail.com/2-0001-Make-Windows-FD_CLOSE-reporting-sticky.patch)
download | inline diff:
From d3cfff53911ce068d4a31fe7ac7933958936d1b0 Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Mon, 10 Jan 2022 14:45:05 +1300
Subject: [PATCH] Make Windows FD_CLOSE reporting sticky.
XXX Just testing an idea...
---
src/backend/storage/ipc/latch.c | 16 ++++++++++++++++
src/include/storage/latch.h | 1 +
2 files changed, 17 insertions(+)
diff --git a/src/backend/storage/ipc/latch.c b/src/backend/storage/ipc/latch.c
index 1d893cf863..4a61b31cd5 100644
--- a/src/backend/storage/ipc/latch.c
+++ b/src/backend/storage/ipc/latch.c
@@ -899,6 +899,7 @@ AddWaitEventToSet(WaitEventSet *set, uint32 events, pgsocket fd, Latch *latch,
event->user_data = user_data;
#ifdef WIN32
event->reset = false;
+ event->closed = false;
#endif
if (events == WL_LATCH_SET)
@@ -1882,6 +1883,20 @@ WaitEventSetWaitBlock(WaitEventSet *set, int cur_timeout,
return 1;
}
}
+
+ /*
+ * XXX: HYPOTHESIS TO TEST
+ * Windows only reports FD_CLOSE once. If we've seen that already,
+ * continue to report it.
+ */
+ if ((cur_event & WL_SOCKET_MASK) && cur_event->closed)
+ {
+ occurred_events->pos = cur_event->pos;
+ occurred_events->user_data = cur_event->user_data;
+ occurred_events->events = (cur_event->events & WL_SOCKET_MASK);
+ occurred_events->fd = cur_event->fd;
+ return 1;
+ }
}
/*
@@ -2002,6 +2017,7 @@ WaitEventSetWaitBlock(WaitEventSet *set, int cur_timeout,
{
/* EOF/error, so signal all caller-requested socket flags */
occurred_events->events |= (cur_event->events & WL_SOCKET_MASK);
+ cur_event->closed = true;
}
if (occurred_events->events != 0)
diff --git a/src/include/storage/latch.h b/src/include/storage/latch.h
index 44f9368c64..c24f46dc37 100644
--- a/src/include/storage/latch.h
+++ b/src/include/storage/latch.h
@@ -147,6 +147,7 @@ typedef struct WaitEvent
void *user_data; /* pointer provided in AddWaitEventToSet */
#ifdef WIN32
bool reset; /* Is reset of the event required? */
+ bool closed; /* Has FD_CLOSE event been reported? */
#endif
} WaitEvent;
--
2.33.1
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
@ 2022-01-10 07:00 ` Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Alexander Lakhin @ 2022-01-10 07:00 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
10.01.2022 05:00, Thomas Munro wrote:
> On Mon, Jan 10, 2022 at 8:06 AM Thomas Munro <[email protected]> wrote:
>> On Mon, Jan 10, 2022 at 12:00 AM Alexander Lakhin <[email protected]> wrote:
>>> Going down through the call chain, I see that at the end of it
>>> WaitForMultipleObjects() hangs while waiting for the primary connection
>>> socket event. So it looks like the socket, that is closed by the
>>> primary, can get into a state unsuitable for WaitForMultipleObjects().
>> I wonder if FD_CLOSE is edge-triggered, and it's already told us once.
> Can you reproduce it with this patch?
Unfortunately, this fix (with the correction "(cur_event &
WL_SOCKET_MASK)" -> "(cur_event->events & WL_SOCKET_MASK") doesn't work,
because we have two separate calls to libpqrcv_PQgetResult():
> Then we get COMMAND_OK here:
> res = libpqrcv_PQgetResult(conn->streamConn);
> if (PQresultStatus(res) == PGRES_COMMAND_OK)
> and finally just hang at:
> /* Verify that there are no more results. */
> res = libpqrcv_PQgetResult(conn->streamConn);
The libpqrcv_PQgetResult function, in turn, invokes WaitLatchOrSocket()
where WaitEvents are defined locally, and the closed flag set on the
first invocation but expected to be checked on second.
>> I've managed to reproduce this failure too.
>> Removing "shutdown(MyProcPort->sock, SD_SEND);" doesn't help here, so
>> the culprit is exactly "closesocket(MyProcPort->sock);".
>>
> Ugh. Did you try removing the closesocket and keeping shutdown?
> I don't recall if we tried that combination before.
Even with shutdown() only I still observe WaitForMultipleObjects()
hanging (and WSAPoll() returns POLLHUP for the socket).
As to your concern regarding other clients, I suspect that this issue is
caused by libpqwalreceiver' specific call pattern and may be other
clients just don't do that. I need some more time to analyze this.
Best regards,
Alexander
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
@ 2022-01-10 09:20 ` Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Thomas Munro @ 2022-01-10 09:20 UTC (permalink / raw)
To: Alexander Lakhin <[email protected]>; +Cc: Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
On Mon, Jan 10, 2022 at 8:00 PM Alexander Lakhin <[email protected]> wrote:
> The libpqrcv_PQgetResult function, in turn, invokes WaitLatchOrSocket()
> where WaitEvents are defined locally, and the closed flag set on the
> first invocation but expected to be checked on second.
D'oh, right. There's also a WaitLatchOrSocket call in walreceiver.c.
We'd need a long-lived WaitEventSet common across all of these sites,
which is hard here (because the socket might change under you, as
discussed in other threads that introduced long lived WaitEventSets to
other places but not here).
/me wonders if it's possible that graceful FD_CLOSE is reported only
once, but abortive/error FD_CLOSE is reported multiple times...
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
@ 2022-01-10 09:40 ` Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Thomas Munro @ 2022-01-10 09:40 UTC (permalink / raw)
To: Alexander Lakhin <[email protected]>; +Cc: Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
On Mon, Jan 10, 2022 at 10:20 PM Thomas Munro <[email protected]> wrote:
> On Mon, Jan 10, 2022 at 8:00 PM Alexander Lakhin <[email protected]> wrote:
> > The libpqrcv_PQgetResult function, in turn, invokes WaitLatchOrSocket()
> > where WaitEvents are defined locally, and the closed flag set on the
> > first invocation but expected to be checked on second.
>
> D'oh, right. There's also a WaitLatchOrSocket call in walreceiver.c.
> We'd need a long-lived WaitEventSet common across all of these sites,
> which is hard here (because the socket might change under you, as
> discussed in other threads that introduced long lived WaitEventSets to
> other places but not here).
This is super quick-and-dirty code (and doesn't handle some errors or
socket changes correctly), but does it detect the closed socket?
Attachments:
[text/x-patch] v2-0001-Make-Windows-FD_CLOSE-reporting-sticky.patch (2.1K, ../../CA+hUKGKPpbC2EM2tSy5-1cfj+-1d2PVwLU4kBNg8E+og+B3fbg@mail.gmail.com/2-v2-0001-Make-Windows-FD_CLOSE-reporting-sticky.patch)
download | inline diff:
From 3590a8c9b3e8992a65dea7a9d6aecdca2f25582d Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Mon, 10 Jan 2022 14:45:05 +1300
Subject: [PATCH v2 1/2] Make Windows FD_CLOSE reporting sticky.
XXX Just testing an idea...
---
src/backend/storage/ipc/latch.c | 16 ++++++++++++++++
src/include/storage/latch.h | 1 +
2 files changed, 17 insertions(+)
diff --git a/src/backend/storage/ipc/latch.c b/src/backend/storage/ipc/latch.c
index 1d893cf863..8f3176a00f 100644
--- a/src/backend/storage/ipc/latch.c
+++ b/src/backend/storage/ipc/latch.c
@@ -899,6 +899,7 @@ AddWaitEventToSet(WaitEventSet *set, uint32 events, pgsocket fd, Latch *latch,
event->user_data = user_data;
#ifdef WIN32
event->reset = false;
+ event->closed = false;
#endif
if (events == WL_LATCH_SET)
@@ -1882,6 +1883,20 @@ WaitEventSetWaitBlock(WaitEventSet *set, int cur_timeout,
return 1;
}
}
+
+ /*
+ * XXX: HYPOTHESIS TO TEST
+ * Windows only reports FD_CLOSE once. If we've seen that already,
+ * continue to report it.
+ */
+ if ((cur_event->events & WL_SOCKET_MASK) && cur_event->closed)
+ {
+ occurred_events->pos = cur_event->pos;
+ occurred_events->user_data = cur_event->user_data;
+ occurred_events->events = (cur_event->events & WL_SOCKET_MASK);
+ occurred_events->fd = cur_event->fd;
+ return 1;
+ }
}
/*
@@ -2002,6 +2017,7 @@ WaitEventSetWaitBlock(WaitEventSet *set, int cur_timeout,
{
/* EOF/error, so signal all caller-requested socket flags */
occurred_events->events |= (cur_event->events & WL_SOCKET_MASK);
+ cur_event->closed = true;
}
if (occurred_events->events != 0)
diff --git a/src/include/storage/latch.h b/src/include/storage/latch.h
index 44f9368c64..c24f46dc37 100644
--- a/src/include/storage/latch.h
+++ b/src/include/storage/latch.h
@@ -147,6 +147,7 @@ typedef struct WaitEvent
void *user_data; /* pointer provided in AddWaitEventToSet */
#ifdef WIN32
bool reset; /* Is reset of the event required? */
+ bool closed; /* Has FD_CLOSE event been reported? */
#endif
} WaitEvent;
--
2.33.1
[text/x-patch] v2-0002-XXX-Quick-hack-to-use-persistent-WaitEventSet-in-.patch (12.0K, ../../CA+hUKGKPpbC2EM2tSy5-1cfj+-1d2PVwLU4kBNg8E+og+B3fbg@mail.gmail.com/3-v2-0002-XXX-Quick-hack-to-use-persistent-WaitEventSet-in-.patch)
download | inline diff:
From 755ee3da7934d33a706d59eb0abc0e4f1829acb0 Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Mon, 10 Jan 2022 22:02:21 +1300
Subject: [PATCH v2 2/2] XXX Quick hack to use persistent WaitEventSet in
walreceiver
XXX There are several things wrong with this, it's just experimental code.
---
.../libpqwalreceiver/libpqwalreceiver.c | 97 ++++++++++++-------
src/backend/replication/walreceiver.c | 12 +--
src/include/replication/walreceiver.h | 8 ++
3 files changed, 75 insertions(+), 42 deletions(-)
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index c08e599eef..f8dd76e2d0 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -40,6 +40,7 @@ void _PG_init(void);
struct WalReceiverConn
{
+ WaitEventSet *wes;
/* Current connection to the primary, if any */
PGconn *streamConn;
/* Used to remember if the connection is logical or physical */
@@ -82,6 +83,7 @@ static WalRcvExecResult *libpqrcv_exec(WalReceiverConn *conn,
const int nRetTypes,
const Oid *retTypes);
static void libpqrcv_disconnect(WalReceiverConn *conn);
+static int libpqrcv_wait(WalReceiverConn *conn, int timeout, int wait_event);
static WalReceiverFunctionsType PQWalReceiverFunctions = {
libpqrcv_connect,
@@ -98,12 +100,13 @@ static WalReceiverFunctionsType PQWalReceiverFunctions = {
libpqrcv_create_slot,
libpqrcv_get_backend_pid,
libpqrcv_exec,
- libpqrcv_disconnect
+ libpqrcv_disconnect,
+ libpqrcv_wait
};
/* Prototypes for private functions */
-static PGresult *libpqrcv_PQexec(PGconn *streamConn, const char *query);
-static PGresult *libpqrcv_PQgetResult(PGconn *streamConn);
+static PGresult *libpqrcv_PQexec(WalReceiverConn *conn, const char *query);
+static PGresult *libpqrcv_PQgetResult(WalReceiverConn *conn);
static char *stringlist_to_identifierstr(PGconn *conn, List *strings);
/*
@@ -182,6 +185,15 @@ libpqrcv_connect(const char *conninfo, bool logical, const char *appname,
return NULL;
}
+ conn->wes = CreateWaitEventSet(TopMemoryContext, 3);
+ AddWaitEventToSet(conn->wes, WL_SOCKET_WRITEABLE,
+ PQsocket(conn->streamConn), NULL, NULL);
+ AddWaitEventToSet(conn->wes, WL_LATCH_SET, PGINVALID_SOCKET,
+ MyLatch, NULL);
+ AddWaitEventToSet(conn->wes, WL_EXIT_ON_PM_DEATH, PGINVALID_SOCKET,
+ NULL, NULL);
+
+
/*
* Poll connection until we have OK or FAILED status.
*
@@ -191,7 +203,7 @@ libpqrcv_connect(const char *conninfo, bool logical, const char *appname,
do
{
int io_flag;
- int rc;
+ WaitEvent event;
if (status == PGRES_POLLING_READING)
io_flag = WL_SOCKET_READABLE;
@@ -203,21 +215,20 @@ libpqrcv_connect(const char *conninfo, bool logical, const char *appname,
else
io_flag = WL_SOCKET_WRITEABLE;
- rc = WaitLatchOrSocket(MyLatch,
- WL_EXIT_ON_PM_DEATH | WL_LATCH_SET | io_flag,
- PQsocket(conn->streamConn),
- 0,
- WAIT_EVENT_LIBPQWALRECEIVER_CONNECT);
+ ModifyWaitEvent(conn->wes, 0, io_flag, NULL);
+ if (WaitEventSetWait(conn->wes, -1, &event, 1,
+ WAIT_EVENT_LIBPQWALRECEIVER_CONNECT) < 1)
+ continue;
/* Interrupted? */
- if (rc & WL_LATCH_SET)
+ if (event.events & WL_LATCH_SET)
{
ResetLatch(MyLatch);
ProcessWalRcvInterrupts();
}
/* If socket is ready, advance the libpq state machine */
- if (rc & io_flag)
+ if (event.events & io_flag)
status = PQconnectPoll(conn->streamConn);
} while (status != PGRES_POLLING_OK && status != PGRES_POLLING_FAILED);
@@ -231,7 +242,7 @@ libpqrcv_connect(const char *conninfo, bool logical, const char *appname,
{
PGresult *res;
- res = libpqrcv_PQexec(conn->streamConn,
+ res = libpqrcv_PQexec(conn,
ALWAYS_SECURE_SEARCH_PATH_SQL);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
@@ -359,7 +370,7 @@ libpqrcv_identify_system(WalReceiverConn *conn, TimeLineID *primary_tli)
* Get the system identifier and timeline ID as a DataRow message from the
* primary server.
*/
- res = libpqrcv_PQexec(conn->streamConn, "IDENTIFY_SYSTEM");
+ res = libpqrcv_PQexec(conn, "IDENTIFY_SYSTEM");
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
PQclear(res);
@@ -482,7 +493,7 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
options->proto.physical.startpointTLI);
/* Start streaming. */
- res = libpqrcv_PQexec(conn->streamConn, cmd.data);
+ res = libpqrcv_PQexec(conn, cmd.data);
pfree(cmd.data);
if (PQresultStatus(res) == PGRES_COMMAND_OK)
@@ -532,7 +543,7 @@ libpqrcv_endstreaming(WalReceiverConn *conn, TimeLineID *next_tli)
* If we had not yet received CopyDone from the backend, PGRES_COPY_OUT is
* also possible in case we aborted the copy in mid-stream.
*/
- res = libpqrcv_PQgetResult(conn->streamConn);
+ res = libpqrcv_PQgetResult(conn);
if (PQresultStatus(res) == PGRES_TUPLES_OK)
{
/*
@@ -547,7 +558,7 @@ libpqrcv_endstreaming(WalReceiverConn *conn, TimeLineID *next_tli)
PQclear(res);
/* the result set should be followed by CommandComplete */
- res = libpqrcv_PQgetResult(conn->streamConn);
+ res = libpqrcv_PQgetResult(conn);
}
else if (PQresultStatus(res) == PGRES_COPY_OUT)
{
@@ -561,7 +572,7 @@ libpqrcv_endstreaming(WalReceiverConn *conn, TimeLineID *next_tli)
pchomp(PQerrorMessage(conn->streamConn)))));
/* CommandComplete should follow */
- res = libpqrcv_PQgetResult(conn->streamConn);
+ res = libpqrcv_PQgetResult(conn);
}
if (PQresultStatus(res) != PGRES_COMMAND_OK)
@@ -572,7 +583,7 @@ libpqrcv_endstreaming(WalReceiverConn *conn, TimeLineID *next_tli)
PQclear(res);
/* Verify that there are no more results */
- res = libpqrcv_PQgetResult(conn->streamConn);
+ res = libpqrcv_PQgetResult(conn);
if (res != NULL)
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -597,7 +608,7 @@ libpqrcv_readtimelinehistoryfile(WalReceiverConn *conn,
* Request the primary to send over the history file for given timeline.
*/
snprintf(cmd, sizeof(cmd), "TIMELINE_HISTORY %u", tli);
- res = libpqrcv_PQexec(conn->streamConn, cmd);
+ res = libpqrcv_PQexec(conn, cmd);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
PQclear(res);
@@ -641,8 +652,9 @@ libpqrcv_readtimelinehistoryfile(WalReceiverConn *conn,
* May return NULL, rather than an error result, on failure.
*/
static PGresult *
-libpqrcv_PQexec(PGconn *streamConn, const char *query)
+libpqrcv_PQexec(WalReceiverConn *conn, const char *query)
{
+ PGconn *streamConn = conn->streamConn;
PGresult *lastResult = NULL;
/*
@@ -665,7 +677,7 @@ libpqrcv_PQexec(PGconn *streamConn, const char *query)
/* Wait for, and collect, the next PGresult. */
PGresult *result;
- result = libpqrcv_PQgetResult(streamConn);
+ result = libpqrcv_PQgetResult(conn);
if (result == NULL)
break; /* query is complete, or failure */
@@ -690,30 +702,31 @@ libpqrcv_PQexec(PGconn *streamConn, const char *query)
* Perform the equivalent of PQgetResult(), but watch for interrupts.
*/
static PGresult *
-libpqrcv_PQgetResult(PGconn *streamConn)
+libpqrcv_PQgetResult(WalReceiverConn *conn)
{
+ PGconn *streamConn = conn->streamConn;
+
/*
* Collect data until PQgetResult is ready to get the result without
* blocking.
*/
while (PQisBusy(streamConn))
{
- int rc;
+ WaitEvent event;
/*
* We don't need to break down the sleep into smaller increments,
* since we'll get interrupted by signals and can handle any
* interrupts here.
*/
- rc = WaitLatchOrSocket(MyLatch,
- WL_EXIT_ON_PM_DEATH | WL_SOCKET_READABLE |
- WL_LATCH_SET,
- PQsocket(streamConn),
- 0,
- WAIT_EVENT_LIBPQWALRECEIVER_RECEIVE);
+ ModifyWaitEvent(conn->wes, 0, WL_SOCKET_READABLE, NULL);
+
+ if (WaitEventSetWait(conn->wes, -1, &event, 1,
+ WAIT_EVENT_LIBPQWALRECEIVER_RECEIVE) < 1)
+ continue;
/* Interrupted? */
- if (rc & WL_LATCH_SET)
+ if (event.events & WL_LATCH_SET)
{
ResetLatch(MyLatch);
ProcessWalRcvInterrupts();
@@ -737,12 +750,28 @@ libpqrcv_PQgetResult(PGconn *streamConn)
static void
libpqrcv_disconnect(WalReceiverConn *conn)
{
+ FreeWaitEventSet(conn->wes);
PQfinish(conn->streamConn);
if (conn->recvBuf != NULL)
PQfreemem(conn->recvBuf);
pfree(conn);
}
+/*
+ * Wait for data, latch, or timeout.
+ */
+static int
+libpqrcv_wait(WalReceiverConn *conn, int timeout, int wait_event)
+{
+ WaitEvent event;
+
+ if (WaitEventSetWait(conn->wes, timeout, &event, 1, wait_event) == 1)
+ return event.events;
+
+ return 0;
+}
+
+
/*
* Receive a message available from XLOG stream.
*
@@ -793,13 +822,13 @@ libpqrcv_receive(WalReceiverConn *conn, char **buffer,
{
PGresult *res;
- res = libpqrcv_PQgetResult(conn->streamConn);
+ res = libpqrcv_PQgetResult(conn);
if (PQresultStatus(res) == PGRES_COMMAND_OK)
{
PQclear(res);
/* Verify that there are no more results. */
- res = libpqrcv_PQgetResult(conn->streamConn);
+ res = libpqrcv_PQgetResult(conn);
if (res != NULL)
{
PQclear(res);
@@ -941,7 +970,7 @@ libpqrcv_create_slot(WalReceiverConn *conn, const char *slotname,
appendStringInfoString(&cmd, " PHYSICAL RESERVE_WAL");
}
- res = libpqrcv_PQexec(conn->streamConn, cmd.data);
+ res = libpqrcv_PQexec(conn, cmd.data);
pfree(cmd.data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
@@ -1068,7 +1097,7 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("the query interface requires a database connection")));
- pgres = libpqrcv_PQexec(conn->streamConn, query);
+ pgres = libpqrcv_PQexec(conn, query);
switch (PQresultStatus(pgres))
{
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 7a7eb3784e..f308ea9c24 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -507,13 +507,9 @@ WalReceiverMain(void)
* could add and remove just the socket each time, potentially
* avoiding some system calls.
*/
- Assert(wait_fd != PGINVALID_SOCKET);
- rc = WaitLatchOrSocket(MyLatch,
- WL_EXIT_ON_PM_DEATH | WL_SOCKET_READABLE |
- WL_TIMEOUT | WL_LATCH_SET,
- wait_fd,
- NAPTIME_PER_CYCLE,
- WAIT_EVENT_WAL_RECEIVER_MAIN);
+ //Assert(wait_fd != PGINVALID_SOCKET);
+ rc = walrcv_wait(wrconn, NAPTIME_PER_CYCLE,
+ WAIT_EVENT_WAL_RECEIVER_MAIN);
if (rc & WL_LATCH_SET)
{
ResetLatch(MyLatch);
@@ -532,7 +528,7 @@ WalReceiverMain(void)
XLogWalRcvSendReply(true, false);
}
}
- if (rc & WL_TIMEOUT)
+ if (rc == 0)
{
/*
* We didn't receive anything new. If we haven't heard
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 0b607ed777..3e80ff63ce 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -380,6 +380,11 @@ typedef WalRcvExecResult *(*walrcv_exec_fn) (WalReceiverConn *conn,
*/
typedef void (*walrcv_disconnect_fn) (WalReceiverConn *conn);
+/*
+ * XXX Wait for readable data, timeout, or latch.
+ */
+typedef int (*walrcv_wait_fn) (WalReceiverConn *conn, int timeout, int wait_event);
+
typedef struct WalReceiverFunctionsType
{
walrcv_connect_fn walrcv_connect;
@@ -397,6 +402,7 @@ typedef struct WalReceiverFunctionsType
walrcv_get_backend_pid_fn walrcv_get_backend_pid;
walrcv_exec_fn walrcv_exec;
walrcv_disconnect_fn walrcv_disconnect;
+ walrcv_wait_fn walrcv_wait;
} WalReceiverFunctionsType;
extern PGDLLIMPORT WalReceiverFunctionsType *WalReceiverFunctions;
@@ -431,6 +437,8 @@ extern PGDLLIMPORT WalReceiverFunctionsType *WalReceiverFunctions;
WalReceiverFunctions->walrcv_exec(conn, exec, nRetTypes, retTypes)
#define walrcv_disconnect(conn) \
WalReceiverFunctions->walrcv_disconnect(conn)
+#define walrcv_wait(conn, timeout, wait_event) \
+ WalReceiverFunctions->walrcv_wait(conn, timeout, wait_event)
static inline void
walrcv_clear_result(WalRcvExecResult *walres)
--
2.33.1
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
@ 2022-01-10 17:00 ` Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Alexander Lakhin @ 2022-01-10 17:00 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
10.01.2022 12:40, Thomas Munro wrote:
> This is super quick-and-dirty code (and doesn't handle some errors or
> socket changes correctly), but does it detect the closed socket?
Yes, it fixes the behaviour and makes the 002_standby test pass (100 of
100 iterations). I'm yet to find out whether the other
WaitLatchOrSocket' users (e. g. postgres_fdw) can suffer from the
disconnected socket state, but this approach definitely works for
walreceiver.
Best regards,
Alexander
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
@ 2022-01-10 20:52 ` Thomas Munro <[email protected]>
2022-01-10 21:21 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andrew Dunstan <[email protected]>
2022-01-10 21:25 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
0 siblings, 3 replies; 38+ messages in thread
From: Thomas Munro @ 2022-01-10 20:52 UTC (permalink / raw)
To: Alexander Lakhin <[email protected]>; +Cc: Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
On Tue, Jan 11, 2022 at 6:00 AM Alexander Lakhin <[email protected]> wrote:
> 10.01.2022 12:40, Thomas Munro wrote:
> > This is super quick-and-dirty code (and doesn't handle some errors or
> > socket changes correctly), but does it detect the closed socket?
> Yes, it fixes the behaviour and makes the 002_standby test pass (100 of
> 100 iterations).
Thanks for testing. That result does seem to confirm the hypothesis
that FD_CLOSE is reported only once for the socket on graceful
shutdown (that is, it's edge-triggered and incidentally you won't get
FD_READ), so you need to keep track of it carefully. Incidentally,
another observation is that your WSAPoll() test appears to be
returning POLLHUP where at least Linux, FreeBSD and Solaris would not:
a socket that is only half shut down (the primary shut down its end
gracefully, but walreceiver did not), so I suspect Windows' POLLHUP
might have POLLRDHUP semantics.
> I'm yet to find out whether the other
> WaitLatchOrSocket' users (e. g. postgres_fdw) can suffer from the
> disconnected socket state, but this approach definitely works for
> walreceiver.
I see where you're going: there might be safe call sequences and
unsafe call sequences, and maybe walreceiver is asking for trouble by
double-polling. I'm not sure about that; I got the impression
recently that it's possible to get FD_CLOSE while you still have
buffered data to read, so then the next recv() will return > 0 and
then we don't have any state left anywhere to remember that we saw
FD_CLOSE, even if you're careful to poll and read in the ideal
sequence. I could be wrong, and it would be nice if there is an easy
fix along those lines... The documentation around FD_CLOSE is
unclear.
I do plan to make a higher quality patch like the one I showed
(material from earlier unfinished work[1] that needs a bit more
infrastructure), but to me that's new feature/efficiency work, not
something we'd want to back-patch.
Hmm, one thing I'm still unclear on: did this problem really start
with 6051857fc/ed52c3707? My initial email in this thread lists
similar failures going back further, doesn't it? (And what's tern
doing mixed up in this mess?)
[1] https://www.postgresql.org/message-id/flat/CA%2BhUKGJPaygh-6WHEd0FnH89GrkTpVyN_ew9ckv3%2BnwjmLcSeg%4...
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
@ 2022-01-10 21:21 ` Andrew Dunstan <[email protected]>
2 siblings, 0 replies; 38+ messages in thread
From: Andrew Dunstan @ 2022-01-10 21:21 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; Alexander Lakhin <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
On 1/10/22 15:52, Thomas Munro wrote:
>
> Hmm, one thing I'm still unclear on: did this problem really start
> with 6051857fc/ed52c3707? My initial email in this thread lists
> similar failures going back further, doesn't it? (And what's tern
> doing mixed up in this mess?)
Your list contains at least some false positives. e.g.
<https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=jacana&dt=2019-12-22%2014:19:02;
which has a different script failing.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
@ 2022-01-10 21:25 ` Tom Lane <[email protected]>
2022-03-19 08:47 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Noah Misch <[email protected]>
2 siblings, 1 reply; 38+ messages in thread
From: Tom Lane @ 2022-01-10 21:25 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Alexander Lakhin <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
Thomas Munro <[email protected]> writes:
> Hmm, one thing I'm still unclear on: did this problem really start
> with 6051857fc/ed52c3707? My initial email in this thread lists
> similar failures going back further, doesn't it? (And what's tern
> doing mixed up in this mess?)
Well, those earlier ones may be committs failures, but a lot of
them contain different-looking symptoms, eg pg_ctl failures.
tern's failure at
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=tern&dt=2021-07-17+10%3A57%3A49
does look similar, but we can see in its log that the standby
*did* notice the primary disconnection and then reconnect:
2021-07-17 16:29:08.248 UTC [17498380:2] LOG: replication terminated by primary server
2021-07-17 16:29:08.248 UTC [17498380:3] DETAIL: End of WAL reached on timeline 1 at 0/30378F8.
2021-07-17 16:29:08.248 UTC [17498380:4] FATAL: could not send end-of-streaming message to primary: no COPY in progress
2021-07-17 16:29:08.248 UTC [25166230:5] LOG: invalid record length at 0/30378F8: wanted 24, got 0
2021-07-17 16:29:08.350 UTC [16318578:1] FATAL: could not connect to the primary server: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2021-07-17 16:29:36.369 UTC [7077918:1] FATAL: could not connect to the primary server: FATAL: the database system is starting up
2021-07-17 16:29:36.380 UTC [11338028:1] FATAL: could not connect to the primary server: FATAL: the database system is starting up
...
2021-07-17 16:29:36.881 UTC [17367092:1] LOG: started streaming WAL from primary at 0/3000000 on timeline 1
So I'm not sure what happened there, but it's not an instance
of this problem. One thing that looks a bit suspicious is
this in the primary's log:
2021-07-17 16:26:47.832 UTC [12386550:1] LOG: using stale statistics instead of current ones because stats collector is not responding
which makes me wonder if the timeout is down to out-of-date
pg_stats data. The loop in 002_standby.pl doesn't appear to
depend on the stats collector:
my $primary_lsn =
$primary->safe_psql('postgres', 'select pg_current_wal_lsn()');
$standby->poll_query_until('postgres',
qq{SELECT '$primary_lsn'::pg_lsn <= pg_last_wal_replay_lsn()})
or die "standby never caught up";
but maybe I'm missing the connection.
Apropos of that, it's worth noting that wait_for_catchup *is*
dependent on up-to-date stats, and here's a recent run where
it sure looks like the timeout cause is AWOL stats collector:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sungazer&dt=2022-01-10%2004%3A51%3A34
I wonder if we should refactor wait_for_catchup to probe the
standby directly instead of relying on the upstream's view.
regards, tom lane
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 21:25 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
@ 2022-03-19 08:47 ` Noah Misch <[email protected]>
2022-03-22 02:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Noah Misch @ 2022-03-19 08:47 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Thomas Munro <[email protected]>; Alexander Lakhin <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
On Mon, Jan 10, 2022 at 04:25:27PM -0500, Tom Lane wrote:
> Apropos of that, it's worth noting that wait_for_catchup *is*
> dependent on up-to-date stats, and here's a recent run where
> it sure looks like the timeout cause is AWOL stats collector:
>
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sungazer&dt=2022-01-10%2004%3A51%3A34
>
> I wonder if we should refactor wait_for_catchup to probe the
> standby directly instead of relying on the upstream's view.
It would be nice. For logical replication tests, do we have a monitoring API
independent of the stats collector? If not and we don't want to add one, a
hacky alternative might be for wait_for_catchup to run a WAL-writing command
every ~20s. That way, if the stats collector misses the datagram about the
standby reaching a certain LSN, the stats collector would have more chances.
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 21:25 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-03-19 08:47 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Noah Misch <[email protected]>
@ 2022-03-22 02:16 ` Thomas Munro <[email protected]>
2022-03-22 03:13 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Thomas Munro @ 2022-03-22 02:16 UTC (permalink / raw)
To: Noah Misch <[email protected]>; +Cc: Tom Lane <[email protected]>; Alexander Lakhin <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
I have a new socket abstraction patch that should address the known
Windows socket/event bugs, but it's a little bigger than I thought it
would be, not quite ready, and now too late to expect people to review
for 15, so I think it should go into the next cycle. I've bounced
https://commitfest.postgresql.org/37/3523/ into the next CF. We'll
need to do something like 75674c7e for master.
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 21:25 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-03-19 08:47 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Noah Misch <[email protected]>
2022-03-22 02:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
@ 2022-03-22 03:13 ` Tom Lane <[email protected]>
0 siblings, 0 replies; 38+ messages in thread
From: Tom Lane @ 2022-03-22 03:13 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Noah Misch <[email protected]>; Alexander Lakhin <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
Thomas Munro <[email protected]> writes:
> I have a new socket abstraction patch that should address the known
> Windows socket/event bugs, but it's a little bigger than I thought it
> would be, not quite ready, and now too late to expect people to review
> for 15, so I think it should go into the next cycle. I've bounced
> https://commitfest.postgresql.org/37/3523/ into the next CF. We'll
> need to do something like 75674c7e for master.
OK. You want me to push 75674c7e to HEAD?
regards, tom lane
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
@ 2022-01-11 15:00 ` Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2 siblings, 1 reply; 38+ messages in thread
From: Alexander Lakhin @ 2022-01-11 15:00 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
10.01.2022 23:52, Thomas Munro wrote:
>> I'm yet to find out whether the other
>> WaitLatchOrSocket' users (e. g. postgres_fdw) can suffer from the
>> disconnected socket state, but this approach definitely works for
>> walreceiver.
> I see where you're going: there might be safe call sequences and
> unsafe call sequences, and maybe walreceiver is asking for trouble by
> double-polling. I'm not sure about that; I got the impression
> recently that it's possible to get FD_CLOSE while you still have
> buffered data to read, so then the next recv() will return > 0 and
> then we don't have any state left anywhere to remember that we saw
> FD_CLOSE, even if you're careful to poll and read in the ideal
> sequence. I could be wrong, and it would be nice if there is an easy
> fix along those lines... The documentation around FD_CLOSE is
> unclear.
I had no strong opinion regarding unsafe sequence, though initially I
suspected that exactly the second libpqrcv_PQgetResult call could cause
the issue. But after digging into WaitLatchOrSocket I'd inclined to put
the fix deeper to satisfy all possible callers.
At the other hand, I've shared Tom's concerns regarding other clients,
that can stuck on WaitForMultipleObjects() just as walreceiver does, and
hoped that only walreceiver suffer from a graceful server socket closing.
So to get these doubts cleared, I've made a simple test for postgres_fdw
(please look at the attachment; you can put it into
contrib/postgres_fdw/t and run `vcregress taptest contrib\postgres_fdw`).
This test shows for me:
===
...
t/001_disconnection.pl .. # 12:13:39.481084 executing query...
# 12:13:43.245277 result: 0
# 0|0
# 12:13:43.246342 executing query...
# 12:13:46.525924 result: 0
# 0|0
# 12:13:46.527097 executing query...
# 12:13:47.745176 result: 3
#
# psql:<stdin>:1: WARNING: no connection to the server
# psql:<stdin>:1: ERROR: FATAL: terminating connection due to
administrator co
mmand
# server closed the connection unexpectedly
# This probably means the server terminated abnormally
# before or while processing the request.
# CONTEXT: remote SQL command: FETCH 100 FROM c1
# 12:13:47.794612 executing query...
# 12:13:51.073318 result: 0
# 0|0
# 12:13:51.074347 executing query...
===
With the simple logging added to connection.c:
/* Sleep until there's something to do */
elog(LOG, "pgfdw_get_result before WaitLatchOrSocket");
wc = WaitLatchOrSocket(MyLatch,
WL_LATCH_SET | WL_SOCKET_READABLE |
WL_EXIT_ON_PM_DEATH,
PQsocket(conn),
-1L, PG_WAIT_EXTENSION);
elog(LOG, "pgfdw_get_result after WaitLatchOrSocket");
I see in 001_disconnection_local.log:
...
2022-01-11 15:13:52.875 MSK|Administrator|postgres|61dd747f.5e4|LOG:
pgfdw_get_result after WaitLatchOrSocket
2022-01-11 15:13:52.875
MSK|Administrator|postgres|61dd747f.5e4|STATEMENT: SELECT * FROM large
WHERE a = fx2(a)
2022-01-11 15:13:52.875 MSK|Administrator|postgres|61dd747f.5e4|LOG:
pgfdw_get_result before WaitLatchOrSocket
2022-01-11 15:13:52.875
MSK|Administrator|postgres|61dd747f.5e4|STATEMENT: SELECT * FROM large
WHERE a = fx2(a)
2022-01-11 15:14:36.976 MSK|||61dd74ac.840|DEBUG: autovacuum:
processing database "postgres"
2022-01-11 15:14:51.088 MSK|Administrator|postgres|61dd747f.5e4|LOG:
pgfdw_get_result after WaitLatchOrSocket
2022-01-11 15:14:51.088
MSK|Administrator|postgres|61dd747f.5e4|STATEMENT: SELECT * FROM large
WHERE a = fx2(a)
2022-01-11 15:14:51.089 MSK|Administrator|postgres|61dd747f.5e4|LOG:
pgfdw_get_result before WaitLatchOrSocket
2022-01-11 15:14:51.089
MSK|Administrator|postgres|61dd747f.5e4|STATEMENT: SELECT * FROM large
WHERE a = fx2(a)
2022-01-11 15:15:37.006 MSK|||61dd74e9.9e8|DEBUG: autovacuum:
processing database "postgres"
2022-01-11 15:16:37.116 MSK|||61dd7525.ad0|DEBUG: autovacuum:
processing database "postgres"
2022-01-11 15:17:37.225 MSK|||61dd7561.6a0|DEBUG: autovacuum:
processing database "postgres"
2022-01-11 15:18:36.916 MSK|||61dd7470.704|LOG: checkpoint starting: time
...
2022-01-11 15:36:38.225 MSK|||61dd79d6.2a0|DEBUG: autovacuum:
processing database "postgres"
...
So here we get similar hanging on WaitLatchOrSocket().
Just to make sure that it's indeed the same issue, I've removed socket
shutdown&close and the test executed to the end (several times). Argh.
Best regards,
Alexander
Attachments:
[application/x-perl] 001_disconnection.pl (2.2K, ../../[email protected]/2-001_disconnection.pl)
download
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
@ 2022-01-11 20:10 ` Thomas Munro <[email protected]>
2022-01-11 20:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Thomas Munro @ 2022-01-11 20:10 UTC (permalink / raw)
To: Alexander Lakhin <[email protected]>; +Cc: Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
On Wed, Jan 12, 2022 at 4:00 AM Alexander Lakhin <[email protected]> wrote:
> So here we get similar hanging on WaitLatchOrSocket().
> Just to make sure that it's indeed the same issue, I've removed socket
> shutdown&close and the test executed to the end (several times). Argh.
Ouch. I think our options at this point are:
1. Revert 6051857fc (and put it back when we have a working
long-lived WES as I showed). This is not very satisfying, now that we
understand the bug, because even without that change I guess you must
be able to reach the hanging condition by using Windows postgres_fdw
to talk to a non-Windows server (ie a normal TCP stack with graceful
shutdown/linger on process exit).
2. Put your poll() check into the READABLE side. There's some
precedent for that sort of kludge on the WRITEABLE side (and a
rejection of the fragile idea that clients of latch.c should only
perform "safe" sequences):
/*
* Windows does not guarantee to log an FD_WRITE network event
* indicating that more data can be sent unless the previous send()
* failed with WSAEWOULDBLOCK. While our caller might well have made
* such a call, we cannot assume that here. Therefore, if waiting for
* write-ready, force the issue by doing a dummy send(). If the dummy
* send() succeeds, assume that the socket is in fact write-ready, and
* return immediately. Also, if it fails with something other than
* WSAEWOULDBLOCK, return a write-ready indication to let our caller
* deal with the error condition.
*/
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
@ 2022-01-11 20:16 ` Tom Lane <[email protected]>
2022-01-12 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Tom Lane @ 2022-01-11 20:16 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Alexander Lakhin <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
Thomas Munro <[email protected]> writes:
> Ouch. I think our options at this point are:
> 1. Revert 6051857fc (and put it back when we have a working
> long-lived WES as I showed). This is not very satisfying, now that we
> understand the bug, because even without that change I guess you must
> be able to reach the hanging condition by using Windows postgres_fdw
> to talk to a non-Windows server (ie a normal TCP stack with graceful
> shutdown/linger on process exit).
It'd be worth checking, perhaps. One thing I've been wondering all
along is how much of this behavior is specific to the local-loopback
case where Windows can see both ends of the connection. You'd think
that they couldn't long get away with such blatant violations of the
TCP specs when talking to external servers, because the failures
would be visible to everyone with a web browser.
regards, tom lane
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 20:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
@ 2022-01-12 07:00 ` Alexander Lakhin <[email protected]>
2022-01-13 06:36 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Alexander Lakhin @ 2022-01-12 07:00 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Thomas Munro <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; pgsql-hackers
> 11.01.2022 23:16, Tom Lane wrote:
>> Thomas Munro <[email protected]> writes:
>>> Ouch. I think our options at this point are:
>>> 1. Revert 6051857fc (and put it back when we have a working
>>> long-lived WES as I showed). This is not very satisfying, now that we
>>> understand the bug, because even without that change I guess you must
>>> be able to reach the hanging condition by using Windows postgres_fdw
>>> to talk to a non-Windows server (ie a normal TCP stack with graceful
>>> shutdown/linger on process exit).
>> It'd be worth checking, perhaps. One thing I've been wondering all
>> along is how much of this behavior is specific to the local-loopback
>> case where Windows can see both ends of the connection. You'd think
>> that they couldn't long get away with such blatant violations of the
>> TCP specs when talking to external servers, because the failures
>> would be visible to everyone with a web browser.
I've split my test (both parts attached) and run it on two virtual
machines with clean builds from master (ac7c8075) on both (just the
debugging output added to connection.c). I provide probably redundant
info (also see attached screenshot) just to make sure that I didn't make
a mistake.
The excerpt from 001_disconnection1_local.log:
...
2022-01-12 09:29:48.099 MSK|Administrator|postgres|61de755a.a54|LOG:
pgfdw_get_result: before WaitLatchOrSocket
2022-01-12 09:29:48.099
MSK|Administrator|postgres|61de755a.a54|STATEMENT: SELECT * FROM large
WHERE a = fx2(a)
2022-01-12 09:29:48.100 MSK|Administrator|postgres|61de755a.a54|LOG:
pgfdw_get_result: after WaitLatchOrSocket
2022-01-12 09:29:48.100
MSK|Administrator|postgres|61de755a.a54|STATEMENT: SELECT * FROM large
WHERE a = fx2(a)
2022-01-12 09:29:48.100 MSK|Administrator|postgres|61de755a.a54|LOG:
pgfdw_get_result: before WaitLatchOrSocket
2022-01-12 09:29:48.100
MSK|Administrator|postgres|61de755a.a54|STATEMENT: SELECT * FROM large
WHERE a = fx2(a)
2022-01-12 09:29:48.100 MSK|Administrator|postgres|61de755a.a54|LOG:
pgfdw_get_result: after WaitLatchOrSocket
2022-01-12 09:29:48.100
MSK|Administrator|postgres|61de755a.a54|STATEMENT: SELECT * FROM large
WHERE a = fx2(a)
2022-01-12 09:29:48.100 MSK|Administrator|postgres|61de755a.a54|ERROR:
FATAL: terminating connection due to administrator command
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-01-12 09:29:48.100
MSK|Administrator|postgres|61de755a.a54|CONTEXT: remote SQL command:
FETCH 100 FROM c1
2022-01-12 09:29:48.100
MSK|Administrator|postgres|61de755a.a54|WARNING: no connection to the
server
2022-01-12 09:29:48.100
MSK|Administrator|postgres|61de755a.a54|CONTEXT: remote SQL command:
ABORT TRANSACTION
2022-01-12 09:29:48.107 MSK|Administrator|postgres|61de755a.a54|LOG:
disconnection: session time: 0:00:01.577 user=Administrator
database=postgres host=127.0.0.1 port=49752
2022-01-12 09:29:48.257 MSK|[unknown]|[unknown]|61de755c.a4c|LOG:
connection received: host=127.0.0.1 port=49754
2022-01-12 09:29:48.261 MSK|Administrator|postgres|61de755c.a4c|LOG:
connection authenticated: identity="WIN-FCPSOVMM1JC\Administrator"
method=sspi
(C:/src/postgrespro/contrib/postgres_fdw/tmp_check/t_001_disconnection1_local_data/pgdata/pg_hba.conf:2)
2022-01-12 09:29:48.261 MSK|Administrator|postgres|61de755c.a4c|LOG:
connection authorized: user=Administrator database=postgres
application_name=001_disconnection1.pl
2022-01-12 09:29:48.263 MSK|Administrator|postgres|61de755c.a4c|LOG:
statement: SELECT * FROM large WHERE a = fx2(a)
2022-01-12 09:29:48.285 MSK|Administrator|postgres|61de755c.a4c|LOG:
pgfdw_get_result: before WaitLatchOrSocket
2022-01-12 09:29:48.285
MSK|Administrator|postgres|61de755c.a4c|STATEMENT: SELECT * FROM large
WHERE a = fx2(a)
...
By the look of things, you are right and this is the localhost-only issue.
I've rechecked that the test 001_disconnection.pl (local-loopback
version) hangs on both machines while 001_disconnection1.pl performs
successfully in both directions. I'm not sure whether the Windows client
and non-Windows server or reverse combinations are of interest in light
of the above.
Best regards,
Alexander
Attachments:
[application/x-perl] 001_disconnection1.pl (1.4K, ../../[email protected]/2-001_disconnection1.pl)
download
[application/x-perl] 001_restart.pl (1.0K, ../../[email protected]/3-001_restart.pl)
download
[image/png] Screenshot_20220112_093004.png (192.0K, ../../[email protected]/4-Screenshot_20220112_093004.png)
download | view image
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 20:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-12 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
@ 2022-01-13 06:36 ` Thomas Munro <[email protected]>
2022-01-13 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-14 07:31 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 08:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
0 siblings, 3 replies; 38+ messages in thread
From: Thomas Munro @ 2022-01-13 06:36 UTC (permalink / raw)
To: Alexander Lakhin <[email protected]>; +Cc: Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
On Wed, Jan 12, 2022 at 8:00 PM Alexander Lakhin <[email protected]> wrote:
> By the look of things, you are right and this is the localhost-only issue.
But can't that be explained with timing races? You change some stuff
around and it becomes less likely that you get a FIN to arrive in a
super narrow window, which I'm guessing looks something like: recv ->
EWOULDBLOCK, [receive FIN], wait -> FD_CLOSE, wait [hangs]. Note that
it's not happening on several Windows BF animals, and the ones it is
happening on only do it only every few weeks.
Here's a draft attempt at a fix. First I tried to use recv(fd, &c, 1,
MSG_PEEK) == 0 to detect EOF, which seemed to me to be a reasonable
enough candidate, but somehow it corrupts the stream (!?), so I used
Alexander's POLLHUP idea, except I pushed it down to a more principled
place IMHO. Then I suppressed it after the initial check because then
the logic from my earlier patch takes over, so stuff like FeBeWaitSet
doesn't suffer from extra calls, only these two paths that haven't
been converted to long-lived WESes yet. Does this pass the test?
I wonder if this POLLHUP technique is reliable enough (I know that
wouldn't work on other systems[1], which is why I was trying to make
MSG_PEEK work...).
What about environment variable PG_TEST_USE_UNIX_SOCKETS=1, does it
reproduce with that set, and does the patch fix it? I'm hoping that
explains some Windows CI failures from a nearby thread[2].
[1] https://illumos.topicbox.com/groups/developer/T5576767e764aa26a-Maf8f3460c2866513b0ac51bf
[2] https://www.postgresql.org/message-id/flat/CALT9ZEG%3DC%3DJSypzt2gz6NoNtx-ew2tYHbwiOfY_xNo%2ByBY_%3D...
Attachments:
[text/x-patch] 0001-Fix-handling-of-socket-FD_CLOSE-events-on-Windows.patch (3.3K, ../../CA+hUKGKFajQiUMerS4h_=eCNKoXMYoYSkyvwMZip7WV3KUckrg@mail.gmail.com/2-0001-Fix-handling-of-socket-FD_CLOSE-events-on-Windows.patch)
download | inline diff:
From a2cf2b016afc03d384a1682cd02aab760273d6fb Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Thu, 13 Jan 2022 15:48:14 +1300
Subject: [PATCH] Fix handling of socket FD_CLOSE events on Windows.
In some situations we could hang waiting for an FD_CLOSE event that has
already been reported. Fix this by adding some state to WaitEvent to
remember that we've already been told the socket is closed. To handle
client code that creates and destroys multiple temporary WaitEventSet
objects and thus would lose that state, check for EOF explicitly the
first time through.
---
src/backend/storage/ipc/latch.c | 41 +++++++++++++++++++++++++++++++++
src/include/storage/latch.h | 1 +
2 files changed, 42 insertions(+)
diff --git a/src/backend/storage/ipc/latch.c b/src/backend/storage/ipc/latch.c
index 61c876beff..86b86e71c4 100644
--- a/src/backend/storage/ipc/latch.c
+++ b/src/backend/storage/ipc/latch.c
@@ -899,6 +899,7 @@ AddWaitEventToSet(WaitEventSet *set, uint32 events, pgsocket fd, Latch *latch,
event->user_data = user_data;
#ifdef WIN32
event->reset = false;
+ event->eof = -1; /* unknown */
#endif
if (events == WL_LATCH_SET)
@@ -1851,6 +1852,45 @@ WaitEventSetWaitBlock(WaitEventSet *set, int cur_timeout,
cur_event->reset = false;
}
+ /*
+ * Windows does not report FD_CLOSE repeatedly, and does not report
+ * FD_READ on EOF. Therefore we need to remember if we've seen
+ * FD_CLOSE, to defend against callers that wait twice in a row
+ * without a recv() that fails with WSAEWOULDBLOCK in between.
+ */
+ if (cur_event->events & WL_SOCKET_READABLE)
+ {
+ /*
+ * Check for EOF explicitly the first time through, to bridge the
+ * gap between temporary WaitEventSet objects (such as those
+ * created by WaitLatchOrSocket()).
+ */
+ if (cur_event->eof == -1)
+ {
+ WSAPOLLFD pollfd;
+ int rc;
+
+ pollfd.fd = cur_event->fd;
+ pollfd.events = POLLRDNORM;
+ pollfd.revents = 0;
+ rc = WSAPoll(&pollfd, 1, 0);
+ if (rc == 1 && pollfd.revents & POLLHUP)
+ cur_event->eof = 1; /* EOF */
+ else
+ cur_event->eof = 0; /* data or error available */
+ }
+
+ /* If we've seen EOF or FD_CLOSE, keep reporting readiness. */
+ if (cur_event->eof == 1)
+ {
+ occurred_events->pos = cur_event->pos;
+ occurred_events->user_data = cur_event->user_data;
+ occurred_events->events = WL_SOCKET_READABLE;
+ occurred_events->fd = cur_event->fd;
+ return 1;
+ }
+ }
+
/*
* Windows does not guarantee to log an FD_WRITE network event
* indicating that more data can be sent unless the previous send()
@@ -2002,6 +2042,7 @@ WaitEventSetWaitBlock(WaitEventSet *set, int cur_timeout,
{
/* EOF/error, so signal all caller-requested socket flags */
occurred_events->events |= (cur_event->events & WL_SOCKET_MASK);
+ cur_event->eof = 1;
}
if (occurred_events->events != 0)
diff --git a/src/include/storage/latch.h b/src/include/storage/latch.h
index 3aa7b33834..41a857c0c3 100644
--- a/src/include/storage/latch.h
+++ b/src/include/storage/latch.h
@@ -147,6 +147,7 @@ typedef struct WaitEvent
void *user_data; /* pointer provided in AddWaitEventToSet */
#ifdef WIN32
bool reset; /* Is reset of the event required? */
+ int8 eof; /* Has EOF been reached on a socket? */
#endif
} WaitEvent;
--
2.33.1
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 20:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-12 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-13 06:36 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
@ 2022-01-13 07:00 ` Alexander Lakhin <[email protected]>
2 siblings, 0 replies; 38+ messages in thread
From: Alexander Lakhin @ 2022-01-13 07:00 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
13.01.2022 09:36, Thomas Munro wrote:
> On Wed, Jan 12, 2022 at 8:00 PM Alexander Lakhin <[email protected]> wrote:
>> By the look of things, you are right and this is the localhost-only issue.
> But can't that be explained with timing races? You change some stuff
> around and it becomes less likely that you get a FIN to arrive in a
> super narrow window, which I'm guessing looks something like: recv ->
> EWOULDBLOCK, [receive FIN], wait -> FD_CLOSE, wait [hangs]. Note that
> it's not happening on several Windows BF animals, and the ones it is
> happening on only do it only every few weeks.
But I still see the issue when I run both test parts on a single
machine: first instance is `vcregress taptest src\test\restart` and the
second `set NO_TEMP_INSTALL=1 & vcregress taptest contrib/postgres_fdw`
(see attachment).
I'll try new tests and continue investigation later today/tomorrow. Thanks!
Best regards,
Alexander
Attachments:
[image/png] Screenshot_20220113_095249.png (63.2K, ../../[email protected]/2-Screenshot_20220113_095249.png)
download | view image
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 20:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-12 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-13 06:36 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
@ 2022-01-14 07:31 ` Thomas Munro <[email protected]>
2022-01-14 20:28 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
2 siblings, 1 reply; 38+ messages in thread
From: Thomas Munro @ 2022-01-14 07:31 UTC (permalink / raw)
To: Alexander Lakhin <[email protected]>; +Cc: Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
On Thu, Jan 13, 2022 at 7:36 PM Thomas Munro <[email protected]> wrote:
> ... First I tried to use recv(fd, &c, 1,
> MSG_PEEK) == 0 to detect EOF, which seemed to me to be a reasonable
> enough candidate, but somehow it corrupts the stream (!?),
Ahh, that'd be because recv() and friends are redirected to our
wrappers in socket.c, where we use the overlapped Winsock API (that
is, async network IO), which is documented as not supporting MSG_PEEK.
OK then.
Andres and I chatted about this stuff off list and he pointed out
something else about the wrappers in socket.c: there are more paths in
there that work with socket events, which means more ways to lose the
precious FD_CLOSE event. I don't know if any of those paths are
reachable in the relevant cases, but it does look a little bit like
the lack of graceful shutdown might have been hiding a whole class of
event tracking bug.
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 20:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-12 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-13 06:36 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 07:31 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
@ 2022-01-14 20:28 ` Andres Freund <[email protected]>
2022-01-14 20:47 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
2022-01-14 21:59 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
0 siblings, 2 replies; 38+ messages in thread
From: Andres Freund @ 2022-01-14 20:28 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Alexander Lakhin <[email protected]>; Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
Hi,
On 2022-01-14 20:31:22 +1300, Thomas Munro wrote:
> Andres and I chatted about this stuff off list and he pointed out
> something else about the wrappers in socket.c: there are more paths in
> there that work with socket events, which means more ways to lose the
> precious FD_CLOSE event.
I think it doesn't even need to touch socket.c to cause breakage. Using two
different WaitEventSets is enough.
https://docs.microsoft.com/en-us/windows/win32/api/winsock2/nf-winsock2-wsaeventselect
says:
> The FD_CLOSE network event is recorded when a close indication is received
> for the virtual circuit corresponding to the socket. In TCP terms, this
> means that the FD_CLOSE is recorded when the connection goes into the TIME
> WAIT or CLOSE WAIT states. This results from the remote end performing a
> shutdown on the send side or a closesocket. FD_CLOSE being posted after all
> data is read from a socket
So FD_CLOSE is *recorded* internally when the connection is closed. But only
posted to the visible event once all data is read. All good so far. But
combine that with:
> Issuing a WSAEventSelect for a socket cancels any previous WSAAsyncSelect or
> WSAEventSelect for the same socket and clears the internal network event
> record.
Note the bit about clearing the internal network event record. Which seems to
pretty precisely explain why we're loosing FD_CLOSEs?
And it does also explain why this is more likely after the shutdown changes:
It's more likely the network stack knows it has readable data *and* that the
connection closed. Which is recorded in the "internal network event
record". But once all the data is read, walsender.c will do another
WaitLatchOrSocket(), which does WSAEventSelect(), clearing the "internal event
record" and loosing the FD_CLOSE.
My first inclination was that we ought to wrap the socket created for windows
in pgwin32_socket() in a custom type with some additional data - including
information about already received events, an EVENT, etc. I think that might
help to remove a lot of the messy workarounds we have in socket.c etc. But: It
wouldn't do much good here, because the socket is not a socket created by
socket.c but by libpq :(.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 20:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-12 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-13 06:36 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 07:31 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 20:28 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
@ 2022-01-14 20:47 ` Andres Freund <[email protected]>
2022-01-15 00:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
1 sibling, 1 reply; 38+ messages in thread
From: Andres Freund @ 2022-01-14 20:47 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Alexander Lakhin <[email protected]>; Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers; Robert Haas <[email protected]>
Hi,
On 2022-01-14 12:28:48 -0800, Andres Freund wrote:
> But once all the data is read, walsender.c will do another
> WaitLatchOrSocket(), which does WSAEventSelect(), clearing the "internal event
> record" and loosing the FD_CLOSE.
Walreceiver only started using WES in
2016-03-29 [314cbfc5d] Add new replication mode synchronous_commit = 'remote_ap
With that came the following comment:
/*
* Ideally we would reuse a WaitEventSet object repeatedly
* here to avoid the overheads of WaitLatchOrSocket on epoll
* systems, but we can't be sure that libpq (or any other
* walreceiver implementation) has the same socket (even if
* the fd is the same number, it may have been closed and
* reopened since the last time). In future, if there is a
* function for removing sockets from WaitEventSet, then we
* could add and remove just the socket each time, potentially
* avoiding some system calls.
*/
Assert(wait_fd != PGINVALID_SOCKET);
rc = WaitLatchOrSocket(MyLatch,
WL_EXIT_ON_PM_DEATH | WL_SOCKET_READABLE |
WL_TIMEOUT | WL_LATCH_SET,
wait_fd,
NAPTIME_PER_CYCLE,
WAIT_EVENT_WAL_RECEIVER_MAIN);
I don't really see how libpq could have changed the socket underneath us, as
long as we get it the first time after the connection successfully was
established? I mean, there's a running command that we're processing the
result of? Nor do I understand what "any other walreceiver implementation"
refers to?
Thomas, I think you wrote that?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 20:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-12 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-13 06:36 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 07:31 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 20:28 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
2022-01-14 20:47 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
@ 2022-01-15 00:40 ` Thomas Munro <[email protected]>
0 siblings, 0 replies; 38+ messages in thread
From: Thomas Munro @ 2022-01-15 00:40 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Alexander Lakhin <[email protected]>; Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers; Robert Haas <[email protected]>
On Sat, Jan 15, 2022 at 9:47 AM Andres Freund <[email protected]> wrote:
> Walreceiver only started using WES in
> 2016-03-29 [314cbfc5d] Add new replication mode synchronous_commit = 'remote_ap
>
> With that came the following comment:
>
> /*
> * Ideally we would reuse a WaitEventSet object repeatedly
> * here to avoid the overheads of WaitLatchOrSocket on epoll
> * systems, but we can't be sure that libpq (or any other
> * walreceiver implementation) has the same socket (even if
> * the fd is the same number, it may have been closed and
> * reopened since the last time). In future, if there is a
> * function for removing sockets from WaitEventSet, then we
> * could add and remove just the socket each time, potentially
> * avoiding some system calls.
> */
> Assert(wait_fd != PGINVALID_SOCKET);
> rc = WaitLatchOrSocket(MyLatch,
> WL_EXIT_ON_PM_DEATH | WL_SOCKET_READABLE |
> WL_TIMEOUT | WL_LATCH_SET,
> wait_fd,
> NAPTIME_PER_CYCLE,
> WAIT_EVENT_WAL_RECEIVER_MAIN);
>
> I don't really see how libpq could have changed the socket underneath us, as
> long as we get it the first time after the connection successfully was
> established? I mean, there's a running command that we're processing the
> result of?
Erm, I didn't analyse the situation much back then, I just knew that
libpq could reconnect in early phases. I can see that once you reach
that stage you can count on socket stability though, so yeah that
should work as long as you can handle it correctly in the earlier
connection phase (probably using the other patch I posted and
Alexander tested), it should all work nicely. You'd probably want to
formalise the interface/documentation on that point.
> Nor do I understand what "any other walreceiver implementation"
> refers to?
I think I meant that it goes via function pointers to talk to
libpqwalreceiver.c, but I know now that we don't actually support
using that to switch to different code, it's just a solution to a
backend/frontend linking problem. The comment was probably just
paranoia based on the way the interface works.
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 20:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-12 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-13 06:36 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 07:31 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 20:28 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
@ 2022-01-14 21:59 ` Thomas Munro <[email protected]>
2022-01-14 22:01 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 22:44 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
2022-01-14 22:51 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
1 sibling, 3 replies; 38+ messages in thread
From: Thomas Munro @ 2022-01-14 21:59 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Alexander Lakhin <[email protected]>; Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
On Sat, Jan 15, 2022 at 9:28 AM Andres Freund <[email protected]> wrote:
> I think it doesn't even need to touch socket.c to cause breakage. Using two
> different WaitEventSets is enough.
Right. I was interested in your observation because so far we'd
*only* been considering the two-consecutive-WaitEventSets case, which
we could grok experimentally. The patch Alexander tested most
recently uses a tri-state eof flag, so (1) the WES event starts out in
"unknown" state and polls with WSAPoll() to figure out whether the
socket was already closed when it wasn't looking, and then (2) it
switches to believing that we'll definitely get an FD_CLOSE so we
don't need to make the extra call on every wait. That does seem to do
the job, but if there are *other* places that can eat FD_CLOSE event
once we've switched to believing that it will come, that might be
fatal to the second part of that idea, and we might have to assume
"unknown" all the time, which would be somewhat similar to the way we
do a dummy WSASend() every time when waiting for WRITEABLE.
(That patch is assuming that we're looking for something simple to
back-patch, in the event that we decide not to just revert the
graceful shutdown patch from back branches. Reverting might be a
better idea for now, and then we could fix all this stuff going
forward.)
> > Issuing a WSAEventSelect for a socket cancels any previous WSAAsyncSelect or
> > WSAEventSelect for the same socket and clears the internal network event
> > record.
>
> Note the bit about clearing the internal network event record. Which seems to
> pretty precisely explain why we're loosing FD_CLOSEs?
Indeed.
> And it does also explain why this is more likely after the shutdown changes:
> It's more likely the network stack knows it has readable data *and* that the
> connection closed. Which is recorded in the "internal network event
> record". But once all the data is read, walsender.c will do another
> WaitLatchOrSocket(), which does WSAEventSelect(), clearing the "internal event
> record" and loosing the FD_CLOSE.
Yeah.
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 20:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-12 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-13 06:36 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 07:31 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 20:28 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
2022-01-14 21:59 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
@ 2022-01-14 22:01 ` Thomas Munro <[email protected]>
2 siblings, 0 replies; 38+ messages in thread
From: Thomas Munro @ 2022-01-14 22:01 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Alexander Lakhin <[email protected]>; Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
On Sat, Jan 15, 2022 at 10:59 AM Thomas Munro <[email protected]> wrote:
> (That patch is assuming that we're looking for something simple to
> back-patch, in the event that we decide not to just revert the
> graceful shutdown patch from back branches. Reverting might be a
> better idea for now, and then we could fix all this stuff going
> forward.)
(Though, as mentioned already, reverting isn't really enough either,
because other OSes that Windows might be talking to use lingering
sockets... and there may still be ways for this to break...)
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 20:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-12 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-13 06:36 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 07:31 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 20:28 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
2022-01-14 21:59 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
@ 2022-01-14 22:44 ` Andres Freund <[email protected]>
2022-01-15 00:19 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-16 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2 siblings, 2 replies; 38+ messages in thread
From: Andres Freund @ 2022-01-14 22:44 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Alexander Lakhin <[email protected]>; Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
Hi,
On 2022-01-15 10:59:00 +1300, Thomas Munro wrote:
> On Sat, Jan 15, 2022 at 9:28 AM Andres Freund <[email protected]> wrote:
> > I think it doesn't even need to touch socket.c to cause breakage. Using two
> > different WaitEventSets is enough.
>
> Right. I was interested in your observation because so far we'd
> *only* been considering the two-consecutive-WaitEventSets case, which
> we could grok experimentally.
There likely are further problems in other parts, but I think socket.c is
unlikely to be involved in walreceiver case - there shouldn't be any socket.c
style socket in walreceiver itself, nor do I think we are doing a
send/recv/select backed by socket.c.
> The patch Alexander tested most recently uses a tri-state eof flag [...]
What about instead giving WalReceiverConn an internal WaitEventSet, and using
that consistently? I've attached a draft for that.
Alexander, could you test with that patch applied?
Greetings,
Andres Freund
Attachments:
[text/x-diff] 0001-WIP-use-long-lived-WaitEventSet-for-libpqwalreceiver.patch (19.7K, ../../[email protected]/2-0001-WIP-use-long-lived-WaitEventSet-for-libpqwalreceiver.patch)
download | inline diff:
From 8139d324687364164c9abbd3e2b2a54b5d5bbcad Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Fri, 14 Jan 2022 14:39:57 -0800
Subject: [PATCH] WIP: use long-lived WaitEventSet for libpqwalreceiver
connections.
---
src/include/replication/walreceiver.h | 29 +++--
.../libpqwalreceiver/libpqwalreceiver.c | 112 ++++++++++++------
src/backend/replication/logical/tablesync.c | 11 +-
src/backend/replication/logical/worker.c | 13 +-
src/backend/replication/walreceiver.c | 26 +---
5 files changed, 112 insertions(+), 79 deletions(-)
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 92f73a55b8d..07a16470d48 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -253,6 +253,17 @@ typedef void (*walrcv_check_conninfo_fn) (const char *conninfo);
*/
typedef char *(*walrcv_get_conninfo_fn) (WalReceiverConn *conn);
+/*
+ * walrcv_wait_fn
+ *
+ * Waits for socket to become readable, or a latch interrupt.
+ *
+ * timeout is passed on to WaitEventSetWait(), return value is
+ * like WaitLatchOrSocket's.
+ */
+typedef int (*walrcv_wait_fn) (WalReceiverConn *conn, long timeout,
+ uint32 wait_event_info);
+
/*
* walrcv_get_senderinfo_fn
*
@@ -317,14 +328,13 @@ typedef void (*walrcv_endstreaming_fn) (WalReceiverConn *conn,
/*
* walrcv_receive_fn
*
- * Receive a message available from the WAL stream. 'buffer' is a pointer
- * to a buffer holding the message received. Returns the length of the data,
- * 0 if no data is available yet ('wait_fd' is a socket descriptor which can
- * be waited on before a retry), and -1 if the cluster ended the COPY.
+ * Receive a message available from the WAL stream. 'buffer' is a pointer to
+ * a buffer holding the message received. Returns the length of the data, 0
+ * if no data is available yet (see walrcv_wait()), and -1 if the cluster
+ * ended the COPY.
*/
typedef int (*walrcv_receive_fn) (WalReceiverConn *conn,
- char **buffer,
- pgsocket *wait_fd);
+ char **buffer);
/*
* walrcv_send_fn
@@ -385,6 +395,7 @@ typedef struct WalReceiverFunctionsType
walrcv_connect_fn walrcv_connect;
walrcv_check_conninfo_fn walrcv_check_conninfo;
walrcv_get_conninfo_fn walrcv_get_conninfo;
+ walrcv_wait_fn walrcv_wait;
walrcv_get_senderinfo_fn walrcv_get_senderinfo;
walrcv_identify_system_fn walrcv_identify_system;
walrcv_server_version_fn walrcv_server_version;
@@ -407,6 +418,8 @@ extern PGDLLIMPORT WalReceiverFunctionsType *WalReceiverFunctions;
WalReceiverFunctions->walrcv_check_conninfo(conninfo)
#define walrcv_get_conninfo(conn) \
WalReceiverFunctions->walrcv_get_conninfo(conn)
+#define walrcv_wait(conn, timeout, wait_event_info) \
+ WalReceiverFunctions->walrcv_wait(conn, timeout, wait_event_info)
#define walrcv_get_senderinfo(conn, sender_host, sender_port) \
WalReceiverFunctions->walrcv_get_senderinfo(conn, sender_host, sender_port)
#define walrcv_identify_system(conn, primary_tli) \
@@ -419,8 +432,8 @@ extern PGDLLIMPORT WalReceiverFunctionsType *WalReceiverFunctions;
WalReceiverFunctions->walrcv_startstreaming(conn, options)
#define walrcv_endstreaming(conn, next_tli) \
WalReceiverFunctions->walrcv_endstreaming(conn, next_tli)
-#define walrcv_receive(conn, buffer, wait_fd) \
- WalReceiverFunctions->walrcv_receive(conn, buffer, wait_fd)
+#define walrcv_receive(conn, buffer) \
+ WalReceiverFunctions->walrcv_receive(conn, buffer)
#define walrcv_send(conn, buffer, nbytes) \
WalReceiverFunctions->walrcv_send(conn, buffer, nbytes)
#define walrcv_create_slot(conn, slotname, temporary, two_phase, snapshot_action, lsn) \
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 0d89db4e6a6..24f062f779c 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -46,6 +46,8 @@ struct WalReceiverConn
bool logical;
/* Buffer for currently read records */
char *recvBuf;
+
+ WaitEventSet *wes;
};
/* Prototypes for interface functions */
@@ -54,6 +56,8 @@ static WalReceiverConn *libpqrcv_connect(const char *conninfo,
char **err);
static void libpqrcv_check_conninfo(const char *conninfo);
static char *libpqrcv_get_conninfo(WalReceiverConn *conn);
+static int libpqrcv_wait(WalReceiverConn *conn, long timeout,
+ uint32 wait_event_info);
static void libpqrcv_get_senderinfo(WalReceiverConn *conn,
char **sender_host, int *sender_port);
static char *libpqrcv_identify_system(WalReceiverConn *conn,
@@ -66,8 +70,7 @@ static bool libpqrcv_startstreaming(WalReceiverConn *conn,
const WalRcvStreamOptions *options);
static void libpqrcv_endstreaming(WalReceiverConn *conn,
TimeLineID *next_tli);
-static int libpqrcv_receive(WalReceiverConn *conn, char **buffer,
- pgsocket *wait_fd);
+static int libpqrcv_receive(WalReceiverConn *conn, char **buffer);
static void libpqrcv_send(WalReceiverConn *conn, const char *buffer,
int nbytes);
static char *libpqrcv_create_slot(WalReceiverConn *conn,
@@ -87,6 +90,7 @@ static WalReceiverFunctionsType PQWalReceiverFunctions = {
libpqrcv_connect,
libpqrcv_check_conninfo,
libpqrcv_get_conninfo,
+ libpqrcv_wait,
libpqrcv_get_senderinfo,
libpqrcv_identify_system,
libpqrcv_server_version,
@@ -102,8 +106,8 @@ static WalReceiverFunctionsType PQWalReceiverFunctions = {
};
/* Prototypes for private functions */
-static PGresult *libpqrcv_PQexec(PGconn *streamConn, const char *query);
-static PGresult *libpqrcv_PQgetResult(PGconn *streamConn);
+static PGresult *libpqrcv_PQexec(WalReceiverConn *conn, const char *query);
+static PGresult *libpqrcv_PQgetResult(WalReceiverConn *conn);
static char *stringlist_to_identifierstr(PGconn *conn, List *strings);
/*
@@ -203,6 +207,11 @@ libpqrcv_connect(const char *conninfo, bool logical, const char *appname,
else
io_flag = WL_SOCKET_WRITEABLE;
+ /*
+ * NB: cannot yet use/create conn->wes, during connection
+ * establishment the socket can change.
+ */
+
rc = WaitLatchOrSocket(MyLatch,
WL_EXIT_ON_PM_DEATH | WL_LATCH_SET | io_flag,
PQsocket(conn->streamConn),
@@ -227,11 +236,20 @@ libpqrcv_connect(const char *conninfo, bool logical, const char *appname,
return NULL;
}
+ conn->wes = CreateWaitEventSet(CurrentMemoryContext, 3);
+ AddWaitEventToSet(conn->wes, WL_LATCH_SET, PGINVALID_SOCKET,
+ MyLatch, NULL);
+ /* XXX: This isn't necessarily nice to do unconditionally? */
+ AddWaitEventToSet(conn->wes, WL_EXIT_ON_PM_DEATH, PGINVALID_SOCKET,
+ NULL, NULL);
+ AddWaitEventToSet(conn->wes, WL_SOCKET_READABLE, PQsocket(conn->streamConn),
+ NULL, NULL);
+
if (logical)
{
PGresult *res;
- res = libpqrcv_PQexec(conn->streamConn,
+ res = libpqrcv_PQexec(conn,
ALWAYS_SECURE_SEARCH_PATH_SQL);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
@@ -359,7 +377,7 @@ libpqrcv_identify_system(WalReceiverConn *conn, TimeLineID *primary_tli)
* Get the system identifier and timeline ID as a DataRow message from the
* primary server.
*/
- res = libpqrcv_PQexec(conn->streamConn, "IDENTIFY_SYSTEM");
+ res = libpqrcv_PQexec(conn, "IDENTIFY_SYSTEM");
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
PQclear(res);
@@ -482,7 +500,7 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
options->proto.physical.startpointTLI);
/* Start streaming. */
- res = libpqrcv_PQexec(conn->streamConn, cmd.data);
+ res = libpqrcv_PQexec(conn, cmd.data);
pfree(cmd.data);
if (PQresultStatus(res) == PGRES_COMMAND_OK)
@@ -532,7 +550,7 @@ libpqrcv_endstreaming(WalReceiverConn *conn, TimeLineID *next_tli)
* If we had not yet received CopyDone from the backend, PGRES_COPY_OUT is
* also possible in case we aborted the copy in mid-stream.
*/
- res = libpqrcv_PQgetResult(conn->streamConn);
+ res = libpqrcv_PQgetResult(conn);
if (PQresultStatus(res) == PGRES_TUPLES_OK)
{
/*
@@ -547,7 +565,7 @@ libpqrcv_endstreaming(WalReceiverConn *conn, TimeLineID *next_tli)
PQclear(res);
/* the result set should be followed by CommandComplete */
- res = libpqrcv_PQgetResult(conn->streamConn);
+ res = libpqrcv_PQgetResult(conn);
}
else if (PQresultStatus(res) == PGRES_COPY_OUT)
{
@@ -561,7 +579,7 @@ libpqrcv_endstreaming(WalReceiverConn *conn, TimeLineID *next_tli)
pchomp(PQerrorMessage(conn->streamConn)))));
/* CommandComplete should follow */
- res = libpqrcv_PQgetResult(conn->streamConn);
+ res = libpqrcv_PQgetResult(conn);
}
if (PQresultStatus(res) != PGRES_COMMAND_OK)
@@ -572,7 +590,7 @@ libpqrcv_endstreaming(WalReceiverConn *conn, TimeLineID *next_tli)
PQclear(res);
/* Verify that there are no more results */
- res = libpqrcv_PQgetResult(conn->streamConn);
+ res = libpqrcv_PQgetResult(conn);
if (res != NULL)
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -597,7 +615,7 @@ libpqrcv_readtimelinehistoryfile(WalReceiverConn *conn,
* Request the primary to send over the history file for given timeline.
*/
snprintf(cmd, sizeof(cmd), "TIMELINE_HISTORY %u", tli);
- res = libpqrcv_PQexec(conn->streamConn, cmd);
+ res = libpqrcv_PQexec(conn, cmd);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
PQclear(res);
@@ -641,7 +659,7 @@ libpqrcv_readtimelinehistoryfile(WalReceiverConn *conn,
* May return NULL, rather than an error result, on failure.
*/
static PGresult *
-libpqrcv_PQexec(PGconn *streamConn, const char *query)
+libpqrcv_PQexec(WalReceiverConn *conn, const char *query)
{
PGresult *lastResult = NULL;
@@ -657,7 +675,7 @@ libpqrcv_PQexec(PGconn *streamConn, const char *query)
* theoretically block. In practice, since we don't send very long query
* strings, the risk seems negligible.
*/
- if (!PQsendQuery(streamConn, query))
+ if (!PQsendQuery(conn->streamConn, query))
return NULL;
for (;;)
@@ -665,7 +683,7 @@ libpqrcv_PQexec(PGconn *streamConn, const char *query)
/* Wait for, and collect, the next PGresult. */
PGresult *result;
- result = libpqrcv_PQgetResult(streamConn);
+ result = libpqrcv_PQgetResult(conn);
if (result == NULL)
break; /* query is complete, or failure */
@@ -679,24 +697,49 @@ libpqrcv_PQexec(PGconn *streamConn, const char *query)
if (PQresultStatus(lastResult) == PGRES_COPY_IN ||
PQresultStatus(lastResult) == PGRES_COPY_OUT ||
PQresultStatus(lastResult) == PGRES_COPY_BOTH ||
- PQstatus(streamConn) == CONNECTION_BAD)
+ PQstatus(conn->streamConn) == CONNECTION_BAD)
break;
}
return lastResult;
}
+/*
+ * XXX: Currently the WES is configured to only check if socket is readable,
+ * not writeable. That might need to be configurable in the future.
+ */
+static int
+libpqrcv_wait(WalReceiverConn *conn, long timeout, uint32 wait_event_info)
+{
+ int nevents;
+ WaitEvent event;
+ int ret = 0;
+
+ nevents = WaitEventSetWait(conn->wes, timeout, &event, 1, wait_event_info);
+
+ if (nevents == 0)
+ ret |= WL_TIMEOUT;
+ else
+ {
+ ret |= event.events & (WL_LATCH_SET |
+ WL_POSTMASTER_DEATH |
+ WL_SOCKET_MASK);
+ }
+
+ return ret;
+}
+
/*
* Perform the equivalent of PQgetResult(), but watch for interrupts.
*/
static PGresult *
-libpqrcv_PQgetResult(PGconn *streamConn)
+libpqrcv_PQgetResult(WalReceiverConn *conn)
{
/*
* Collect data until PQgetResult is ready to get the result without
* blocking.
*/
- while (PQisBusy(streamConn))
+ while (PQisBusy(conn->streamConn))
{
int rc;
@@ -705,12 +748,7 @@ libpqrcv_PQgetResult(PGconn *streamConn)
* since we'll get interrupted by signals and can handle any
* interrupts here.
*/
- rc = WaitLatchOrSocket(MyLatch,
- WL_EXIT_ON_PM_DEATH | WL_SOCKET_READABLE |
- WL_LATCH_SET,
- PQsocket(streamConn),
- 0,
- WAIT_EVENT_LIBPQWALRECEIVER_RECEIVE);
+ rc = libpqrcv_wait(conn, -1, WAIT_EVENT_LIBPQWALRECEIVER_RECEIVE);
/* Interrupted? */
if (rc & WL_LATCH_SET)
@@ -720,7 +758,7 @@ libpqrcv_PQgetResult(PGconn *streamConn)
}
/* Consume whatever data is available from the socket */
- if (PQconsumeInput(streamConn) == 0)
+ if (PQconsumeInput(conn->streamConn) == 0)
{
/* trouble; return NULL */
return NULL;
@@ -728,7 +766,7 @@ libpqrcv_PQgetResult(PGconn *streamConn)
}
/* Now we can collect and return the next PGresult */
- return PQgetResult(streamConn);
+ return PQgetResult(conn->streamConn);
}
/*
@@ -737,6 +775,12 @@ libpqrcv_PQgetResult(PGconn *streamConn)
static void
libpqrcv_disconnect(WalReceiverConn *conn)
{
+ if (conn->wes != NULL)
+ {
+ FreeWaitEventSet(conn->wes);
+ conn->wes = NULL;
+ }
+
PQfinish(conn->streamConn);
if (conn->recvBuf != NULL)
PQfreemem(conn->recvBuf);
@@ -752,16 +796,15 @@ libpqrcv_disconnect(WalReceiverConn *conn)
* point to a buffer holding the received message. The buffer is only valid
* until the next libpqrcv_* call.
*
- * If no data was available immediately, returns 0, and *wait_fd is set to a
- * socket descriptor which can be waited on before trying again.
+ * If no data was available immediately, returns 0. In that case walrcv_wait()
+ * can be used to wait for socket readiness or interrupts before trying again.
*
* -1 if the server ended the COPY.
*
* ereports on error.
*/
static int
-libpqrcv_receive(WalReceiverConn *conn, char **buffer,
- pgsocket *wait_fd)
+libpqrcv_receive(WalReceiverConn *conn, char **buffer)
{
int rawlen;
@@ -785,7 +828,6 @@ libpqrcv_receive(WalReceiverConn *conn, char **buffer,
if (rawlen == 0)
{
/* Tell caller to try again when our socket is ready. */
- *wait_fd = PQsocket(conn->streamConn);
return 0;
}
}
@@ -793,13 +835,13 @@ libpqrcv_receive(WalReceiverConn *conn, char **buffer,
{
PGresult *res;
- res = libpqrcv_PQgetResult(conn->streamConn);
+ res = libpqrcv_PQgetResult(conn);
if (PQresultStatus(res) == PGRES_COMMAND_OK)
{
PQclear(res);
/* Verify that there are no more results. */
- res = libpqrcv_PQgetResult(conn->streamConn);
+ res = libpqrcv_PQgetResult(conn);
if (res != NULL)
{
PQclear(res);
@@ -941,7 +983,7 @@ libpqrcv_create_slot(WalReceiverConn *conn, const char *slotname,
appendStringInfoString(&cmd, " PHYSICAL RESERVE_WAL");
}
- res = libpqrcv_PQexec(conn->streamConn, cmd.data);
+ res = libpqrcv_PQexec(conn, cmd.data);
pfree(cmd.data);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
@@ -1068,7 +1110,7 @@ libpqrcv_exec(WalReceiverConn *conn, const char *query,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("the query interface requires a database connection")));
- pgres = libpqrcv_PQexec(conn->streamConn, query);
+ pgres = libpqrcv_PQexec(conn, query);
switch (PQresultStatus(pgres))
{
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index e596b69d466..9eb78d1faab 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -637,14 +637,13 @@ copy_read_data(void *outbuf, int minread, int maxread)
while (maxread > 0 && bytesread < minread)
{
- pgsocket fd = PGINVALID_SOCKET;
int len;
char *buf = NULL;
for (;;)
{
/* Try read the data. */
- len = walrcv_receive(LogRepWorkerWalRcvConn, &buf, &fd);
+ len = walrcv_receive(LogRepWorkerWalRcvConn, &buf);
CHECK_FOR_INTERRUPTS();
@@ -676,11 +675,8 @@ copy_read_data(void *outbuf, int minread, int maxread)
/*
* Wait for more data or latch.
*/
- (void) WaitLatchOrSocket(MyLatch,
- WL_SOCKET_READABLE | WL_LATCH_SET |
- WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
- fd, 1000L, WAIT_EVENT_LOGICAL_SYNC_DATA);
-
+ (void) walrcv_wait(LogRepWorkerWalRcvConn, 1000L,
+ WAIT_EVENT_LOGICAL_SYNC_DATA);
ResetLatch(MyLatch);
}
@@ -967,6 +963,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos)
* application_name, so that it is different from the main apply worker,
* so that synchronous replication can distinguish them.
*/
+ Assert(LogRepWorkerWalRcvConn == NULL);
LogRepWorkerWalRcvConn =
walrcv_connect(MySubscription->conninfo, true, slotname, &err);
if (LogRepWorkerWalRcvConn == NULL)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index c9af775bc18..95d4d0c7d1f 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2597,7 +2597,6 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
/* This outer loop iterates once per wait. */
for (;;)
{
- pgsocket fd = PGINVALID_SOCKET;
int rc;
int len;
char *buf = NULL;
@@ -2608,7 +2607,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
MemoryContextSwitchTo(ApplyMessageContext);
- len = walrcv_receive(LogRepWorkerWalRcvConn, &buf, &fd);
+ len = walrcv_receive(LogRepWorkerWalRcvConn, &buf);
if (len != 0)
{
@@ -2688,7 +2687,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
MemoryContextReset(ApplyMessageContext);
}
- len = walrcv_receive(LogRepWorkerWalRcvConn, &buf, &fd);
+ len = walrcv_receive(LogRepWorkerWalRcvConn, &buf);
}
}
@@ -2729,11 +2728,8 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
else
wait_time = NAPTIME_PER_CYCLE;
- rc = WaitLatchOrSocket(MyLatch,
- WL_SOCKET_READABLE | WL_LATCH_SET |
- WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
- fd, wait_time,
- WAIT_EVENT_LOGICAL_APPLY_MAIN);
+ rc = walrcv_wait(LogRepWorkerWalRcvConn, wait_time,
+ WAIT_EVENT_LOGICAL_APPLY_MAIN);
if (rc & WL_LATCH_SET)
{
@@ -3541,6 +3537,7 @@ ApplyWorkerMain(Datum main_arg)
origin_startpos = replorigin_session_get_progress(false);
CommitTransactionCommand();
+ Assert(LogRepWorkerWalRcvConn == NULL);
LogRepWorkerWalRcvConn = walrcv_connect(MySubscription->conninfo, true,
MySubscription->name, &err);
if (LogRepWorkerWalRcvConn == NULL)
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index b39fce8c23c..d70fc462bea 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -423,7 +423,6 @@ WalReceiverMain(void)
char *buf;
int len;
bool endofwal = false;
- pgsocket wait_fd = PGINVALID_SOCKET;
int rc;
/*
@@ -446,7 +445,7 @@ WalReceiverMain(void)
}
/* See if we can read data immediately */
- len = walrcv_receive(wrconn, &buf, &wait_fd);
+ len = walrcv_receive(wrconn, &buf);
if (len != 0)
{
/*
@@ -478,7 +477,7 @@ WalReceiverMain(void)
endofwal = true;
break;
}
- len = walrcv_receive(wrconn, &buf, &wait_fd);
+ len = walrcv_receive(wrconn, &buf);
}
/* Let the primary know that we received some data. */
@@ -496,24 +495,9 @@ WalReceiverMain(void)
if (endofwal)
break;
- /*
- * Ideally we would reuse a WaitEventSet object repeatedly
- * here to avoid the overheads of WaitLatchOrSocket on epoll
- * systems, but we can't be sure that libpq (or any other
- * walreceiver implementation) has the same socket (even if
- * the fd is the same number, it may have been closed and
- * reopened since the last time). In future, if there is a
- * function for removing sockets from WaitEventSet, then we
- * could add and remove just the socket each time, potentially
- * avoiding some system calls.
- */
- Assert(wait_fd != PGINVALID_SOCKET);
- rc = WaitLatchOrSocket(MyLatch,
- WL_EXIT_ON_PM_DEATH | WL_SOCKET_READABLE |
- WL_TIMEOUT | WL_LATCH_SET,
- wait_fd,
- NAPTIME_PER_CYCLE,
- WAIT_EVENT_WAL_RECEIVER_MAIN);
+ rc = walrcv_wait(wrconn, NAPTIME_PER_CYCLE,
+ WAIT_EVENT_WAL_RECEIVER_MAIN);
+
if (rc & WL_LATCH_SET)
{
ResetLatch(MyLatch);
--
2.34.0
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 20:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-12 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-13 06:36 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 07:31 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 20:28 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
2022-01-14 21:59 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 22:44 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
@ 2022-01-15 00:19 ` Thomas Munro <[email protected]>
2022-01-15 00:51 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
1 sibling, 1 reply; 38+ messages in thread
From: Thomas Munro @ 2022-01-15 00:19 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Alexander Lakhin <[email protected]>; Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
On Sat, Jan 15, 2022 at 11:44 AM Andres Freund <[email protected]> wrote:
> > The patch Alexander tested most recently uses a tri-state eof flag [...]
>
> What about instead giving WalReceiverConn an internal WaitEventSet, and using
> that consistently? I've attached a draft for that.
>
> Alexander, could you test with that patch applied?
Isn't your patch nearly identical to one that I already posted, that
Alexander tested and reported success with here?
https://www.postgresql.org/message-id/5d507424-13ce-d19f-2f5d-ab4c6a987316%40gmail.com
I can believe that fixes walreceiver (if we're sure that there isn't a
libpq-changes-the-socket problem), but AFAICS the same problem exists
for postgres_fdw and async append. That's why I moved to trying to
fix the multiple-WES thing (though of course I agree we should be
using long lived WESes wherever possible, I just didn't think that
seemed back-patchable, so it's more of a feature patch for the
future).
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 20:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-12 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-13 06:36 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 07:31 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 20:28 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
2022-01-14 21:59 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 22:44 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
2022-01-15 00:19 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
@ 2022-01-15 00:51 ` Andres Freund <[email protected]>
0 siblings, 0 replies; 38+ messages in thread
From: Andres Freund @ 2022-01-15 00:51 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Alexander Lakhin <[email protected]>; Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
Hi,
On 2022-01-15 13:19:42 +1300, Thomas Munro wrote:
> On Sat, Jan 15, 2022 at 11:44 AM Andres Freund <[email protected]> wrote:
> > > The patch Alexander tested most recently uses a tri-state eof flag [...]
> >
> > What about instead giving WalReceiverConn an internal WaitEventSet, and using
> > that consistently? I've attached a draft for that.
> >
> > Alexander, could you test with that patch applied?
>
> Isn't your patch nearly identical to one that I already posted, that
> Alexander tested and reported success with here?
Sorry, somehow I missed that across the many patches in the thread... And yes,
it does look remarkably similar.
> https://www.postgresql.org/message-id/5d507424-13ce-d19f-2f5d-ab4c6a987316%40gmail.com
>
> I can believe that fixes walreceiver
One thing that still bothers me around this is that we didn't detect the
problem of the dead walreceiver connection, even after missing the
FD_CLOSE. There's plenty other ways that a connection can get stalled for
prolonged was, that we'd not get notified about either. That's why there's
wal_receiver_timeout, after all.
But from what I can see wal_receiver_timeout doesn't work even halfway
reliably, because of all the calls down to libpqrcv_PQgetResult, where we just
block indefinitely?
This actually seems like a significant issue to me, and not just on windows.
> (if we're sure that there isn't a libpq-changes-the-socket problem)
I just don't see what that problem could be, once connection is
established. The only way a the socket fd could change is a reconnect, which
doesn't happen automatically.
I actually was searching the archives for threads on it, but I didn't find
much besides the references around [1]. And I didn't see a concrete risk
explained there?
> but AFAICS the same problem exists for postgres_fdw and async append.
Perhaps - but I suspect it'll matter far less with them than with walreceiver.
> That's why I moved to trying to
> fix the multiple-WES thing (though of course I agree we should be
> using long lived WESes wherever possible
That approach seems like a very very leaky bandaid, with a decent potential
for unintended consequences. Perhaps there's nothing better that we can do,
but I'd rather try to fix the problem closer to the root...
> I just didn't think that seemed back-patchable, so it's more of a feature
> patch for the future).
Hm, it doesn't seem crazy invasive to me. But I guess we might be looking at a
revert of the shutdown changes for now anyway? In that case we should be
fixing this anyway, but we might be able to afford doing it in master only?
Greetings,
Andres Freund
[1] https://www.postgresql.org/message-id/CA%2BhUKG%2BzCNJZBXcURPdQvdY-tjyD0y7Li2wZEC6XChyUej1S5w%40mail...
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 20:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-12 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-13 06:36 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 07:31 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 20:28 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
2022-01-14 21:59 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 22:44 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
@ 2022-01-16 11:00 ` Alexander Lakhin <[email protected]>
1 sibling, 0 replies; 38+ messages in thread
From: Alexander Lakhin @ 2022-01-16 11:00 UTC (permalink / raw)
To: Andres Freund <[email protected]>; Thomas Munro <[email protected]>; +Cc: Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
Hello Andres,
15.01.2022 01:44, Andres Freund wrote:
> What about instead giving WalReceiverConn an internal WaitEventSet, and using
> that consistently? I've attached a draft for that.
>
> Alexander, could you test with that patch applied?
Unfortunately, this patch doesn't fix the issue. The test
commit_ts/002_standby still fails (on iterations 3, 1, 8 for me).
With the debugging logging added:
...
elog(LOG, "libpqrcv_wait() before WaitEventSetWait(%p)", conn->wes);
nevents = WaitEventSetWait(conn->wes, timeout, &event, 1,
wait_event_info);
elog(LOG, "libpqrcv_wait() after WaitEventSetWait");
...
elog(LOG, "WaitEventSetWaitBlock before WaitForMultipleObjects(%p)",
set->handles);
rc = WaitForMultipleObjects(set->nevents + 1, set->handles, FALSE,
cur_timeout);
elog(LOG, "WaitEventSetWaitBlock aftet WaitForMultipleObjects");
...
and so on.
I see in 002_standby_standby.log
...
2022-01-16 13:31:36.244 MSK [1336] LOG: libpqrcv_receive PQgetCopyData
returned: 145
2022-01-16 13:31:36.244 MSK [1336] LOG: libpqrcv_receive PQgetCopyData
returned: 0
2022-01-16 13:31:36.244 MSK [1336] LOG: libpqrcv_wait() before
WaitEventSetWait(000000000063ABA8)
2022-01-16 13:31:36.244 MSK [1336] LOG: WaitEventSetWait before
WaitEventSetWaitBlock(000000000063ABA8)
2022-01-16 13:31:36.244 MSK [1336] LOG: WaitEventSetWaitBlock before
WaitForMultipleObjects(000000000063AC30)
2022-01-16 13:31:36.244 MSK [2820] LOG: WaitEventSetWaitBlock aftet
WaitForMultipleObjects
2022-01-16 13:31:36.244 MSK [2820] LOG: WaitEventSetWait after
WaitEventSetWaitBlock
2022-01-16 13:31:36.244 MSK [1336] LOG: WaitEventSetWaitBlock aftet
WaitForMultipleObjects
2022-01-16 13:31:36.244 MSK [1336] LOG: WaitEventSetWait after
WaitEventSetWaitBlock
2022-01-16 13:31:36.244 MSK [1336] LOG: libpqrcv_wait() after
WaitEventSetWait
2022-01-16 13:31:36.244 MSK [1336] LOG: libpqrcv_receive PQgetCopyData
returned: 0
2022-01-16 13:31:36.244 MSK [1336] LOG: libpqrcv_wait() before
WaitEventSetWait(000000000063ABA8)
2022-01-16 13:31:36.244 MSK [1336] LOG: WaitEventSetWait before
WaitEventSetWaitBlock(000000000063ABA8)
2022-01-16 13:31:36.244 MSK [1336] LOG: WaitEventSetWaitBlock before
WaitForMultipleObjects(000000000063AC30)
2022-01-16 13:31:36.244 MSK [2820] LOG: WaitEventSetWait before
WaitEventSetWaitBlock(0000000000649FB8)
2022-01-16 13:31:36.244 MSK [2820] LOG: WaitEventSetWaitBlock before
WaitForMultipleObjects(000000000064A020)
2022-01-16 13:31:36.247 MSK [1336] LOG: WaitEventSetWaitBlock aftet
WaitForMultipleObjects
2022-01-16 13:31:36.247 MSK [1336] LOG: WaitEventSetWait after
WaitEventSetWaitBlock
2022-01-16 13:31:36.247 MSK [1336] LOG: libpqrcv_wait() after
WaitEventSetWait
2022-01-16 13:31:36.247 MSK [1336] LOG: libpqrcv_receive PQgetCopyData
returned: -1
2022-01-16 13:31:36.247 MSK [1336] LOG: libpqrcv_receive before
libpqrcv_PQgetResult(1)
2022-01-16 13:31:36.247 MSK [1336] LOG: libpqrcv_receive
libpqrcv_PQgetResult(1) returned 0000000000692400
2022-01-16 13:31:36.247 MSK [1336] LOG: libpqrcv_receive before
libpqrcv_PQgetResult(2)
2022-01-16 13:31:36.247 MSK [1336] LOG: libpqrcv_wait() before
WaitEventSetWait(000000000063ABA8)
2022-01-16 13:31:36.247 MSK [1336] LOG: WaitEventSetWait before
WaitEventSetWaitBlock(000000000063ABA8)
2022-01-16 13:31:36.247 MSK [1336] LOG: WaitEventSetWaitBlock before
WaitForMultipleObjects(000000000063AC30)
2022-01-16 13:31:36.368 MSK [984] LOG: WaitEventSetWaitBlock aftet
WaitForMultipleObjects
2022-01-16 13:31:36.368 MSK [984] LOG: WaitEventSetWait after
WaitEventSetWaitBlock
...
After that, the process 1336 hangs till shutdown.
Best regards,
Alexander
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 20:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-12 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-13 06:36 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 07:31 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 20:28 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
2022-01-14 21:59 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
@ 2022-01-14 22:51 ` Tom Lane <[email protected]>
2022-01-24 20:02 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
2 siblings, 1 reply; 38+ messages in thread
From: Tom Lane @ 2022-01-14 22:51 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Andres Freund <[email protected]>; Alexander Lakhin <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
Thomas Munro <[email protected]> writes:
> (That patch is assuming that we're looking for something simple to
> back-patch, in the event that we decide not to just revert the
> graceful shutdown patch from back branches. Reverting might be a
> better idea for now, and then we could fix all this stuff going
> forward.)
FWIW, I'm just fine with reverting, particularly in the back branches.
It seems clear that this dank corner of Windows contains even more
creepy-crawlies than we thought.
regards, tom lane
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 20:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-12 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-13 06:36 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 07:31 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 20:28 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
2022-01-14 21:59 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 22:51 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
@ 2022-01-24 20:02 ` Andres Freund <[email protected]>
2022-01-24 20:35 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Andres Freund @ 2022-01-24 20:02 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Thomas Munro <[email protected]>; Alexander Lakhin <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
Hi,
On 2022-01-14 17:51:52 -0500, Tom Lane wrote:
> FWIW, I'm just fine with reverting, particularly in the back branches.
> It seems clear that this dank corner of Windows contains even more
> creepy-crawlies than we thought.
Seems we should revert now-ish? There's a minor release coming up and I think
it'd be bad to ship these changes to users.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 20:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-12 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-13 06:36 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 07:31 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 20:28 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
2022-01-14 21:59 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 22:51 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-24 20:02 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
@ 2022-01-24 20:35 ` Tom Lane <[email protected]>
2022-01-24 21:28 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
0 siblings, 1 reply; 38+ messages in thread
From: Tom Lane @ 2022-01-24 20:35 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Thomas Munro <[email protected]>; Alexander Lakhin <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
Andres Freund <[email protected]> writes:
> On 2022-01-14 17:51:52 -0500, Tom Lane wrote:
>> FWIW, I'm just fine with reverting, particularly in the back branches.
>> It seems clear that this dank corner of Windows contains even more
>> creepy-crawlies than we thought.
> Seems we should revert now-ish? There's a minor release coming up and I think
> it'd be bad to ship these changes to users.
Sure. Do we want to revert in HEAD too?
regards, tom lane
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 20:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-12 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-13 06:36 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 07:31 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 20:28 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
2022-01-14 21:59 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-14 22:51 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-24 20:02 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Andres Freund <[email protected]>
2022-01-24 20:35 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
@ 2022-01-24 21:28 ` Andres Freund <[email protected]>
0 siblings, 0 replies; 38+ messages in thread
From: Andres Freund @ 2022-01-24 21:28 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Thomas Munro <[email protected]>; +Cc: Alexander Lakhin <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
Hi,
On 2022-01-24 15:35:25 -0500, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > On 2022-01-14 17:51:52 -0500, Tom Lane wrote:
> >> FWIW, I'm just fine with reverting, particularly in the back branches.
> >> It seems clear that this dank corner of Windows contains even more
> >> creepy-crawlies than we thought.
>
> > Seems we should revert now-ish? There's a minor release coming up and I think
> > it'd be bad to ship these changes to users.
>
> Sure. Do we want to revert in HEAD too?
Not sure. I'm also OK with trying to go with Thomas' patch to walreceiver and
try a bit longer to get all this working. Thomas?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 38+ messages in thread
* Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 11:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-09 19:06 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 02:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 09:40 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-10 17:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 15:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
2022-01-11 20:16 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-12 07:00 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Alexander Lakhin <[email protected]>
2022-01-13 06:36 ` Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Thomas Munro <[email protected]>
@ 2022-01-14 08:00 ` Alexander Lakhin <[email protected]>
2 siblings, 0 replies; 38+ messages in thread
From: Alexander Lakhin @ 2022-01-14 08:00 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers
13.01.2022 09:36, Thomas Munro wrote:
> Here's a draft attempt at a fix. First I tried to use recv(fd, &c, 1,
> MSG_PEEK) == 0 to detect EOF, which seemed to me to be a reasonable
> enough candidate, but somehow it corrupts the stream (!?), so I used
> Alexander's POLLHUP idea, except I pushed it down to a more principled
> place IMHO. Then I suppressed it after the initial check because then
> the logic from my earlier patch takes over, so stuff like FeBeWaitSet
> doesn't suffer from extra calls, only these two paths that haven't
> been converted to long-lived WESes yet. Does this pass the test?
Yes, this fix eliminates the flakiness for me. The test commit_ts (with
002_standby and 003_standby_2) passed 2x200 iterations.
It also makes my test postgres_fdw/001_disconnection pass reliably.
> I wonder if this POLLHUP technique is reliable enough (I know that
> wouldn't work on other systems[1], which is why I was trying to make
> MSG_PEEK work...).
>
> What about environment variable PG_TEST_USE_UNIX_SOCKETS=1, does it
> reproduce with that set, and does the patch fix it? I'm hoping that
> explains some Windows CI failures from a nearby thread[2].
With PG_TEST_USE_UNIX_SOCKETS=1 the test commit_ts/002_standby fails on
the unpatched HEAD:
t/002_standby.pl .... 1/4 # poll_query_until timed out executing this query:
# SELECT '0/303C628'::pg_lsn <= pg_last_wal_replay_lsn()
# expecting this output:
# t
# last actual query output:
# f
# with stderr:
# Looks like your test exited with 25 just after 1.
t/002_standby.pl .... Dubious, test returned 25 (wstat 6400, 0x1900)
002_standby_primary.log contains:
2022-01-13 18:57:32.925 PST [1448] LOG: starting PostgreSQL 15devel,
compiled by Visual C++ build 1928, 64-bit
2022-01-13 18:57:32.926 PST [1448] LOG: listening on Unix socket
"C:/Users/1/AppData/Local/Temp/yOKQPH1FoO/.s.PGSQL.62733"
The same with my postgres_fdw test:
# 03:41:12.533246 result: 0
# 0|0
# 03:41:12.534758 executing query...
# 03:41:14.267594 result: 3
#
# psql:<stdin>:1: WARNING: no connection to the server
# psql:<stdin>:1: ERROR: FATAL: terminating connection due to
administrator command
# server closed the connection unexpectedly
# This probably means the server terminated abnormally
# before or while processing the request.
# CONTEXT: remote SQL command: FETCH 100 FROM c1
# 03:41:14.270449 executing query...
# 03:41:14.334437 result: 3
#
# psql:<stdin>:1: ERROR: could not connect to server "fpg"
# DETAIL: connection to server on socket
"C:/Users/1/AppData/Local/Temp/hJWD9mzPHM/.s.PGSQL.57414" failed:
Connection refused (0x0000274D/10061)
# Is the server running locally and accepting connections on that
socket?
# 03:41:14.336918 executing query...
# 03:41:14.422381 result: 3
#
# psql:<stdin>:1: ERROR: could not connect to server "fpg"
# DETAIL: connection to server on socket
"C:/Users/1/AppData/Local/Temp/hJWD9mzPHM/.s.PGSQL.57414" failed:
Connection refused (0x0000274D/10061)
# Is the server running locally and accepting connections on that
socket?
# 03:41:14.425628 executing query...
...hang...
With the patch these tests pass successfully.
I can also confirm that on Windows 10 20H2 (previous tests were
performed on Windows Server 2012) the unpatched HEAD +
PG_TEST_USE_UNIX_SOCKETS=1 hangs on src\test\recovery\001_stream_rep (on
iterations 1, 1, 4 for me).
(v9-0001-Add-option-for-amcheck-and-pg_amcheck-to-check-un.patch [1] not
required to see that.)
001_stream_rep_primary.log contains:
...
2022-01-13 19:46:48.287 PST [1364] LOG: listening on Unix socket
"C:/Users/1/AppData/Local/Temp/EWzapwiXjV/.s.PGSQL.58248"
2022-01-13 19:46:48.317 PST [6736] LOG: database system was shut down
at 2022-01-13 19:46:37 PST
2022-01-13 19:46:48.331 PST [1364] LOG: database system is ready to
accept connections
2022-01-13 19:46:49.536 PST [1088] 001_stream_rep.pl LOG: statement:
CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a
2022-01-13 19:46:49.646 PST [3028] 001_stream_rep.pl LOG: statement:
SELECT pg_current_wal_insert_lsn()
2022-01-13 19:46:49.745 PST [3360] 001_stream_rep.pl LOG: statement:
SELECT '0/3023268' <= replay_lsn AND state = 'streaming' FROM
pg_catalog.pg_stat_replication WHERE application_name = 'standby_1';
...
2022-01-13 19:50:39.755 PST [4924] 001_stream_rep.pl LOG: statement:
SELECT '0/3023268' <= replay_lsn AND state = 'streaming' FROM
pg_catalog.pg_stat_replication WHERE application_name = 'standby_1';
2022-01-13 19:50:39.928 PST [5924] 001_stream_rep.pl LOG: statement:
SELECT '0/3023268' <= replay_lsn AND state = 'streaming' FROM
pg_catalog.pg_stat_replication WHERE application_name = 'standby_1';
Without PG_TEST_USE_UNIX_SOCKETS=1 and without the fix the
001_stream_rep hangs too (but on iterations 3, 8, 2). So it seems that
using unix sockets increases the fail rate.
With the fix 100 iterations with PG_TEST_USE_UNIX_SOCKETS=1 and 40 (and
still counting) iterations without PG_TEST_USE_UNIX_SOCKETS pass.
So the fix looks as absolutely working to me with the tests that we have
for now.
[1]
https://www.postgresql.org/message-id/CALT9ZEHx2%2B9rqAeAANkUXJCsTueQqdx2Tt6ypaig9tozJkWvkQ%40mail.g...
^ permalink raw reply [nested|flat] 38+ messages in thread
* [PATCH 1/6] Add REPACK command
@ 2026-01-22 08:23 Antonin Houska <[email protected]>
0 siblings, 0 replies; 38+ messages in thread
From: Antonin Houska @ 2026-01-22 08:23 UTC (permalink / raw)
REPACK absorbs the functionality of VACUUM FULL and CLUSTER in a single
command. Because this functionality is completely different from
regular VACUUM, having it separate from VACUUM makes it easier for users
to understand; as for CLUSTER, the term is heavily overloaded in the
IT world and even in Postgres itself, so it's good that we can avoid it.
This also adds pg_repackdb, a new utility that can invoke the new
commands. This is heavily based on vacuumdb.
Author: Antonin Houska <[email protected]>
Co-authored-by: =C3=81lvaro Herrera <[email protected]>
Reviewed-by: Mihail Nikalayeu <[email protected]>
Reviewed-by: Robert Treat <[email protected]>
Reviewed-by: Euler Taveira <[email protected]>
Reviewed-by: Matheus Alcantara <[email protected]>
Reviewed-by: Junwang Zhao <[email protected]>
Reviewed-by: jian he <[email protected]>
Discussion: https://postgr.es/m/82651.1720540558@antos
Discussion: https://postgr.es/m/[email protected]
---
doc/src/sgml/monitoring.sgml | 223 +++++-
doc/src/sgml/ref/allfiles.sgml | 2 +
doc/src/sgml/ref/cluster.sgml | 97 +--
doc/src/sgml/ref/clusterdb.sgml | 5 +
doc/src/sgml/ref/pg_repackdb.sgml | 488 +++++++++++++
doc/src/sgml/ref/repack.sgml | 328 +++++++++
doc/src/sgml/ref/vacuum.sgml | 33 +-
doc/src/sgml/reference.sgml | 2 +
src/backend/access/heap/heapam_handler.c | 32 +-
src/backend/catalog/index.c | 2 +-
src/backend/catalog/system_views.sql | 29 +-
src/backend/commands/cluster.c | 842 +++++++++++++++--------
src/backend/commands/vacuum.c | 6 +-
src/backend/parser/gram.y | 86 ++-
src/backend/tcop/utility.c | 23 +-
src/backend/utils/adt/pgstatfuncs.c | 4 +-
src/bin/psql/tab-complete.in.c | 42 +-
src/bin/scripts/Makefile | 3 +
src/bin/scripts/meson.build | 2 +
src/bin/scripts/pg_repackdb.c | 240 +++++++
src/bin/scripts/t/103_repackdb.pl | 47 ++
src/bin/scripts/vacuuming.c | 102 ++-
src/bin/scripts/vacuuming.h | 3 +
src/include/commands/cluster.h | 8 +-
src/include/commands/progress.h | 50 +-
src/include/nodes/parsenodes.h | 35 +-
src/include/parser/kwlist.h | 1 +
src/include/tcop/cmdtaglist.h | 1 +
src/include/utils/backend_progress.h | 2 +-
src/test/regress/expected/cluster.out | 134 +++-
src/test/regress/expected/rules.out | 72 +-
src/test/regress/sql/cluster.sql | 70 +-
src/tools/pgindent/typedefs.list | 2 +
33 files changed, 2480 insertions(+), 536 deletions(-)
create mode 100644 doc/src/sgml/ref/pg_repackdb.sgml
create mode 100644 doc/src/sgml/ref/repack.sgml
create mode 100644 src/bin/scripts/pg_repackdb.c
create mode 100644 src/bin/scripts/t/103_repackdb.pl
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 817fd9f4ca7..b07fe3294cd 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -405,6 +405,14 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss =
11:34 0:00 postgres: ser
</entry>
</row>
=20
+ <row>
+ <entry><structname>pg_stat_progress_repack</structname><indexterm><p=
rimary>pg_stat_progress_repack</primary></indexterm></entry>
+ <entry>One row for each backend running
+ <command>REPACK</command>, showing current progress. See
+ <xref linkend=3D"repack-progress-reporting"/>.
+ </entry>
+ </row>
+
<row>
<entry><structname>pg_stat_progress_basebackup</structname><indexter=
m><primary>pg_stat_progress_basebackup</primary></indexterm></entry>
<entry>One row for each WAL sender process streaming a base backup,
@@ -5609,7 +5617,8 @@ FROM pg_stat_get_backend_idset() AS backendid;
certain commands during command execution. Currently, the only commands
which support progress reporting are <command>ANALYZE</command>,
<command>CLUSTER</command>,
- <command>CREATE INDEX</command>, <command>VACUUM</command>,
+ <command>CREATE INDEX</command>, <command>REPACK</command>,
+ <command>VACUUM</command>,
<command>COPY</command>,
and <xref linkend=3D"protocol-replication-base-backup"/> (i.e., replica=
tion
command that <xref linkend=3D"app-pgbasebackup"/> issues to take
@@ -6093,6 +6102,218 @@ FROM pg_stat_get_backend_idset() AS backendid;
</table>
</sect2>
=20
+ <sect2 id=3D"repack-progress-reporting">
+ <title>REPACK Progress Reporting</title>
+
+ <indexterm>
+ <primary>pg_stat_progress_repack</primary>
+ </indexterm>
+
+ <para>
+ Whenever <command>REPACK</command> is running,
+ the <structname>pg_stat_progress_repack</structname> view will contain a
+ row for each backend that is currently running the command. The tables
+ below describe the information that will be reported and provide
+ information about how to interpret it.
+ </para>
+
+ <table id=3D"pg-stat-progress-repack-view" xreflabel=3D"pg_stat_progress=
_repack">
+ <title><structname>pg_stat_progress_repack</structname> View</title>
+ <tgroup cols=3D"1">
+ <thead>
+ <row>
+ <entry role=3D"catalog_table_entry"><para role=3D"column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role=3D"catalog_table_entry"><para role=3D"column_definition">
+ <structfield>pid</structfield> <type>integer</type>
+ </para>
+ <para>
+ Process ID of backend.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role=3D"catalog_table_entry"><para role=3D"column_definition">
+ <structfield>datid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role=3D"catalog_table_entry"><para role=3D"column_definition">
+ <structfield>datname</structfield> <type>name</type>
+ </para>
+ <para>
+ Name of the database to which this backend is connected.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role=3D"catalog_table_entry"><para role=3D"column_definition">
+ <structfield>relid</structfield> <type>oid</type>
+ </para>
+ <para>
+ OID of the table being repacked.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role=3D"catalog_table_entry"><para role=3D"column_definition">
+ <structfield>phase</structfield> <type>text</type>
+ </para>
+ <para>
+ Current processing phase. See <xref linkend=3D"repack-phases"/>.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role=3D"catalog_table_entry"><para role=3D"column_definition">
+ <structfield>repack_index_relid</structfield> <type>oid</type>
+ </para>
+ <para>
+ If the table is being scanned using an index, this is the OID of the
+ index being used; otherwise, it is zero.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role=3D"catalog_table_entry"><para role=3D"column_definition">
+ <structfield>heap_tuples_scanned</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of heap tuples scanned.
+ This counter only advances when the phase is
+ <literal>seq scanning heap</literal>,
+ <literal>index scanning heap</literal>
+ or <literal>writing new heap</literal>.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role=3D"catalog_table_entry"><para role=3D"column_definition">
+ <structfield>heap_tuples_written</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of heap tuples written.
+ This counter only advances when the phase is
+ <literal>seq scanning heap</literal>,
+ <literal>index scanning heap</literal>
+ or <literal>writing new heap</literal>.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role=3D"catalog_table_entry"><para role=3D"column_definition">
+ <structfield>heap_blks_total</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Total number of heap blocks in the table. This number is reported
+ as of the beginning of <literal>seq scanning heap</literal>.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role=3D"catalog_table_entry"><para role=3D"column_definition">
+ <structfield>heap_blks_scanned</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of heap blocks scanned. This counter only advances when the
+ phase is <literal>seq scanning heap</literal>.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role=3D"catalog_table_entry"><para role=3D"column_definition">
+ <structfield>index_rebuild_count</structfield> <type>bigint</type>
+ </para>
+ <para>
+ Number of indexes rebuilt. This counter only advances when the pha=
se
+ is <literal>rebuilding index</literal>.
+ </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <table id=3D"repack-phases">
+ <title>REPACK Phases</title>
+ <tgroup cols=3D"2">
+ <colspec colname=3D"col1" colwidth=3D"1*"/>
+ <colspec colname=3D"col2" colwidth=3D"2*"/>
+ <thead>
+ <row>
+ <entry>Phase</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry><literal>initializing</literal></entry>
+ <entry>
+ The command is preparing to begin scanning the heap. This phase is
+ expected to be very brief.
+ </entry>
+ </row>
+ <row>
+ <entry><literal>seq scanning heap</literal></entry>
+ <entry>
+ The command is currently scanning the table using a sequential scan.
+ </entry>
+ </row>
+ <row>
+ <entry><literal>index scanning heap</literal></entry>
+ <entry>
+ <command>REPACK</command> is currently scanning the table using an =
index scan.
+ </entry>
+ </row>
+ <row>
+ <entry><literal>sorting tuples</literal></entry>
+ <entry>
+ <command>REPACK</command> is currently sorting tuples.
+ </entry>
+ </row>
+ <row>
+ <entry><literal>writing new heap</literal></entry>
+ <entry>
+ <command>REPACK</command> is currently writing the new heap.
+ </entry>
+ </row>
+ <row>
+ <entry><literal>swapping relation files</literal></entry>
+ <entry>
+ The command is currently swapping newly-built files into place.
+ </entry>
+ </row>
+ <row>
+ <entry><literal>rebuilding index</literal></entry>
+ <entry>
+ The command is currently rebuilding an index.
+ </entry>
+ </row>
+ <row>
+ <entry><literal>performing final cleanup</literal></entry>
+ <entry>
+ The command is performing final cleanup. When this phase is
+ completed, <command>REPACK</command> will end.
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+ </sect2>
+
<sect2 id=3D"copy-progress-reporting">
<title>COPY Progress Reporting</title>
=20
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index e167406c744..5df944d13ca 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -167,6 +167,7 @@ Complete list of usable sgml source files in this direc=
tory.
<!ENTITY refreshMaterializedView SYSTEM "refresh_materialized_view.sgml">
<!ENTITY reindex SYSTEM "reindex.sgml">
<!ENTITY releaseSavepoint SYSTEM "release_savepoint.sgml">
+<!ENTITY repack SYSTEM "repack.sgml">
<!ENTITY reset SYSTEM "reset.sgml">
<!ENTITY revoke SYSTEM "revoke.sgml">
<!ENTITY rollback SYSTEM "rollback.sgml">
@@ -213,6 +214,7 @@ Complete list of usable sgml source files in this direc=
tory.
<!ENTITY pgIsready SYSTEM "pg_isready.sgml">
<!ENTITY pgReceivewal SYSTEM "pg_receivewal.sgml">
<!ENTITY pgRecvlogical SYSTEM "pg_recvlogical.sgml">
+<!ENTITY pgRepackdb SYSTEM "pg_repackdb.sgml">
<!ENTITY pgResetwal SYSTEM "pg_resetwal.sgml">
<!ENTITY pgRestore SYSTEM "pg_restore.sgml">
<!ENTITY pgRewind SYSTEM "pg_rewind.sgml">
diff --git a/doc/src/sgml/ref/cluster.sgml b/doc/src/sgml/ref/cluster.sgml
index 0b47460080b..2cda711bc9f 100644
--- a/doc/src/sgml/ref/cluster.sgml
+++ b/doc/src/sgml/ref/cluster.sgml
@@ -33,51 +33,13 @@ CLUSTER [ ( <replaceable class=3D"parameter">option</re=
placeable> [, ...] ) ] [ <r
<title>Description</title>
=20
<para>
- <command>CLUSTER</command> instructs <productname>PostgreSQL</productna=
me>
- to cluster the table specified
- by <replaceable class=3D"parameter">table_name</replaceable>
- based on the index specified by
- <replaceable class=3D"parameter">index_name</replaceable>. The index mu=
st
- already have been defined on
- <replaceable class=3D"parameter">table_name</replaceable>.
+ The <command>CLUSTER</command> command is equivalent to
+ <xref linkend=3D"sql-repack"/> with an <literal>USING INDEX</literal>
+ clause. See there for more details.
</para>
=20
- <para>
- When a table is clustered, it is physically reordered
- based on the index information. Clustering is a one-time operation:
- when the table is subsequently updated, the changes are
- not clustered. That is, no attempt is made to store new or
- updated rows according to their index order. (If one wishes, one can
- periodically recluster by issuing the command again. Also, setting
- the table's <literal>fillfactor</literal> storage parameter to less than
- 100% can aid in preserving cluster ordering during updates, since updat=
ed
- rows are kept on the same page if enough space is available there.)
- </para>
-
- <para>
- When a table is clustered, <productname>PostgreSQL</productname>
- remembers which index it was clustered by. The form
- <command>CLUSTER <replaceable class=3D"parameter">table_name</replaceab=
le></command>
- reclusters the table using the same index as before. You can also
- use the <literal>CLUSTER</literal> or <literal>SET WITHOUT CLUSTER</lit=
eral>
- forms of <link linkend=3D"sql-altertable"><command>ALTER TABLE</command=
></link> to set the index to be used for
- future cluster operations, or to clear any previous setting.
- </para>
-
- <para>
- <command>CLUSTER</command> without a
- <replaceable class=3D"parameter">table_name</replaceable> reclusters al=
l the
- previously-clustered tables in the current database that the calling us=
er
- has privileges for. This form of <command>CLUSTER</command> cannot be
- executed inside a transaction block.
- </para>
+<!-- Do we need to describe exactly which options map to what? They seem =
obvious to me. -->
=20
- <para>
- When a table is being clustered, an <literal>ACCESS
- EXCLUSIVE</literal> lock is acquired on it. This prevents any other
- database operations (both reads and writes) from operating on the
- table until the <command>CLUSTER</command> is finished.
- </para>
</refsect1>
=20
<refsect1>
@@ -136,63 +98,12 @@ CLUSTER [ ( <replaceable class=3D"parameter">option</r=
eplaceable> [, ...] ) ] [ <r
on the table.
</para>
=20
- <para>
- In cases where you are accessing single rows randomly
- within a table, the actual order of the data in the
- table is unimportant. However, if you tend to access some
- data more than others, and there is an index that groups
- them together, you will benefit from using <command>CLUSTER</command>.
- If you are requesting a range of indexed values from a table, or a
- single indexed value that has multiple rows that match,
- <command>CLUSTER</command> will help because once the index identifies=
the
- table page for the first row that matches, all other rows
- that match are probably already on the same table page,
- and so you save disk accesses and speed up the query.
- </para>
-
- <para>
- <command>CLUSTER</command> can re-sort the table using either an index=
scan
- on the specified index, or (if the index is a b-tree) a sequential
- scan followed by sorting. It will attempt to choose the method that
- will be faster, based on planner cost parameters and available statist=
ical
- information.
- </para>
-
<para>
While <command>CLUSTER</command> is running, the <xref
linkend=3D"guc-search-path"/> is temporarily changed to <literal>pg_ca=
talog,
pg_temp</literal>.
</para>
=20
- <para>
- When an index scan is used, a temporary copy of the table is created t=
hat
- contains the table data in the index order. Temporary copies of each
- index on the table are created as well. Therefore, you need free spac=
e on
- disk at least equal to the sum of the table size and the index sizes.
- </para>
-
- <para>
- When a sequential scan and sort is used, a temporary sort file is
- also created, so that the peak temporary space requirement is as much
- as double the table size, plus the index sizes. This method is often
- faster than the index scan method, but if the disk space requirement is
- intolerable, you can disable this choice by temporarily setting <xref
- linkend=3D"guc-enable-sort"/> to <literal>off</literal>.
- </para>
-
- <para>
- It is advisable to set <xref linkend=3D"guc-maintenance-work-mem"/> to
- a reasonably large value (but not more than the amount of RAM you can
- dedicate to the <command>CLUSTER</command> operation) before clusterin=
g.
- </para>
-
- <para>
- Because the planner records statistics about the ordering of
- tables, it is advisable to run <link linkend=3D"sql-analyze"><command>=
ANALYZE</command></link>
- on the newly clustered table.
- Otherwise, the planner might make poor choices of query plans.
- </para>
-
<para>
Because <command>CLUSTER</command> remembers which indexes are cluster=
ed,
one can cluster the tables one wants clustered manually the first time,
diff --git a/doc/src/sgml/ref/clusterdb.sgml b/doc/src/sgml/ref/clusterdb.s=
gml
index 0d2051bf6f1..b50c9581a98 100644
--- a/doc/src/sgml/ref/clusterdb.sgml
+++ b/doc/src/sgml/ref/clusterdb.sgml
@@ -64,6 +64,11 @@ PostgreSQL documentation
this utility and via other methods for accessing the server.
</para>
=20
+ <para>
+ <application>clusterdb</application> has been superseded by
+ <application>pg_repackdb</application>.
+ </para>
+
</refsect1>
=20
=20
diff --git a/doc/src/sgml/ref/pg_repackdb.sgml b/doc/src/sgml/ref/pg_repack=
db.sgml
new file mode 100644
index 00000000000..a5ea513c6b1
--- /dev/null
+++ b/doc/src/sgml/ref/pg_repackdb.sgml
@@ -0,0 +1,488 @@
+<!--
+doc/src/sgml/ref/pg_repackdb.sgml
+PostgreSQL documentation
+-->
+
+<refentry id=3D"app-pgrepackdb">
+ <indexterm zone=3D"app-pgrepackdb">
+ <primary>pg_repackdb</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle><application>pg_repackdb</application></refentrytitle>
+ <manvolnum>1</manvolnum>
+ <refmiscinfo>Application</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>pg_repackdb</refname>
+ <refpurpose>repack and analyze a <productname>PostgreSQL</productname>
+ database</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_repackdb</command>
+ <arg rep=3D"repeat"><replaceable>connection-option</replaceable></arg>
+ <arg rep=3D"repeat"><replaceable>option</replaceable></arg>
+
+ <arg choice=3D"plain" rep=3D"repeat">
+ <arg choice=3D"opt">
+ <group choice=3D"plain">
+ <arg choice=3D"plain"><option>-t</option></arg>
+ <arg choice=3D"plain"><option>--table</option></arg>
+ </group>
+ <replaceable>table</replaceable>
+ <arg choice=3D"opt">( <replaceable class=3D"parameter">column</replac=
eable> [,...] )</arg>
+ </arg>
+ </arg>
+
+ <arg choice=3D"opt">
+ <group choice=3D"plain">
+ <arg choice=3D"plain"><replaceable>dbname</replaceable></arg>
+ <arg choice=3D"plain"><option>-a</option></arg>
+ <arg choice=3D"plain"><option>--all</option></arg>
+ </group>
+ </arg>
+ </cmdsynopsis>
+
+ <cmdsynopsis>
+ <command>pg_repackdb</command>
+ <arg rep=3D"repeat"><replaceable>connection-option</replaceable></arg>
+ <arg rep=3D"repeat"><replaceable>option</replaceable></arg>
+
+ <arg choice=3D"plain" rep=3D"repeat">
+ <arg choice=3D"opt">
+ <group choice=3D"plain">
+ <arg choice=3D"plain"><option>-n</option></arg>
+ <arg choice=3D"plain"><option>--schema</option></arg>
+ </group>
+ <replaceable>schema</replaceable>
+ </arg>
+ </arg>
+
+ <arg choice=3D"opt">
+ <group choice=3D"plain">
+ <arg choice=3D"plain"><replaceable>dbname</replaceable></arg>
+ <arg choice=3D"plain"><option>-a</option></arg>
+ <arg choice=3D"plain"><option>--all</option></arg>
+ </group>
+ </arg>
+ </cmdsynopsis>
+
+ <cmdsynopsis>
+ <command>pg_repackdb</command>
+ <arg rep=3D"repeat"><replaceable>connection-option</replaceable></arg>
+ <arg rep=3D"repeat"><replaceable>option</replaceable></arg>
+
+ <arg choice=3D"plain" rep=3D"repeat">
+ <arg choice=3D"opt">
+ <group choice=3D"plain">
+ <arg choice=3D"plain"><option>-N</option></arg>
+ <arg choice=3D"plain"><option>--exclude-schema</option></arg>
+ </group>
+ <replaceable>schema</replaceable>
+ </arg>
+ </arg>
+
+ <arg choice=3D"opt">
+ <group choice=3D"plain">
+ <arg choice=3D"plain"><replaceable>dbname</replaceable></arg>
+ <arg choice=3D"plain"><option>-a</option></arg>
+ <arg choice=3D"plain"><option>--all</option></arg>
+ </group>
+ </arg>
+ </cmdsynopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+
+ <para>
+ <application>pg_repackdb</application> is a utility for repacking a
+ <productname>PostgreSQL</productname> database.
+ <application>pg_repackdb</application> will also generate internal
+ statistics used by the <productname>PostgreSQL</productname> query
+ optimizer.
+ </para>
+
+ <para>
+ <application>pg_repackdb</application> is a wrapper around the SQL
+ command <link linkend=3D"sql-repack"><command>REPACK</command></link>. =
There
+ is no effective difference between repacking and analyzing databases via
+ this utility and via other methods for accessing the server.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Options</title>
+
+ <para>
+ <application>pg_repackdb</application> accepts the following command-l=
ine arguments:
+ <variablelist>
+ <varlistentry>
+ <term><option>-a</option></term>
+ <term><option>--all</option></term>
+ <listitem>
+ <para>
+ Repack all databases.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option><optional>-d</optional> <replaceable class=3D"paramete=
r">dbname</replaceable></option></term>
+ <term><option><optional>--dbname=3D</optional><replaceable class=3D"=
parameter">dbname</replaceable></option></term>
+ <listitem>
+ <para>
+ Specifies the name of the database to be repacked or analyzed,
+ when <option>-a</option>/<option>--all</option> is not used. If t=
his
+ is not specified, the database name is read from the environment
+ variable <envar>PGDATABASE</envar>. If that is not set, the user =
name
+ specified for the connection is used.
+ The <replaceable>dbname</replaceable> can be
+ a <link linkend=3D"libpq-connstring">connection string</link>. If=
so,
+ connection string parameters will override any conflicting command
+ line options.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-e</option></term>
+ <term><option>--echo</option></term>
+ <listitem>
+ <para>
+ Echo the commands that <application>pg_repackdb</application>
+ generates and sends to the server.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--index<optional>=3D<replaceable class=3D"parameter">i=
ndex_name</replaceable></optional></option></term>
+ <listitem>
+ <para>
+ Pass the <literal>USING INDEX</literal> clause to <literal>REPACK<=
/literal>,
+ and optionally the index name to specify.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-j <replaceable class=3D"parameter">njobs</replaceable=
></option></term>
+ <term><option>--jobs=3D<replaceable class=3D"parameter">njobs</repla=
ceable></option></term>
+ <listitem>
+ <para>
+ Execute the repack or analyze commands in parallel by running
+ <replaceable class=3D"parameter">njobs</replaceable>
+ commands simultaneously. This option may reduce the processing ti=
me
+ but it also increases the load on the database server.
+ </para>
+ <para>
+ <application>pg_repackdb</application> will open
+ <replaceable class=3D"parameter">njobs</replaceable> connections t=
o the
+ database, so make sure your <xref linkend=3D"guc-max-connections"/>
+ setting is high enough to accommodate all connections.
+ </para>
+ <para>
+ Note that using this mode might cause deadlock failures if certain
+ system catalogs are processed in parallel.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-n <replaceable class=3D"parameter">schema</replaceabl=
e></option></term>
+ <term><option>--schema=3D<replaceable class=3D"parameter">schema</re=
placeable></option></term>
+ <listitem>
+ <para>
+ Repack or analyze all tables in
+ <replaceable class=3D"parameter">schema</replaceable> only. Multi=
ple
+ schemas can be repacked by writing multiple <option>-n</option>
+ switches.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-N <replaceable class=3D"parameter">schema</replaceabl=
e></option></term>
+ <term><option>--exclude-schema=3D<replaceable class=3D"parameter">sc=
hema</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not repack or analyze any tables in
+ <replaceable class=3D"parameter">schema</replaceable>. Multiple s=
chemas
+ can be excluded by writing multiple <option>-N</option> switches.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-q</option></term>
+ <term><option>--quiet</option></term>
+ <listitem>
+ <para>
+ Do not display progress messages.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-t <replaceable class=3D"parameter">table</replaceable=
> [ (<replaceable class=3D"parameter">column</replaceable> [,...]) ]</optio=
n></term>
+ <term><option>--table=3D<replaceable class=3D"parameter">table</repl=
aceable> [ (<replaceable class=3D"parameter">column</replaceable> [,...]) ]=
</option></term>
+ <listitem>
+ <para>
+ Repack or analyze <replaceable class=3D"parameter">table</replacea=
ble>
+ only. Column names can be specified only in conjunction with
+ the <option>--analyze</option> option. Multiple tables can be
+ repacked by writing multiple
+ <option>-t</option> switches.
+ </para>
+ <tip>
+ <para>
+ If you specify columns, you probably have to escape the parenthes=
es
+ from the shell. (See examples below.)
+ </para>
+ </tip>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-v</option></term>
+ <term><option>--verbose</option></term>
+ <listitem>
+ <para>
+ Print detailed information during processing.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-V</option></term>
+ <term><option>--version</option></term>
+ <listitem>
+ <para>
+ Print the <application>pg_repackdb</application> version and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-z</option></term>
+ <term><option>--analyze</option></term>
+ <listitem>
+ <para>
+ Also calculate statistics for use by the optimizer. If a column n=
ame
+ list is given, only compute statistics for those columns.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-?</option></term>
+ <term><option>--help</option></term>
+ <listitem>
+ <para>
+ Show help about <application>pg_repackdb</application> command line
+ arguments, and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ <para>
+ <application>pg_repackdb</application> also accepts
+ the following command-line arguments for connection parameters:
+ <variablelist>
+ <varlistentry>
+ <term><option>-h <replaceable class=3D"parameter">host</replaceable>=
</option></term>
+ <term><option>--host=3D<replaceable class=3D"parameter">host</replac=
eable></option></term>
+ <listitem>
+ <para>
+ Specifies the host name of the machine on which the server
+ is running. If the value begins with a slash, it is used
+ as the directory for the Unix domain socket.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-p <replaceable class=3D"parameter">port</replaceable>=
</option></term>
+ <term><option>--port=3D<replaceable class=3D"parameter">port</replac=
eable></option></term>
+ <listitem>
+ <para>
+ Specifies the TCP port or local Unix domain socket file
+ extension on which the server
+ is listening for connections.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-U <replaceable class=3D"parameter">username</replacea=
ble></option></term>
+ <term><option>--username=3D<replaceable class=3D"parameter">username=
</replaceable></option></term>
+ <listitem>
+ <para>
+ User name to connect as.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-w</option></term>
+ <term><option>--no-password</option></term>
+ <listitem>
+ <para>
+ Never issue a password prompt. If the server requires
+ password authentication and a password is not available by
+ other means such as a <filename>.pgpass</filename> file, the
+ connection attempt will fail. This option can be useful in
+ batch jobs and scripts where no user is present to enter a
+ password.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-W</option></term>
+ <term><option>--password</option></term>
+ <listitem>
+ <para>
+ Force <application>pg_repackdb</application> to prompt for a
+ password before connecting to a database.
+ </para>
+
+ <para>
+ This option is never essential, since
+ <application>pg_repackdb</application> will automatically prompt
+ for a password if the server demands password authentication.
+ However, <application>pg_repackdb</application> will waste a
+ connection attempt finding out that the server wants a password.
+ In some cases it is worth typing <option>-W</option> to avoid the =
extra
+ connection attempt.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--maintenance-db=3D<replaceable class=3D"parameter">db=
name</replaceable></option></term>
+ <listitem>
+ <para>
+ When the <option>-a</option>/<option>--all</option> is used, conne=
ct
+ to this database to gather the list of databases to repack.
+ If not specified, the <literal>postgres</literal> database will be=
used,
+ or if that does not exist, <literal>template1</literal> will be us=
ed.
+ This can be a <link linkend=3D"libpq-connstring">connection
+ string</link>. If so, connection string parameters will override =
any
+ conflicting command line options. Also, connection string paramet=
ers
+ other than the database name itself will be re-used when connecting
+ to other databases.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </refsect1>
+
+
+ <refsect1>
+ <title>Environment</title>
+
+ <variablelist>
+ <varlistentry>
+ <term><envar>PGDATABASE</envar></term>
+ <term><envar>PGHOST</envar></term>
+ <term><envar>PGPORT</envar></term>
+ <term><envar>PGUSER</envar></term>
+
+ <listitem>
+ <para>
+ Default connection parameters
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><envar>PG_COLOR</envar></term>
+ <listitem>
+ <para>
+ Specifies whether to use color in diagnostic messages. Possible valu=
es
+ are <literal>always</literal>, <literal>auto</literal> and
+ <literal>never</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+
+ <para>
+ This utility, like most other <productname>PostgreSQL</productname> uti=
lities,
+ also uses the environment variables supported by <application>libpq</ap=
plication>
+ (see <xref linkend=3D"libpq-envars"/>).
+ </para>
+
+ </refsect1>
+
+
+ <refsect1>
+ <title>Diagnostics</title>
+
+ <para>
+ In case of difficulty, see
+ <xref linkend=3D"sql-repack"/> and <xref linkend=3D"app-psql"/> for
+ discussions of potential problems and error messages.
+ The database server must be running at the
+ targeted host. Also, any default connection settings and environment
+ variables used by the <application>libpq</application> front-end
+ library will apply.
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Examples</title>
+
+ <para>
+ To repack the database <literal>test</literal>:
+<screen>
+<prompt>$ </prompt><userinput>pg_repackdb test</userinput>
+</screen>
+ </para>
+
+ <para>
+ To repack and analyze for the optimizer a database named
+ <literal>bigdb</literal>:
+<screen>
+<prompt>$ </prompt><userinput>pg_repackdb --analyze bigdb</userinput>
+</screen>
+ </para>
+
+ <para>
+ To repack a single table
+ <literal>foo</literal> in a database named
+ <literal>xyzzy</literal>, and analyze a single column
+ <literal>bar</literal> of the table for the optimizer:
+<screen>
+<prompt>$ </prompt><userinput>pg_repackdb --analyze --verbose --table=3D'f=
oo(bar)' xyzzy</userinput>
+</screen></para>
+
+ <para>
+ To repack all tables in the <literal>foo</literal> and <literal>bar</l=
iteral> schemas
+ in a database named <literal>xyzzy</literal>:
+<screen>
+<prompt>$ </prompt><userinput>pg_repackdb --schema=3D'foo' --schema=3D'bar=
' xyzzy</userinput>
+</screen></para>
+
+
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type=3D"inline">
+ <member><xref linkend=3D"sql-repack"/></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/ref/repack.sgml b/doc/src/sgml/ref/repack.sgml
new file mode 100644
index 00000000000..61d5c2cdef1
--- /dev/null
+++ b/doc/src/sgml/ref/repack.sgml
@@ -0,0 +1,328 @@
+<!--
+doc/src/sgml/ref/repack.sgml
+PostgreSQL documentation
+-->
+
+<refentry id=3D"sql-repack">
+ <indexterm zone=3D"sql-repack">
+ <primary>REPACK</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle>REPACK</refentrytitle>
+ <manvolnum>7</manvolnum>
+ <refmiscinfo>SQL - Language Statements</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>REPACK</refname>
+ <refpurpose>rewrite a table to reclaim disk space</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+<synopsis>
+REPACK [ ( <replaceable class=3D"parameter">option</replaceable> [, ...] )=
] [ <replaceable class=3D"parameter">table_and_columns</replaceable> [ USI=
NG INDEX [ <replaceable class=3D"parameter">index_name</replaceable> ] ] ]
+REPACK [ ( <replaceable class=3D"parameter">option</replaceable> [, ...] )=
] USING INDEX
+
+<phrase>where <replaceable class=3D"parameter">option</replaceable> can be=
one of:</phrase>
+
+ VERBOSE [ <replaceable class=3D"parameter">boolean</replaceable> ]
+ ANALYZE [ <replaceable class=3D"parameter">boolean</replaceable> ]
+
+<phrase>and <replaceable class=3D"parameter">table_and_columns</replaceabl=
e> is:</phrase>
+
+ <replaceable class=3D"parameter">table_name</replaceable> [ ( <replace=
able class=3D"parameter">column_name</replaceable> [, ...] ) ]
+</synopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Description</title>
+
+ <para>
+ <command>REPACK</command> reclaims storage occupied by dead
+ tuples. Unlike <command>VACUUM</command>, it does so by rewriting the
+ entire contents of the table specified
+ by <replaceable class=3D"parameter">table_name</replaceable> into a new=
disk
+ file with no extra space (except for the space guaranteed by
+ the <literal>fillfactor</literal> storage parameter), allowing unused s=
pace
+ to be returned to the operating system.
+ </para>
+
+ <para>
+ Without
+ a <replaceable class=3D"parameter">table_name</replaceable>, <command>R=
EPACK</command>
+ processes every table and materialized view in the current database that
+ the current user has the <literal>MAINTAIN</literal> privilege on. This
+ form of <command>REPACK</command> cannot be executed inside a transacti=
on
+ block.
+ </para>
+
+ <para>
+ If a <literal>USING INDEX</literal> clause is specified, the rows are
+ physically reordered based on information from an index. Please see the
+ notes on clustering below.
+ </para>
+
+ <para>
+ When a table is being repacked, an <literal>ACCESS EXCLUSIVE</literal> =
lock
+ is acquired on it. This prevents any other database operations (both re=
ads
+ and writes) from operating on the table until the <command>REPACK</comm=
and>
+ is finished.
+ </para>
+
+ <refsect2 id=3D"sql-repack-notes-on-clustering" xreflabel=3D"Notes on Cl=
ustering">
+ <title>Notes on Clustering</title>
+
+ <para>
+ If the <literal>USING INDEX</literal> clause is specified, the rows in
+ the table are physically reordered following an index: if an index name
+ is specified in the command, then that index is used; if no index name
+ is specified, then the index that has been configured as the index to
+ cluster on. If no index has been configured in this way, an error is
+ thrown. The index given in the <literal>USING INDEX</literal> clause
+ is configured as the index to cluster on, as well as an index given
+ to the <command>CLUSTER</command> command. An index can be set
+ manually using <command>ALTER TABLE ... CLUSTER ON</command>, and reset
+ with <command>ALTER TABLE ... SET WITHOUT CLUSTER</command>.
+ </para>
+
+ <para>
+ If no table name is specified in <command>REPACK USING INDEX</command>,
+ all tables which have a clustering index defined and which the calling
+ user has privileges for are processed.
+ </para>
+
+ <para>
+ Clustering is a one-time operation: when the table is
+ subsequently updated, the changes are not clustered. That is, no atte=
mpt
+ is made to store new or updated rows according to their index order. =
(If
+ one wishes, one can periodically recluster by issuing the command agai=
n.
+ Also, setting the table's <literal>fillfactor</literal> storage parame=
ter
+ to less than 100% can aid in preserving cluster ordering during update=
s,
+ since updated rows are kept on the same page if enough space is availa=
ble
+ there.)
+ </para>
+
+ <para>
+ In cases where you are accessing single rows randomly within a table, =
the
+ actual order of the data in the table is unimportant. However, if you =
tend
+ to access some data more than others, and there is an index that groups
+ them together, you will benefit from using clustering. If
+ you are requesting a range of indexed values from a table, or a single
+ indexed value that has multiple rows that match,
+ <command>REPACK</command> will help because once the index identifies =
the
+ table page for the first row that matches, all other rows that match a=
re
+ probably already on the same table page, and so you save disk accesses=
and
+ speed up the query.
+ </para>
+
+ <para>
+ <command>REPACK</command> can re-sort the table using either an index =
scan
+ on the specified index (if the index is a b-tree), or a sequential scan
+ followed by sorting. It will attempt to choose the method that will be
+ faster, based on planner cost parameters and available statistical
+ information.
+ </para>
+
+ <para>
+ Because the planner records statistics about the ordering of tables, i=
t is
+ advisable to
+ run <link linkend=3D"sql-analyze"><command>ANALYZE</command></link> on=
the
+ newly repacked table. Otherwise, the planner might make poor choices =
of
+ query plans.
+ </para>
+ </refsect2>
+
+ <refsect2 id=3D"sql-repack-notes-on-resources" xreflabel=3D"Notes on Res=
ources">
+ <title>Notes on Resources</title>
+
+ <para>
+ When an index scan or a sequential scan without sort is used, a tempor=
ary
+ copy of the table is created that contains the table data in the index
+ order. Temporary copies of each index on the table are created as wel=
l.
+ Therefore, you need free space on disk at least equal to the sum of the
+ table size and the index sizes.
+ </para>
+
+ <para>
+ When a sequential scan and sort is used, a temporary sort file is also
+ created, so that the peak temporary space requirement is as much as do=
uble
+ the table size, plus the index sizes. This method is often faster than
+ the index scan method, but if the disk space requirement is intolerabl=
e,
+ you can disable this choice by temporarily setting
+ <xref linkend=3D"guc-enable-sort"/> to <literal>off</literal>.
+ </para>
+
+ <para>
+ It is advisable to set <xref linkend=3D"guc-maintenance-work-mem"/> to=
a
+ reasonably large value (but not more than the amount of RAM you can
+ dedicate to the <command>REPACK</command> operation) before repacking.
+ </para>
+ </refsect2>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Parameters</title>
+
+ <variablelist>
+ <varlistentry>
+ <term><replaceable class=3D"parameter">table_name</replaceable></term>
+ <listitem>
+ <para>
+ The name (possibly schema-qualified) of a table.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class=3D"parameter">column_name</replaceable></term>
+ <listitem>
+ <para>
+ The name of a specific column to analyze. Defaults to all columns.
+ If a column list is specific, <literal>ANALYZE</literal> must also
+ be specified.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class=3D"parameter">index_name</replaceable></term>
+ <listitem>
+ <para>
+ The name of an index.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>VERBOSE</literal></term>
+ <listitem>
+ <para>
+ Prints a progress report as each table is repacked
+ at <literal>INFO</literal> level.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>ANALYZE</literal></term>
+ <term><literal>ANALYSE</literal></term>
+ <listitem>
+ <para>
+ Applies <xref linkend=3D"sql-analyze"/> on the table after repacking=
. This is
+ currently only supported when a single (non-partitioned) table is sp=
ecified.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class=3D"parameter">boolean</replaceable></term>
+ <listitem>
+ <para>
+ Specifies whether the selected option should be turned on or off.
+ You can write <literal>TRUE</literal>, <literal>ON</literal>, or
+ <literal>1</literal> to enable the option, and <literal>FALSE</liter=
al>,
+ <literal>OFF</literal>, or <literal>0</literal> to disable it. The
+ <replaceable class=3D"parameter">boolean</replaceable> value can also
+ be omitted, in which case <literal>TRUE</literal> is assumed.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </refsect1>
+
+ <refsect1>
+ <title>Notes</title>
+
+ <para>
+ To repack a table, one must have the <literal>MAINTAIN</literal> privi=
lege
+ on the table.
+ </para>
+
+ <para>
+ While <command>REPACK</command> is running, the <xref
+ linkend=3D"guc-search-path"/> is temporarily changed to <literal>pg_ca=
talog,
+ pg_temp</literal>.
+ </para>
+
+ <para>
+ Each backend running <command>REPACK</command> will report its progress
+ in the <structname>pg_stat_progress_repack</structname> view. See
+ <xref linkend=3D"repack-progress-reporting"/> for details.
+ </para>
+
+ <para>
+ Repacking a partitioned table repacks each of its partitions. If an in=
dex
+ is specified, each partition is repacked using the partition of that
+ index. <command>REPACK</command> on a partitioned table cannot be exec=
uted
+ inside a transaction block.
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Examples</title>
+
+ <para>
+ Repack the table <literal>employees</literal>:
+<programlisting>
+REPACK employees;
+</programlisting>
+ </para>
+
+ <para>
+ Repack the table <literal>employees</literal> on the basis of its
+ index <literal>employees_ind</literal> (Since index is used here, this =
is
+ effectively clustering):
+<programlisting>
+REPACK employees USING INDEX employees_ind;
+</programlisting>
+ </para>
+
+ <para>
+ Repack the table <literal>cases</literal> on physical ordering,
+ running an <command>ANALYZE</command> on the given columns once
+ repacking is done, showing informational messages:
+<programlisting>
+REPACK (ANALYZE, VERBOSE) cases (district, case_nr);
+</programlisting>
+ </para>
+
+ <para>
+ Repack all tables in the database on which you have
+ the <literal>MAINTAIN</literal> privilege:
+<programlisting>
+REPACK;
+</programlisting>
+ </para>
+
+ <para>
+ Repack all tables for which a clustering index has previously been
+ configured on which you have the <literal>MAINTAIN</literal> privilege,
+ showing informational messages:
+<programlisting>
+REPACK (VERBOSE) USING INDEX;
+</programlisting>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Compatibility</title>
+
+ <para>
+ There is no <command>REPACK</command> statement in the SQL standard.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type=3D"inline">
+ <member><xref linkend=3D"app-pgrepackdb"/></member>
+ <member><xref linkend=3D"repack-progress-reporting"/></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/ref/vacuum.sgml b/doc/src/sgml/ref/vacuum.sgml
index 6d0fdd43cfb..ac5d083d468 100644
--- a/doc/src/sgml/ref/vacuum.sgml
+++ b/doc/src/sgml/ref/vacuum.sgml
@@ -25,7 +25,6 @@ VACUUM [ ( <replaceable class=3D"parameter">option</repla=
ceable> [, ...] ) ] [ <re
=20
<phrase>where <replaceable class=3D"parameter">option</replaceable> can be=
one of:</phrase>
=20
- FULL [ <replaceable class=3D"parameter">boolean</replaceable> ]
FREEZE [ <replaceable class=3D"parameter">boolean</replaceable> ]
VERBOSE [ <replaceable class=3D"parameter">boolean</replaceable> ]
ANALYZE [ <replaceable class=3D"parameter">boolean</replaceable> ]
@@ -39,6 +38,7 @@ VACUUM [ ( <replaceable class=3D"parameter">option</repla=
ceable> [, ...] ) ] [ <re
SKIP_DATABASE_STATS [ <replaceable class=3D"parameter">boolean</replac=
eable> ]
ONLY_DATABASE_STATS [ <replaceable class=3D"parameter">boolean</replac=
eable> ]
BUFFER_USAGE_LIMIT <replaceable class=3D"parameter">size</replaceable>
+ FULL [ <replaceable class=3D"parameter">boolean</replaceable> ]
=20
<phrase>and <replaceable class=3D"parameter">table_and_columns</replaceabl=
e> is:</phrase>
=20
@@ -95,20 +95,6 @@ VACUUM [ ( <replaceable class=3D"parameter">option</repl=
aceable> [, ...] ) ] [ <re
<title>Parameters</title>
=20
<variablelist>
- <varlistentry>
- <term><literal>FULL</literal></term>
- <listitem>
- <para>
- Selects <quote>full</quote> vacuum, which can reclaim more
- space, but takes much longer and exclusively locks the table.
- This method also requires extra disk space, since it writes a
- new copy of the table and doesn't release the old copy until
- the operation is complete. Usually this should only be used when a
- significant amount of space needs to be reclaimed from within the ta=
ble.
- </para>
- </listitem>
- </varlistentry>
-
<varlistentry>
<term><literal>FREEZE</literal></term>
<listitem>
@@ -362,6 +348,23 @@ VACUUM [ ( <replaceable class=3D"parameter">option</re=
placeable> [, ...] ) ] [ <re
</listitem>
</varlistentry>
=20
+ <varlistentry>
+ <term><literal>FULL</literal></term>
+ <listitem>
+ <para>
+ This option, which is deprecated, makes <command>VACUUM</command>
+ behave like <command>REPACK</command> without a
+ <literal>USING INDEX</literal> clause.
+ This method of compacting the table takes much longer than
+ <command>VACUUM</command> and exclusively locks the table.
+ This method also requires extra disk space, since it writes a
+ new copy of the table and doesn't release the old copy until
+ the operation is complete. Usually this should only be used when a
+ significant amount of space needs to be reclaimed from within the ta=
ble.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><replaceable class=3D"parameter">boolean</replaceable></term>
<listitem>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index 2cf02c37b17..5d9a8a25a02 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -195,6 +195,7 @@
&refreshMaterializedView;
&reindex;
&releaseSavepoint;
+ &repack;
&reset;
&revoke;
&rollback;
@@ -258,6 +259,7 @@
&pgIsready;
&pgReceivewal;
&pgRecvlogical;
+ &pgRepackdb;
&pgRestore;
&pgVerifyBackup;
&psqlRef;
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/=
heap/heapam_handler.c
index cbef73e5d4b..7d4b48e5a97 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -741,13 +741,13 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Re=
lation NewHeap,
if (OldIndex !=3D NULL && !use_sort)
{
const int ci_index[] =3D {
- PROGRESS_CLUSTER_PHASE,
- PROGRESS_CLUSTER_INDEX_RELID
+ PROGRESS_REPACK_PHASE,
+ PROGRESS_REPACK_INDEX_RELID
};
int64 ci_val[2];
=20
/* Set phase and OIDOldIndex to columns */
- ci_val[0] =3D PROGRESS_CLUSTER_PHASE_INDEX_SCAN_HEAP;
+ ci_val[0] =3D PROGRESS_REPACK_PHASE_INDEX_SCAN_HEAP;
ci_val[1] =3D RelationGetRelid(OldIndex);
pgstat_progress_update_multi_param(2, ci_index, ci_val);
=20
@@ -759,15 +759,15 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Re=
lation NewHeap,
else
{
/* In scan-and-sort mode and also VACUUM FULL, set phase */
- pgstat_progress_update_param(PROGRESS_CLUSTER_PHASE,
- PROGRESS_CLUSTER_PHASE_SEQ_SCAN_HEAP);
+ pgstat_progress_update_param(PROGRESS_REPACK_PHASE,
+ PROGRESS_REPACK_PHASE_SEQ_SCAN_HEAP);
=20
tableScan =3D table_beginscan(OldHeap, SnapshotAny, 0, (ScanKey) NULL);
heapScan =3D (HeapScanDesc) tableScan;
indexScan =3D NULL;
=20
/* Set total heap blocks */
- pgstat_progress_update_param(PROGRESS_CLUSTER_TOTAL_HEAP_BLKS,
+ pgstat_progress_update_param(PROGRESS_REPACK_TOTAL_HEAP_BLKS,
heapScan->rs_nblocks);
}
=20
@@ -809,7 +809,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Rela=
tion NewHeap,
* is manually updated to the correct value when the table
* scan finishes.
*/
- pgstat_progress_update_param(PROGRESS_CLUSTER_HEAP_BLKS_SCANNED,
+ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_BLKS_SCANNED,
heapScan->rs_nblocks);
break;
}
@@ -825,7 +825,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Rela=
tion NewHeap,
*/
if (prev_cblock !=3D heapScan->rs_cblock)
{
- pgstat_progress_update_param(PROGRESS_CLUSTER_HEAP_BLKS_SCANNED,
+ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_BLKS_SCANNED,
(heapScan->rs_cblock +
heapScan->rs_nblocks -
heapScan->rs_startblock
@@ -926,14 +926,14 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Re=
lation NewHeap,
* In scan-and-sort mode, report increase in number of tuples
* scanned
*/
- pgstat_progress_update_param(PROGRESS_CLUSTER_HEAP_TUPLES_SCANNED,
+ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_SCANNED,
*num_tuples);
}
else
{
const int ct_index[] =3D {
- PROGRESS_CLUSTER_HEAP_TUPLES_SCANNED,
- PROGRESS_CLUSTER_HEAP_TUPLES_WRITTEN
+ PROGRESS_REPACK_HEAP_TUPLES_SCANNED,
+ PROGRESS_REPACK_HEAP_TUPLES_WRITTEN
};
int64 ct_val[2];
=20
@@ -966,14 +966,14 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Re=
lation NewHeap,
double n_tuples =3D 0;
=20
/* Report that we are now sorting tuples */
- pgstat_progress_update_param(PROGRESS_CLUSTER_PHASE,
- PROGRESS_CLUSTER_PHASE_SORT_TUPLES);
+ pgstat_progress_update_param(PROGRESS_REPACK_PHASE,
+ PROGRESS_REPACK_PHASE_SORT_TUPLES);
=20
tuplesort_performsort(tuplesort);
=20
/* Report that we are now writing new heap */
- pgstat_progress_update_param(PROGRESS_CLUSTER_PHASE,
- PROGRESS_CLUSTER_PHASE_WRITE_NEW_HEAP);
+ pgstat_progress_update_param(PROGRESS_REPACK_PHASE,
+ PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP);
=20
for (;;)
{
@@ -991,7 +991,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Rela=
tion NewHeap,
values, isnull,
rwstate);
/* Report n_tuples */
- pgstat_progress_update_param(PROGRESS_CLUSTER_HEAP_TUPLES_WRITTEN,
+ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_WRITTEN,
n_tuples);
}
=20
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 43de42ce39e..5ee6389d39c 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -4077,7 +4077,7 @@ reindex_relation(const ReindexStmt *stmt, Oid relid, =
int flags,
Assert(!ReindexIsProcessingIndex(indexOid));
=20
/* Set index rebuild count */
- pgstat_progress_update_param(PROGRESS_CLUSTER_INDEX_REBUILD_COUNT,
+ pgstat_progress_update_param(PROGRESS_REPACK_INDEX_REBUILD_COUNT,
i);
i++;
}
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/sys=
tem_views.sql
index 7553f31fef0..3f05ba3083a 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1283,14 +1283,15 @@ CREATE VIEW pg_stat_progress_vacuum AS
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid =3D D.oid;
=20
-CREATE VIEW pg_stat_progress_cluster AS
+CREATE VIEW pg_stat_progress_repack AS
SELECT
S.pid AS pid,
S.datid AS datid,
D.datname AS datname,
S.relid AS relid,
CASE S.param1 WHEN 1 THEN 'CLUSTER'
- WHEN 2 THEN 'VACUUM FULL'
+ WHEN 2 THEN 'REPACK'
+ WHEN 3 THEN 'VACUUM FULL'
END AS command,
CASE S.param2 WHEN 0 THEN 'initializing'
WHEN 1 THEN 'seq scanning heap'
@@ -1301,15 +1302,35 @@ CREATE VIEW pg_stat_progress_cluster AS
WHEN 6 THEN 'rebuilding index'
WHEN 7 THEN 'performing final cleanup'
END AS phase,
- CAST(S.param3 AS oid) AS cluster_index_relid,
+ CAST(S.param3 AS oid) AS repack_index_relid,
S.param4 AS heap_tuples_scanned,
S.param5 AS heap_tuples_written,
S.param6 AS heap_blks_total,
S.param7 AS heap_blks_scanned,
S.param8 AS index_rebuild_count
- FROM pg_stat_get_progress_info('CLUSTER') AS S
+ FROM pg_stat_get_progress_info('REPACK') AS S
LEFT JOIN pg_database D ON S.datid =3D D.oid;
=20
+-- This view is as the one above, except for renaming a column and avoiding
+-- 'REPACK' as a command name to report.
+CREATE VIEW pg_stat_progress_cluster AS
+ SELECT
+ pid,
+ datid,
+ datname,
+ relid,
+ CASE WHEN command IN ('CLUSTER', 'VACUUM FULL') THEN command
+ WHEN repack_index_relid =3D 0 THEN 'VACUUM FULL'
+ ELSE 'CLUSTER' END AS command,
+ phase,
+ repack_index_relid AS cluster_index_relid,
+ heap_tuples_scanned,
+ heap_tuples_written,
+ heap_blks_total,
+ heap_blks_scanned,
+ index_rebuild_count
+ FROM pg_stat_progress_repack;
+
CREATE VIEW pg_stat_progress_create_index AS
SELECT
S.pid AS pid, S.datid AS datid, D.datname AS datname,
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 60a4617a585..e19675a6d05 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1,7 +1,8 @@
/*-------------------------------------------------------------------------
*
* cluster.c
- * CLUSTER a table on an index. This is now also used for VACUUM FULL.
+ * CLUSTER a table on an index. This is now also used for VACUUM FULL a=
nd
+ * REPACK.
*
* There is hardly anything left of Paul Brown's original implementation...
*
@@ -67,27 +68,35 @@ typedef struct
Oid indexOid;
} RelToCluster;
=20
-
-static void cluster_multiple_rels(List *rtcs, ClusterParams *params);
+static bool cluster_rel_recheck(RepackCommand cmd, Relation OldHeap,
+ Oid indexOid, Oid userid, int options);
static void rebuild_relation(Relation OldHeap, Relation index, bool verbos=
e);
static void copy_table_data(Relation NewHeap, Relation OldHeap, Relation O=
ldIndex,
bool verbose, bool *pSwapToastByContent,
TransactionId *pFreezeXid, MultiXactId *pCutoffMulti);
-static List *get_tables_to_cluster(MemoryContext cluster_context);
-static List *get_tables_to_cluster_partitioned(MemoryContext cluster_conte=
xt,
- Oid indexOid);
-static bool cluster_is_permitted_for_relation(Oid relid, Oid userid);
+static List *get_tables_to_repack(RepackCommand cmd, bool usingindex,
+ MemoryContext permcxt);
+static List *get_tables_to_repack_partitioned(RepackCommand cmd,
+ Oid relid, bool rel_is_index,
+ MemoryContext permcxt);
+static bool cluster_is_permitted_for_relation(RepackCommand cmd,
+ Oid relid, Oid userid);
+static Relation process_single_relation(RepackStmt *stmt,
+ ClusterParams *params);
+static Oid determine_clustered_index(Relation rel, bool usingindex,
+ const char *indexname);
+static const char *RepackCommandAsString(RepackCommand cmd);
=20
=20
-/*------------------------------------------------------------------------=
---
- * This cluster code allows for clustering multiple tables at once. Because
+/*
+ * The repack code allows for processing multiple tables at once. Because
* of this, we cannot just run everything on a single transaction, or we
* would be forced to acquire exclusive locks on all the tables being
* clustered, simultaneously --- very likely leading to deadlock.
*
- * To solve this we follow a similar strategy to VACUUM code,
- * clustering each relation in a separate transaction. For this to work,
- * we need to:
+ * To solve this we follow a similar strategy to VACUUM code, processing e=
ach
+ * relation in a separate transaction. For this to work, we need to:
+ *
* - provide a separate memory context so that we can pass information in
* a way that survives across transactions
* - start a new transaction every time a new relation is clustered
@@ -98,197 +107,166 @@ static bool cluster_is_permitted_for_relation(Oid re=
lid, Oid userid);
*
* The single-relation case does not have any such overhead.
*
- * We also allow a relation to be specified without index. In that case,
- * the indisclustered bit will be looked up, and an ERROR will be thrown
- * if there is no index with the bit set.
- *------------------------------------------------------------------------=
---
+ * We also allow a relation to be repacked following an index, but without
+ * naming a specific one. In that case, the indisclustered bit will be
+ * looked up, and an ERROR will be thrown if no so-marked index is found.
*/
void
-cluster(ParseState *pstate, ClusterStmt *stmt, bool isTopLevel)
+ExecRepack(ParseState *pstate, RepackStmt *stmt, bool isTopLevel)
{
- ListCell *lc;
ClusterParams params =3D {0};
- bool verbose =3D false;
Relation rel =3D NULL;
- Oid indexOid =3D InvalidOid;
- MemoryContext cluster_context;
+ MemoryContext repack_context;
List *rtcs;
=20
/* Parse option list */
- foreach(lc, stmt->params)
+ foreach_node(DefElem, opt, stmt->params)
{
- DefElem *opt =3D (DefElem *) lfirst(lc);
-
if (strcmp(opt->defname, "verbose") =3D=3D 0)
- verbose =3D defGetBoolean(opt);
+ params.options |=3D defGetBoolean(opt) ? CLUOPT_VERBOSE : 0;
+ else if (strcmp(opt->defname, "analyze") =3D=3D 0 ||
+ strcmp(opt->defname, "analyse") =3D=3D 0)
+ params.options |=3D defGetBoolean(opt) ? CLUOPT_ANALYZE : 0;
else
ereport(ERROR,
- (errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("unrecognized %s option \"%s\"",
- "CLUSTER", opt->defname),
- parser_errposition(pstate, opt->location)));
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("unrecognized %s option \"%s\"",
+ RepackCommandAsString(stmt->command),
+ opt->defname),
+ parser_errposition(pstate, opt->location));
}
=20
- params.options =3D (verbose ? CLUOPT_VERBOSE : 0);
-
+ /*
+ * If a single relation is specified, process it and we're done ... unless
+ * the relation is a partitioned table, in which case we fall through.
+ */
if (stmt->relation !=3D NULL)
{
- /* This is the single-relation case. */
- Oid tableOid;
-
- /*
- * Find, lock, and check permissions on the table. We obtain
- * AccessExclusiveLock right away to avoid lock-upgrade hazard in the
- * single-transaction case.
- */
- tableOid =3D RangeVarGetRelidExtended(stmt->relation,
- AccessExclusiveLock,
- 0,
- RangeVarCallbackMaintainsTable,
- NULL);
- rel =3D table_open(tableOid, NoLock);
-
- /*
- * Reject clustering a remote temp table ... their local buffer
- * manager is not going to cope.
- */
- if (RELATION_IS_OTHER_TEMP(rel))
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("cannot cluster temporary tables of other sessions")));
-
- if (stmt->indexname =3D=3D NULL)
- {
- ListCell *index;
-
- /* We need to find the index that has indisclustered set. */
- foreach(index, RelationGetIndexList(rel))
- {
- indexOid =3D lfirst_oid(index);
- if (get_index_isclustered(indexOid))
- break;
- indexOid =3D InvalidOid;
- }
-
- if (!OidIsValid(indexOid))
- ereport(ERROR,
- (errcode(ERRCODE_UNDEFINED_OBJECT),
- errmsg("there is no previously clustered index for table \"%s\"",
- stmt->relation->relname)));
- }
- else
- {
- /*
- * The index is expected to be in the same namespace as the
- * relation.
- */
- indexOid =3D get_relname_relid(stmt->indexname,
- rel->rd_rel->relnamespace);
- if (!OidIsValid(indexOid))
- ereport(ERROR,
- (errcode(ERRCODE_UNDEFINED_OBJECT),
- errmsg("index \"%s\" for table \"%s\" does not exist",
- stmt->indexname, stmt->relation->relname)));
- }
-
- /* For non-partitioned tables, do what we came here to do. */
- if (rel->rd_rel->relkind !=3D RELKIND_PARTITIONED_TABLE)
- {
- cluster_rel(rel, indexOid, ¶ms);
- /* cluster_rel closes the relation, but keeps lock */
-
- return;
- }
+ rel =3D process_single_relation(stmt, ¶ms);
+ if (rel =3D=3D NULL)
+ return; /* all done */
}
=20
+ /*
+ * Don't allow ANALYZE in the multiple-relation case for now. Maybe we
+ * can add support for this later.
+ */
+ if (params.options & CLUOPT_ANALYZE)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot %s multiple tables", "REPACK (ANALYZE)"));
+
/*
* By here, we know we are in a multi-table situation. In order to avoid
* holding locks for too long, we want to process each table in its own
* transaction. This forces us to disallow running inside a user
* transaction block.
*/
- PreventInTransactionBlock(isTopLevel, "CLUSTER");
+ PreventInTransactionBlock(isTopLevel, RepackCommandAsString(stmt->command=
));
=20
/* Also, we need a memory context to hold our list of relations */
- cluster_context =3D AllocSetContextCreate(PortalContext,
- "Cluster",
- ALLOCSET_DEFAULT_SIZES);
+ repack_context =3D AllocSetContextCreate(PortalContext,
+ "Repack",
+ ALLOCSET_DEFAULT_SIZES);
+
+ params.options |=3D CLUOPT_RECHECK;
=20
/*
- * Either we're processing a partitioned table, or we were not given any
- * table name at all. In either case, obtain a list of relations to
- * process.
- *
- * In the former case, an index name must have been given, so we don't
- * need to recheck its "indisclustered" bit, but we have to check that it
- * is an index that we can cluster on. In the latter case, we set the
- * option bit to have indisclustered verified.
- *
- * Rechecking the relation itself is necessary here in all cases.
+ * If we don't have a relation yet, determine a relation list. If we do,
+ * then it must be a partitioned table, and we want to process its
+ * partitions.
*/
- params.options |=3D CLUOPT_RECHECK;
- if (rel !=3D NULL)
+ if (rel =3D=3D NULL)
{
- Assert(rel->rd_rel->relkind =3D=3D RELKIND_PARTITIONED_TABLE);
- check_index_is_clusterable(rel, indexOid, AccessShareLock);
- rtcs =3D get_tables_to_cluster_partitioned(cluster_context, indexOid);
-
- /* close relation, releasing lock on parent table */
- table_close(rel, AccessExclusiveLock);
+ Assert(stmt->indexname =3D=3D NULL);
+ rtcs =3D get_tables_to_repack(stmt->command, stmt->usingindex,
+ repack_context);
+ params.options |=3D CLUOPT_RECHECK_ISCLUSTERED;
}
else
{
- rtcs =3D get_tables_to_cluster(cluster_context);
- params.options |=3D CLUOPT_RECHECK_ISCLUSTERED;
- }
+ Oid relid;
+ bool rel_is_index;
=20
- /* Do the job. */
- cluster_multiple_rels(rtcs, ¶ms);
+ Assert(rel->rd_rel->relkind =3D=3D RELKIND_PARTITIONED_TABLE);
=20
- /* Start a new transaction for the cleanup work. */
- StartTransactionCommand();
+ /*
+ * If USING INDEX was specified, resolve the index name now and pass
+ * it down.
+ */
+ if (stmt->usingindex)
+ {
+ /*
+ * If no index name was specified when repacking a partitioned
+ * table, punt for now. Maybe we can improve this later.
+ */
+ if (!stmt->indexname)
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("there is no previously clustered index for table \"%s\"",
+ RelationGetRelationName(rel)));
+
+ relid =3D determine_clustered_index(rel, stmt->usingindex,
+ stmt->indexname);
+ if (!OidIsValid(relid))
+ elog(ERROR, "unable to determine index to cluster on");
+ /* XXX is this the right place for this check? */
+ check_index_is_clusterable(rel, relid, AccessExclusiveLock);
+ rel_is_index =3D true;
+ }
+ else
+ {
+ relid =3D RelationGetRelid(rel);
+ rel_is_index =3D false;
+ }
=20
- /* Clean up working storage */
- MemoryContextDelete(cluster_context);
-}
+ rtcs =3D get_tables_to_repack_partitioned(stmt->command,
+ relid, rel_is_index,
+ repack_context);
=20
-/*
- * Given a list of relations to cluster, process each of them in a separate
- * transaction.
- *
- * We expect to be in a transaction at start, but there isn't one when we
- * return.
- */
-static void
-cluster_multiple_rels(List *rtcs, ClusterParams *params)
-{
- ListCell *lc;
+ /* close parent relation, releasing lock on it */
+ table_close(rel, AccessExclusiveLock);
+ rel =3D NULL;
+ }
=20
/* Commit to get out of starting transaction */
PopActiveSnapshot();
CommitTransactionCommand();
=20
/* Cluster the tables, each in a separate transaction */
- foreach(lc, rtcs)
+ Assert(rel =3D=3D NULL);
+ foreach_ptr(RelToCluster, rtc, rtcs)
{
- RelToCluster *rtc =3D (RelToCluster *) lfirst(lc);
- Relation rel;
-
/* Start a new transaction for each relation. */
StartTransactionCommand();
=20
+ /*
+ * Open the target table, coping with the case where it has been
+ * dropped.
+ */
+ rel =3D try_table_open(rtc->tableOid, AccessExclusiveLock);
+ if (rel =3D=3D NULL)
+ {
+ CommitTransactionCommand();
+ continue;
+ }
+
/* functions in indexes may want a snapshot set */
PushActiveSnapshot(GetTransactionSnapshot());
=20
- rel =3D table_open(rtc->tableOid, AccessExclusiveLock);
-
/* Process this table */
- cluster_rel(rel, rtc->indexOid, params);
+ cluster_rel(stmt->command, rel, rtc->indexOid, ¶ms);
/* cluster_rel closes the relation, but keeps lock */
=20
PopActiveSnapshot();
CommitTransactionCommand();
}
+
+ /* Start a new transaction for the cleanup work. */
+ StartTransactionCommand();
+
+ /* Clean up working storage */
+ MemoryContextDelete(repack_context);
}
=20
/*
@@ -304,11 +282,14 @@ cluster_multiple_rels(List *rtcs, ClusterParams *para=
ms)
* them incrementally while we load the table.
*
* If indexOid is InvalidOid, the table will be rewritten in physical order
- * instead of index order. This is the new implementation of VACUUM FULL,
- * and error messages should refer to the operation as VACUUM not CLUSTER.
+ * instead of index order.
+ *
+ * 'cmd' indicates which command is being executed, to be used for error
+ * messages.
*/
void
-cluster_rel(Relation OldHeap, Oid indexOid, ClusterParams *params)
+cluster_rel(RepackCommand cmd, Relation OldHeap, Oid indexOid,
+ ClusterParams *params)
{
Oid tableOid =3D RelationGetRelid(OldHeap);
Oid save_userid;
@@ -323,13 +304,8 @@ cluster_rel(Relation OldHeap, Oid indexOid, ClusterPar=
ams *params)
/* Check for user-requested abort. */
CHECK_FOR_INTERRUPTS();
=20
- pgstat_progress_start_command(PROGRESS_COMMAND_CLUSTER, tableOid);
- if (OidIsValid(indexOid))
- pgstat_progress_update_param(PROGRESS_CLUSTER_COMMAND,
- PROGRESS_CLUSTER_COMMAND_CLUSTER);
- else
- pgstat_progress_update_param(PROGRESS_CLUSTER_COMMAND,
- PROGRESS_CLUSTER_COMMAND_VACUUM_FULL);
+ pgstat_progress_start_command(PROGRESS_COMMAND_REPACK, tableOid);
+ pgstat_progress_update_param(PROGRESS_REPACK_COMMAND, cmd);
=20
/*
* Switch to the table owner's userid, so that any index functions are run
@@ -350,86 +326,38 @@ cluster_rel(Relation OldHeap, Oid indexOid, ClusterPa=
rams *params)
* *must* skip the one on indisclustered since it would reject an attempt
* to cluster a not-previously-clustered index.
*/
- if (recheck)
- {
- /* Check that the user still has privileges for the relation */
- if (!cluster_is_permitted_for_relation(tableOid, save_userid))
- {
- relation_close(OldHeap, AccessExclusiveLock);
- goto out;
- }
-
- /*
- * Silently skip a temp table for a remote session. Only doing this
- * check in the "recheck" case is appropriate (which currently means
- * somebody is executing a database-wide CLUSTER or on a partitioned
- * table), because there is another check in cluster() which will stop
- * any attempt to cluster remote temp tables by name. There is
- * another check in cluster_rel which is redundant, but we leave it
- * for extra safety.
- */
- if (RELATION_IS_OTHER_TEMP(OldHeap))
- {
- relation_close(OldHeap, AccessExclusiveLock);
- goto out;
- }
-
- if (OidIsValid(indexOid))
- {
- /*
- * Check that the index still exists
- */
- if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(indexOid)))
- {
- relation_close(OldHeap, AccessExclusiveLock);
- goto out;
- }
-
- /*
- * Check that the index is still the one with indisclustered set,
- * if needed.
- */
- if ((params->options & CLUOPT_RECHECK_ISCLUSTERED) !=3D 0 &&
- !get_index_isclustered(indexOid))
- {
- relation_close(OldHeap, AccessExclusiveLock);
- goto out;
- }
- }
- }
+ if (recheck &&
+ !cluster_rel_recheck(cmd, OldHeap, indexOid, save_userid,
+ params->options))
+ goto out;
=20
/*
- * We allow VACUUM FULL, but not CLUSTER, on shared catalogs. CLUSTER
- * would work in most respects, but the index would only get marked as
- * indisclustered in the current database, leading to unexpected behavior
- * if CLUSTER were later invoked in another database.
+ * We allow repacking shared catalogs only when not using an index. It
+ * would work to use an index in most respects, but the index would only
+ * get marked as indisclustered in the current database, leading to
+ * unexpected behavior if CLUSTER were later invoked in another database.
*/
if (OidIsValid(indexOid) && OldHeap->rd_rel->relisshared)
ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("cannot cluster a shared catalog")));
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot run %s on a shared catalog",
+ RepackCommandAsString(cmd)));
=20
/*
* Don't process temp tables of other backends ... their local buffer
* manager is not going to cope.
*/
if (RELATION_IS_OTHER_TEMP(OldHeap))
- {
- if (OidIsValid(indexOid))
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("cannot cluster temporary tables of other sessions")));
- else
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("cannot vacuum temporary tables of other sessions")));
- }
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot run %s on temporary tables of other sessions",
+ RepackCommandAsString(cmd)));
=20
/*
* Also check for active uses of the relation in the current transaction,
* including open scans and pending AFTER trigger events.
*/
- CheckTableNotInUse(OldHeap, OidIsValid(indexOid) ? "CLUSTER" : "VACUUM");
+ CheckTableNotInUse(OldHeap, RepackCommandAsString(cmd));
=20
/* Check heap and index are valid to cluster on */
if (OidIsValid(indexOid))
@@ -442,6 +370,24 @@ cluster_rel(Relation OldHeap, Oid indexOid, ClusterPar=
ams *params)
else
index =3D NULL;
=20
+ /*
+ * When allow_system_table_mods is turned off, we disallow repacking a
+ * catalog on a particular index unless that's already the clustered index
+ * for that catalog.
+ *
+ * XXX We don't check for this in CLUSTER, because it's historically been
+ * allowed.
+ */
+ if (cmd !=3D REPACK_COMMAND_CLUSTER &&
+ !allowSystemTableMods && OidIsValid(indexOid) &&
+ IsCatalogRelation(OldHeap) && !index->rd_index->indisclustered)
+ ereport(ERROR,
+ errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("permission denied: \"%s\" is a system catalog",
+ RelationGetRelationName(OldHeap)),
+ errdetail("System catalogs can only be clustered by the index they're =
already clustered on, if any, unless \"%s\" is enabled.",
+ "allow_system_table_mods"));
+
/*
* Quietly ignore the request if this is a materialized view which has not
* been populated from its query. No harm is done because there is no data
@@ -482,6 +428,63 @@ out:
pgstat_progress_end_command();
}
=20
+/*
+ * Check if the table (and its index) still meets the requirements of
+ * cluster_rel().
+ */
+static bool
+cluster_rel_recheck(RepackCommand cmd, Relation OldHeap, Oid indexOid,
+ Oid userid, int options)
+{
+ Oid tableOid =3D RelationGetRelid(OldHeap);
+
+ /* Check that the user still has privileges for the relation */
+ if (!cluster_is_permitted_for_relation(cmd, tableOid, userid))
+ {
+ relation_close(OldHeap, AccessExclusiveLock);
+ return false;
+ }
+
+ /*
+ * Silently skip a temp table for a remote session. Only doing this check
+ * in the "recheck" case is appropriate (which currently means somebody is
+ * executing a database-wide CLUSTER or on a partitioned table), because
+ * there is another check in cluster() which will stop any attempt to
+ * cluster remote temp tables by name. There is another check in
+ * cluster_rel which is redundant, but we leave it for extra safety.
+ */
+ if (RELATION_IS_OTHER_TEMP(OldHeap))
+ {
+ relation_close(OldHeap, AccessExclusiveLock);
+ return false;
+ }
+
+ if (OidIsValid(indexOid))
+ {
+ /*
+ * Check that the index still exists
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(indexOid)))
+ {
+ relation_close(OldHeap, AccessExclusiveLock);
+ return false;
+ }
+
+ /*
+ * Check that the index is still the one with indisclustered set, if
+ * needed.
+ */
+ if ((options & CLUOPT_RECHECK_ISCLUSTERED) !=3D 0 &&
+ !get_index_isclustered(indexOid))
+ {
+ relation_close(OldHeap, AccessExclusiveLock);
+ return false;
+ }
+ }
+
+ return true;
+}
+
/*
* Verify that the specified heap and index are valid to cluster on
*
@@ -642,8 +645,8 @@ rebuild_relation(Relation OldHeap, Relation index, bool=
verbose)
Assert(CheckRelationLockedByMe(OldHeap, AccessExclusiveLock, false) &&
(index =3D=3D NULL || CheckRelationLockedByMe(index, AccessExclusiveL=
ock, false)));
=20
- if (index)
- /* Mark the correct index as clustered */
+ /* for CLUSTER or REPACK USING INDEX, mark the index as the one to use */
+ if (index !=3D NULL)
mark_index_clustered(OldHeap, RelationGetRelid(index), true);
=20
/* Remember info about rel before closing OldHeap */
@@ -958,20 +961,20 @@ copy_table_data(Relation NewHeap, Relation OldHeap, R=
elation OldIndex, bool verb
/* Log what we're doing */
if (OldIndex !=3D NULL && !use_sort)
ereport(elevel,
- (errmsg("clustering \"%s.%s\" using index scan on \"%s\"",
- nspname,
- RelationGetRelationName(OldHeap),
- RelationGetRelationName(OldIndex))));
+ errmsg("repacking \"%s.%s\" using index scan on \"%s\"",
+ nspname,
+ RelationGetRelationName(OldHeap),
+ RelationGetRelationName(OldIndex)));
else if (use_sort)
ereport(elevel,
- (errmsg("clustering \"%s.%s\" using sequential scan and sort",
- nspname,
- RelationGetRelationName(OldHeap))));
+ errmsg("repacking \"%s.%s\" using sequential scan and sort",
+ nspname,
+ RelationGetRelationName(OldHeap)));
else
ereport(elevel,
- (errmsg("vacuuming \"%s.%s\"",
- nspname,
- RelationGetRelationName(OldHeap))));
+ errmsg("repacking \"%s.%s\" in physical order",
+ nspname,
+ RelationGetRelationName(OldHeap)));
=20
/*
* Hand off the actual copying to AM specific function, the generic code
@@ -1458,8 +1461,8 @@ finish_heap_swap(Oid OIDOldHeap, Oid OIDNewHeap,
int i;
=20
/* Report that we are now swapping relation files */
- pgstat_progress_update_param(PROGRESS_CLUSTER_PHASE,
- PROGRESS_CLUSTER_PHASE_SWAP_REL_FILES);
+ pgstat_progress_update_param(PROGRESS_REPACK_PHASE,
+ PROGRESS_REPACK_PHASE_SWAP_REL_FILES);
=20
/* Zero out possible results from swapped_relation_files */
memset(mapped_tables, 0, sizeof(mapped_tables));
@@ -1509,14 +1512,14 @@ finish_heap_swap(Oid OIDOldHeap, Oid OIDNewHeap,
reindex_flags |=3D REINDEX_REL_FORCE_INDEXES_PERMANENT;
=20
/* Report that we are now reindexing relations */
- pgstat_progress_update_param(PROGRESS_CLUSTER_PHASE,
- PROGRESS_CLUSTER_PHASE_REBUILD_INDEX);
+ pgstat_progress_update_param(PROGRESS_REPACK_PHASE,
+ PROGRESS_REPACK_PHASE_REBUILD_INDEX);
=20
reindex_relation(NULL, OIDOldHeap, reindex_flags, &reindex_params);
=20
/* Report that we are now doing clean up */
- pgstat_progress_update_param(PROGRESS_CLUSTER_PHASE,
- PROGRESS_CLUSTER_PHASE_FINAL_CLEANUP);
+ pgstat_progress_update_param(PROGRESS_REPACK_PHASE,
+ PROGRESS_REPACK_PHASE_FINAL_CLEANUP);
=20
/*
* If the relation being rebuilt is pg_class, swap_relation_files()
@@ -1632,106 +1635,191 @@ finish_heap_swap(Oid OIDOldHeap, Oid OIDNewHeap,
}
}
=20
-
/*
- * Get a list of tables that the current user has privileges on and
- * have indisclustered set. Return the list in a List * of RelToCluster
- * (stored in the specified memory context), each one giving the tableOid
- * and the indexOid on which the table is already clustered.
+ * Determine which relations to process, when REPACK/CLUSTER is called
+ * without specifying a table name. The exact process depends on whether
+ * USING INDEX was given or not, and in any case we only return tables and
+ * materialized views that the current user has privileges to repack/clust=
er.
+ *
+ * If USING INDEX was given, we scan pg_index to find those that have
+ * indisclustered set; if it was not given, scan pg_class and return all
+ * tables.
+ *
+ * Return it as a list of RelToCluster in the given memory context.
*/
static List *
-get_tables_to_cluster(MemoryContext cluster_context)
+get_tables_to_repack(RepackCommand cmd, bool usingindex, MemoryContext per=
mcxt)
{
- Relation indRelation;
+ Relation catalog;
TableScanDesc scan;
- ScanKeyData entry;
- HeapTuple indexTuple;
- Form_pg_index index;
- MemoryContext old_context;
+ HeapTuple tuple;
List *rtcs =3D NIL;
=20
- /*
- * Get all indexes that have indisclustered set and that the current user
- * has the appropriate privileges for.
- */
- indRelation =3D table_open(IndexRelationId, AccessShareLock);
- ScanKeyInit(&entry,
- Anum_pg_index_indisclustered,
- BTEqualStrategyNumber, F_BOOLEQ,
- BoolGetDatum(true));
- scan =3D table_beginscan_catalog(indRelation, 1, &entry);
- while ((indexTuple =3D heap_getnext(scan, ForwardScanDirection)) !=3D NUL=
L)
+ if (usingindex)
{
- RelToCluster *rtc;
+ ScanKeyData entry;
+
+ catalog =3D table_open(IndexRelationId, AccessShareLock);
+ ScanKeyInit(&entry,
+ Anum_pg_index_indisclustered,
+ BTEqualStrategyNumber, F_BOOLEQ,
+ BoolGetDatum(true));
+ scan =3D table_beginscan_catalog(catalog, 1, &entry);
+ while ((tuple =3D heap_getnext(scan, ForwardScanDirection)) !=3D NULL)
+ {
+ RelToCluster *rtc;
+ Form_pg_index index;
+ MemoryContext oldcxt;
=20
- index =3D (Form_pg_index) GETSTRUCT(indexTuple);
+ index =3D (Form_pg_index) GETSTRUCT(tuple);
=20
- if (!cluster_is_permitted_for_relation(index->indrelid, GetUserId()))
- continue;
+ /*
+ * Try to obtain a light lock on the index's table, to ensure it
+ * doesn't go away while we collect the list. If we cannot, just
+ * disregard it.
+ */
+ if (!ConditionalLockRelationOid(index->indrelid, AccessShareLock))
+ continue;
=20
- /* Use a permanent memory context for the result list */
- old_context =3D MemoryContextSwitchTo(cluster_context);
+ /* Verify that the table still exists */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(index->indrelid)))
+ {
+ /* Release useless lock */
+ UnlockRelationOid(index->indrelid, AccessShareLock);
+ continue;
+ }
=20
- rtc =3D palloc_object(RelToCluster);
- rtc->tableOid =3D index->indrelid;
- rtc->indexOid =3D index->indexrelid;
- rtcs =3D lappend(rtcs, rtc);
+ if (!cluster_is_permitted_for_relation(cmd, index->indrelid,
+ GetUserId()))
+ continue;
+
+ /* Use a permanent memory context for the result list */
+ oldcxt =3D MemoryContextSwitchTo(permcxt);
+ rtc =3D palloc_object(RelToCluster);
+ rtc->tableOid =3D index->indrelid;
+ rtc->indexOid =3D index->indexrelid;
+ rtcs =3D lappend(rtcs, rtc);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ }
+ else
+ {
+ catalog =3D table_open(RelationRelationId, AccessShareLock);
+ scan =3D table_beginscan_catalog(catalog, 0, NULL);
+
+ while ((tuple =3D heap_getnext(scan, ForwardScanDirection)) !=3D NULL)
+ {
+ RelToCluster *rtc;
+ Form_pg_class class;
+ MemoryContext oldcxt;
+
+ class =3D (Form_pg_class) GETSTRUCT(tuple);
+
+ /*
+ * Try to obtain a light lock on the table, to ensure it doesn't
+ * go away while we collect the list. If we cannot, just
+ * disregard the table.
+ */
+ if (!ConditionalLockRelationOid(class->oid, AccessShareLock))
+ continue;
=20
- MemoryContextSwitchTo(old_context);
+ /* Verify that the table still exists */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(class->oid)))
+ {
+ /* Release useless lock */
+ UnlockRelationOid(class->oid, AccessShareLock);
+ continue;
+ }
+
+ /* Can only process plain tables and matviews */
+ if (class->relkind !=3D RELKIND_RELATION &&
+ class->relkind !=3D RELKIND_MATVIEW)
+ continue;
+
+ /* noisily skip rels which the user can't process */
+ if (!cluster_is_permitted_for_relation(cmd, class->oid,
+ GetUserId()))
+ continue;
+
+ /* Use a permanent memory context for the result list */
+ oldcxt =3D MemoryContextSwitchTo(permcxt);
+ rtc =3D palloc_object(RelToCluster);
+ rtc->tableOid =3D class->oid;
+ rtc->indexOid =3D InvalidOid;
+ rtcs =3D lappend(rtcs, rtc);
+ MemoryContextSwitchTo(oldcxt);
+ }
}
- table_endscan(scan);
=20
- relation_close(indRelation, AccessShareLock);
+ table_endscan(scan);
+ relation_close(catalog, AccessShareLock);
=20
return rtcs;
}
=20
/*
- * Given an index on a partitioned table, return a list of RelToCluster for
+ * Given a partitioned table or its index, return a list of RelToCluster f=
or
* all the children leaves tables/indexes.
*
* Like expand_vacuum_rel, but here caller must hold AccessExclusiveLock
* on the table containing the index.
+ *
+ * 'rel_is_index' tells whether 'relid' is that of an index (true) or of t=
he
+ * owning relation.
*/
static List *
-get_tables_to_cluster_partitioned(MemoryContext cluster_context, Oid index=
Oid)
+get_tables_to_repack_partitioned(RepackCommand cmd, Oid relid,
+ bool rel_is_index, MemoryContext permcxt)
{
List *inhoids;
- ListCell *lc;
List *rtcs =3D NIL;
- MemoryContext old_context;
-
- /* Do not lock the children until they're processed */
- inhoids =3D find_all_inheritors(indexOid, NoLock, NULL);
=20
- foreach(lc, inhoids)
+ /*
+ * Do not lock the children until they're processed. Note that we do hold
+ * a lock on the parent partitioned table.
+ */
+ inhoids =3D find_all_inheritors(relid, NoLock, NULL);
+ foreach_oid(child_oid, inhoids)
{
- Oid indexrelid =3D lfirst_oid(lc);
- Oid relid =3D IndexGetRelation(indexrelid, false);
+ Oid table_oid,
+ index_oid;
RelToCluster *rtc;
+ MemoryContext oldcxt;
=20
- /* consider only leaf indexes */
- if (get_rel_relkind(indexrelid) !=3D RELKIND_INDEX)
- continue;
+ if (rel_is_index)
+ {
+ /* consider only leaf indexes */
+ if (get_rel_relkind(child_oid) !=3D RELKIND_INDEX)
+ continue;
+
+ table_oid =3D IndexGetRelation(child_oid, false);
+ index_oid =3D child_oid;
+ }
+ else
+ {
+ /* consider only leaf relations */
+ if (get_rel_relkind(child_oid) !=3D RELKIND_RELATION)
+ continue;
+
+ table_oid =3D child_oid;
+ index_oid =3D InvalidOid;
+ }
=20
/*
* It's possible that the user does not have privileges to CLUSTER the
- * leaf partition despite having such privileges on the partitioned
- * table. We skip any partitions which the user is not permitted to
- * CLUSTER.
+ * leaf partition despite having them on the partitioned table. Skip
+ * if so.
*/
- if (!cluster_is_permitted_for_relation(relid, GetUserId()))
+ if (!cluster_is_permitted_for_relation(cmd, table_oid, GetUserId()))
continue;
=20
/* Use a permanent memory context for the result list */
- old_context =3D MemoryContextSwitchTo(cluster_context);
-
+ oldcxt =3D MemoryContextSwitchTo(permcxt);
rtc =3D palloc_object(RelToCluster);
- rtc->tableOid =3D relid;
- rtc->indexOid =3D indexrelid;
+ rtc->tableOid =3D table_oid;
+ rtc->indexOid =3D index_oid;
rtcs =3D lappend(rtcs, rtc);
-
- MemoryContextSwitchTo(old_context);
+ MemoryContextSwitchTo(oldcxt);
}
=20
return rtcs;
@@ -1742,13 +1830,167 @@ get_tables_to_cluster_partitioned(MemoryContext cl=
uster_context, Oid indexOid)
* function emits a WARNING.
*/
static bool
-cluster_is_permitted_for_relation(Oid relid, Oid userid)
+cluster_is_permitted_for_relation(RepackCommand cmd, Oid relid, Oid userid)
{
+ Assert(cmd =3D=3D REPACK_COMMAND_CLUSTER || cmd =3D=3D REPACK_COMMAND_REP=
ACK);
+
if (pg_class_aclcheck(relid, userid, ACL_MAINTAIN) =3D=3D ACLCHECK_OK)
return true;
=20
ereport(WARNING,
- (errmsg("permission denied to cluster \"%s\", skipping it",
- get_rel_name(relid))));
+ errmsg("permission denied to execute %s on \"%s\", skipping it",
+ RepackCommandAsString(cmd),
+ get_rel_name(relid)));
+
return false;
}
+
+
+/*
+ * Given a RepackStmt with an indicated relation name, resolve the relation
+ * name, obtain lock on it, then determine what to do based on the relation
+ * type: if it's table and not partitioned, repack it as indicated (using =
an
+ * existing clustered index, or following the given one), and return NULL.
+ *
+ * On the other hand, if the table is partitioned, do nothing further and
+ * instead return the opened and locked relcache entry, so that caller can
+ * process the partitions using the multiple-table handling code. In this
+ * case, if an index name is given, it's up to the caller to resolve it.
+ */
+static Relation
+process_single_relation(RepackStmt *stmt, ClusterParams *params)
+{
+ Relation rel;
+ Oid tableOid;
+
+ Assert(stmt->relation !=3D NULL);
+ Assert(stmt->command =3D=3D REPACK_COMMAND_CLUSTER ||
+ stmt->command =3D=3D REPACK_COMMAND_REPACK);
+
+ /*
+ * Find, lock, and check permissions on the table. We obtain
+ * AccessExclusiveLock right away to avoid lock-upgrade hazard in the
+ * single-transaction case.
+ */
+ tableOid =3D RangeVarGetRelidExtended(stmt->relation->relation,
+ AccessExclusiveLock,
+ 0,
+ RangeVarCallbackMaintainsTable,
+ NULL);
+ rel =3D table_open(tableOid, NoLock);
+
+ /*
+ * Reject clustering a remote temp table ... their local buffer manager is
+ * not going to cope.
+ */
+ if (RELATION_IS_OTHER_TEMP(rel))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot execute %s on temporary tables of other sessions",
+ RepackCommandAsString(stmt->command)));
+
+ /*
+ * Make sure ANALYZE is specified if a column list is present.
+ */
+ if ((params->options & CLUOPT_ANALYZE) =3D=3D 0 && stmt->relation->va_col=
s !=3D NIL)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("ANALYZE option must be specified when a column list is provide=
d"));
+
+ /*
+ * For partitioned tables, let caller handle this. Otherwise, process it
+ * here and we're done.
+ */
+ if (rel->rd_rel->relkind =3D=3D RELKIND_PARTITIONED_TABLE)
+ return rel;
+ else
+ {
+ Oid indexOid;
+
+ indexOid =3D determine_clustered_index(rel, stmt->usingindex,
+ stmt->indexname);
+ if (OidIsValid(indexOid))
+ check_index_is_clusterable(rel, indexOid, AccessExclusiveLock);
+ cluster_rel(stmt->command, rel, indexOid, params);
+
+ /* Do an analyze, if requested */
+ if (params->options & CLUOPT_ANALYZE)
+ {
+ VacuumParams vac_params =3D {0};
+
+ vac_params.options |=3D VACOPT_ANALYZE;
+ if (params->options & CLUOPT_VERBOSE)
+ vac_params.options |=3D VACOPT_VERBOSE;
+ analyze_rel(tableOid, NULL, vac_params,
+ stmt->relation->va_cols, true, NULL);
+ }
+
+ return NULL;
+ }
+}
+
+/*
+ * Given a relation and the usingindex/indexname options in a
+ * REPACK USING INDEX or CLUSTER command, return the OID of the
+ * index to use for clustering the table.
+ *
+ * Caller must hold lock on the relation so that the set of indexes
+ * doesn't change, and must call check_index_is_clusterable.
+ */
+static Oid
+determine_clustered_index(Relation rel, bool usingindex, const char *index=
name)
+{
+ Oid indexOid;
+
+ if (indexname =3D=3D NULL && usingindex)
+ {
+ /*
+ * If USING INDEX with no name is given, find a clustered index, or
+ * error out if none.
+ */
+ indexOid =3D InvalidOid;
+ foreach_oid(idxoid, RelationGetIndexList(rel))
+ {
+ if (get_index_isclustered(idxoid))
+ {
+ indexOid =3D idxoid;
+ break;
+ }
+ }
+
+ if (!OidIsValid(indexOid))
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("there is no previously clustered index for table \"%s\"",
+ RelationGetRelationName(rel)));
+ }
+ else if (indexname !=3D NULL)
+ {
+ /* An index was specified; obtain its OID. */
+ indexOid =3D get_relname_relid(indexname, rel->rd_rel->relnamespace);
+ if (!OidIsValid(indexOid))
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("index \"%s\" for table \"%s\" does not exist",
+ indexname, RelationGetRelationName(rel)));
+ }
+ else
+ indexOid =3D InvalidOid;
+
+ return indexOid;
+}
+
+static const char *
+RepackCommandAsString(RepackCommand cmd)
+{
+ switch (cmd)
+ {
+ case REPACK_COMMAND_REPACK:
+ return "REPACK";
+ case REPACK_COMMAND_VACUUMFULL:
+ return "VACUUM";
+ case REPACK_COMMAND_CLUSTER:
+ return "CLUSTER";
+ }
+ return "???"; /* keep compiler quiet */
+}
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 03932f45c8a..aea998260e1 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -351,7 +351,6 @@ ExecVacuum(ParseState *pstate, VacuumStmt *vacstmt, boo=
l isTopLevel)
}
}
=20
-
/*
* Sanity check DISABLE_PAGE_SKIPPING option.
*/
@@ -2289,8 +2288,9 @@ vacuum_rel(Oid relid, RangeVar *relation, VacuumParam=
s params,
if ((params.options & VACOPT_VERBOSE) !=3D 0)
cluster_params.options |=3D CLUOPT_VERBOSE;
=20
- /* VACUUM FULL is now a variant of CLUSTER; see cluster.c */
- cluster_rel(rel, InvalidOid, &cluster_params);
+ /* VACUUM FULL is a variant of REPACK; see cluster.c */
+ cluster_rel(REPACK_COMMAND_VACUUMFULL, rel, InvalidOid,
+ &cluster_params);
/* cluster_rel closes the relation, but keeps lock */
=20
rel =3D NULL;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 713ee5c10a2..54d37c10447 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -287,7 +287,7 @@ static Node *makeRecursiveViewSelect(char *relname, Lis=
t *aliases, Node *query);
AlterCompositeTypeStmt AlterUserMappingStmt
AlterRoleStmt AlterRoleSetStmt AlterPolicyStmt AlterStatsStmt
AlterDefaultPrivilegesStmt DefACLAction
- AnalyzeStmt CallStmt ClosePortalStmt ClusterStmt CommentStmt
+ AnalyzeStmt CallStmt ClosePortalStmt CommentStmt
ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt
CreateDomainStmt CreateExtensionStmt CreateGroupStmt CreateOpClassStmt
CreateOpFamilyStmt AlterOpFamilyStmt CreatePLangStmt
@@ -304,7 +304,7 @@ static Node *makeRecursiveViewSelect(char *relname, Lis=
t *aliases, Node *query);
GrantStmt GrantRoleStmt ImportForeignSchemaStmt IndexStmt InsertStmt
ListenStmt LoadStmt LockStmt MergeStmt NotifyStmt ExplainableStmt Prepar=
ableStmt
CreateFunctionStmt AlterFunctionStmt ReindexStmt RemoveAggrStmt
- RemoveFuncStmt RemoveOperStmt RenameStmt ReturnStmt RevokeStmt RevokeRol=
eStmt
+ RemoveFuncStmt RemoveOperStmt RenameStmt RepackStmt ReturnStmt RevokeStm=
t RevokeRoleStmt
RuleActionStmt RuleActionStmtOrEmpty RuleStmt
SecLabelStmt SelectStmt TransactionStmt TransactionStmtLegacy TruncateSt=
mt
UnlistenStmt UpdateStmt VacuumStmt
@@ -323,7 +323,7 @@ static Node *makeRecursiveViewSelect(char *relname, Lis=
t *aliases, Node *query);
=20
%type <str> opt_single_name
%type <list> opt_qualified_name
-%type <boolean> opt_concurrently
+%type <boolean> opt_concurrently opt_usingindex
%type <dbehavior> opt_drop_behavior
%type <list> opt_utility_option_list
%type <list> opt_wait_with_clause
@@ -773,7 +773,7 @@ static Node *makeRecursiveViewSelect(char *relname, Lis=
t *aliases, Node *query);
QUOTE QUOTES
=20
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
- REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
+ REFRESH REINDEX RELATIVE_P RELEASE RENAME REPACK REPEATABLE REPLACE REPLI=
CA
RESET RESPECT_P RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT RO=
LE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
=20
@@ -1035,7 +1035,6 @@ stmt:
| CallStmt
| CheckPointStmt
| ClosePortalStmt
- | ClusterStmt
| CommentStmt
| ConstraintsSetStmt
| CopyStmt
@@ -1109,6 +1108,7 @@ stmt:
| RemoveFuncStmt
| RemoveOperStmt
| RenameStmt
+ | RepackStmt
| RevokeStmt
| RevokeRoleStmt
| RuleStmt
@@ -1146,6 +1146,11 @@ opt_concurrently:
| /*EMPTY*/ { $$ =3D false; }
;
=20
+opt_usingindex:
+ USING INDEX { $$ =3D true; }
+ | /* EMPTY */ { $$ =3D false; }
+ ;
+
opt_drop_behavior:
CASCADE { $$ =3D DROP_CASCADE; }
| RESTRICT { $$ =3D DROP_RESTRICT; }
@@ -12036,38 +12041,82 @@ CreateConversionStmt:
/*************************************************************************=
****
*
* QUERY:
+ * REPACK [ (options) ] [ <qualified_name> [ <name_list> ] [ USING INDE=
X <index_name> ] ]
+ *
+ * obsolete variants:
* CLUSTER (options) [ <qualified_name> [ USING <index_name> ] ]
* CLUSTER [VERBOSE] [ <qualified_name> [ USING <index_name> ] ]
* CLUSTER [VERBOSE] <index_name> ON <qualified_name> (for pre-8.3)
*
*************************************************************************=
****/
=20
-ClusterStmt:
- CLUSTER '(' utility_option_list ')' qualified_name cluster_index_specif=
ication
+RepackStmt:
+ REPACK opt_utility_option_list vacuum_relation USING INDEX name
{
- ClusterStmt *n =3D makeNode(ClusterStmt);
+ RepackStmt *n =3D makeNode(RepackStmt);
=20
- n->relation =3D $5;
+ n->command =3D REPACK_COMMAND_REPACK;
+ n->relation =3D (VacuumRelation *) $3;
n->indexname =3D $6;
+ n->usingindex =3D true;
+ n->params =3D $2;
+ $$ =3D (Node *) n;
+ }
+ | REPACK opt_utility_option_list vacuum_relation opt_usingindex
+ {
+ RepackStmt *n =3D makeNode(RepackStmt);
+
+ n->command =3D REPACK_COMMAND_REPACK;
+ n->relation =3D (VacuumRelation *) $3;
+ n->indexname =3D NULL;
+ n->usingindex =3D $4;
+ n->params =3D $2;
+ $$ =3D (Node *) n;
+ }
+ | REPACK opt_utility_option_list opt_usingindex
+ {
+ RepackStmt *n =3D makeNode(RepackStmt);
+
+ n->command =3D REPACK_COMMAND_REPACK;
+ n->relation =3D NULL;
+ n->indexname =3D NULL;
+ n->usingindex =3D $3;
+ n->params =3D $2;
+ $$ =3D (Node *) n;
+ }
+ | CLUSTER '(' utility_option_list ')' qualified_name cluster_index_spec=
ification
+ {
+ RepackStmt *n =3D makeNode(RepackStmt);
+
+ n->command =3D REPACK_COMMAND_CLUSTER;
+ n->relation =3D makeNode(VacuumRelation);
+ n->relation->relation =3D $5;
+ n->indexname =3D $6;
+ n->usingindex =3D true;
n->params =3D $3;
$$ =3D (Node *) n;
}
| CLUSTER opt_utility_option_list
{
- ClusterStmt *n =3D makeNode(ClusterStmt);
+ RepackStmt *n =3D makeNode(RepackStmt);
=20
+ n->command =3D REPACK_COMMAND_CLUSTER;
n->relation =3D NULL;
n->indexname =3D NULL;
+ n->usingindex =3D true;
n->params =3D $2;
$$ =3D (Node *) n;
}
/* unparenthesized VERBOSE kept for pre-14 compatibility */
| CLUSTER opt_verbose qualified_name cluster_index_specification
{
- ClusterStmt *n =3D makeNode(ClusterStmt);
+ RepackStmt *n =3D makeNode(RepackStmt);
=20
- n->relation =3D $3;
+ n->command =3D REPACK_COMMAND_CLUSTER;
+ n->relation =3D makeNode(VacuumRelation);
+ n->relation->relation =3D $3;
n->indexname =3D $4;
+ n->usingindex =3D true;
if ($2)
n->params =3D list_make1(makeDefElem("verbose", NULL, @2));
$$ =3D (Node *) n;
@@ -12075,20 +12124,25 @@ ClusterStmt:
/* unparenthesized VERBOSE kept for pre-17 compatibility */
| CLUSTER VERBOSE
{
- ClusterStmt *n =3D makeNode(ClusterStmt);
+ RepackStmt *n =3D makeNode(RepackStmt);
=20
+ n->command =3D REPACK_COMMAND_CLUSTER;
n->relation =3D NULL;
n->indexname =3D NULL;
+ n->usingindex =3D true;
n->params =3D list_make1(makeDefElem("verbose", NULL, @2));
$$ =3D (Node *) n;
}
/* kept for pre-8.3 compatibility */
| CLUSTER opt_verbose name ON qualified_name
{
- ClusterStmt *n =3D makeNode(ClusterStmt);
+ RepackStmt *n =3D makeNode(RepackStmt);
=20
- n->relation =3D $5;
+ n->command =3D REPACK_COMMAND_CLUSTER;
+ n->relation =3D makeNode(VacuumRelation);
+ n->relation->relation =3D $5;
n->indexname =3D $3;
+ n->usingindex =3D true;
if ($2)
n->params =3D list_make1(makeDefElem("verbose", NULL, @2));
$$ =3D (Node *) n;
@@ -18127,6 +18181,7 @@ unreserved_keyword:
| RELATIVE_P
| RELEASE
| RENAME
+ | REPACK
| REPEATABLE
| REPLACE
| REPLICA
@@ -18764,6 +18819,7 @@ bare_label_keyword:
| RELATIVE_P
| RELEASE
| RENAME
+ | REPACK
| REPEATABLE
| REPLACE
| REPLICA
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index 34dd6e18df5..ca737b05115 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -279,9 +279,9 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree)
return COMMAND_OK_IN_RECOVERY | COMMAND_OK_IN_READ_ONLY_TXN;
}
=20
- case T_ClusterStmt:
case T_ReindexStmt:
case T_VacuumStmt:
+ case T_RepackStmt:
{
/*
* These commands write WAL, so they're not strictly
@@ -856,14 +856,14 @@ standard_ProcessUtility(PlannedStmt *pstmt,
ExecuteCallStmt(castNode(CallStmt, parsetree), params, isAtomicContext,=
dest);
break;
=20
- case T_ClusterStmt:
- cluster(pstate, (ClusterStmt *) parsetree, isTopLevel);
- break;
-
case T_VacuumStmt:
ExecVacuum(pstate, (VacuumStmt *) parsetree, isTopLevel);
break;
=20
+ case T_RepackStmt:
+ ExecRepack(pstate, (RepackStmt *) parsetree, isTopLevel);
+ break;
+
case T_ExplainStmt:
ExplainQuery(pstate, (ExplainStmt *) parsetree, params, dest);
break;
@@ -2865,10 +2865,6 @@ CreateCommandTag(Node *parsetree)
tag =3D CMDTAG_CALL;
break;
=20
- case T_ClusterStmt:
- tag =3D CMDTAG_CLUSTER;
- break;
-
case T_VacuumStmt:
if (((VacuumStmt *) parsetree)->is_vacuumcmd)
tag =3D CMDTAG_VACUUM;
@@ -2876,6 +2872,13 @@ CreateCommandTag(Node *parsetree)
tag =3D CMDTAG_ANALYZE;
break;
=20
+ case T_RepackStmt:
+ if (((RepackStmt *) parsetree)->command =3D=3D REPACK_COMMAND_CLUSTER)
+ tag =3D CMDTAG_CLUSTER;
+ else
+ tag =3D CMDTAG_REPACK;
+ break;
+
case T_ExplainStmt:
tag =3D CMDTAG_EXPLAIN;
break;
@@ -3517,7 +3520,7 @@ GetCommandLogLevel(Node *parsetree)
lev =3D LOGSTMT_ALL;
break;
=20
- case T_ClusterStmt:
+ case T_RepackStmt:
lev =3D LOGSTMT_DDL;
break;
=20
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pg=
statfuncs.c
index 73ca0bb0b7f..55a69bf681d 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -287,8 +287,8 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
cmdtype =3D PROGRESS_COMMAND_VACUUM;
else if (pg_strcasecmp(cmd, "ANALYZE") =3D=3D 0)
cmdtype =3D PROGRESS_COMMAND_ANALYZE;
- else if (pg_strcasecmp(cmd, "CLUSTER") =3D=3D 0)
- cmdtype =3D PROGRESS_COMMAND_CLUSTER;
+ else if (pg_strcasecmp(cmd, "REPACK") =3D=3D 0)
+ cmdtype =3D PROGRESS_COMMAND_REPACK;
else if (pg_strcasecmp(cmd, "CREATE INDEX") =3D=3D 0)
cmdtype =3D PROGRESS_COMMAND_CREATE_INDEX;
else if (pg_strcasecmp(cmd, "BASEBACKUP") =3D=3D 0)
diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c
index 8b91bc00062..2a1bb47ff03 100644
--- a/src/bin/psql/tab-complete.in.c
+++ b/src/bin/psql/tab-complete.in.c
@@ -1267,7 +1267,7 @@ static const char *const sql_commands[] =3D {
"DELETE FROM", "DISCARD", "DO", "DROP", "END", "EXECUTE", "EXPLAIN",
"FETCH", "GRANT", "IMPORT FOREIGN SCHEMA", "INSERT INTO", "LISTEN", "LOAD=
", "LOCK",
"MERGE INTO", "MOVE", "NOTIFY", "PREPARE",
- "REASSIGN", "REFRESH MATERIALIZED VIEW", "REINDEX", "RELEASE",
+ "REASSIGN", "REFRESH MATERIALIZED VIEW", "REINDEX", "RELEASE", "REPACK",
"RESET", "REVOKE", "ROLLBACK",
"SAVEPOINT", "SECURITY LABEL", "SELECT", "SET", "SHOW", "START",
"TABLE", "TRUNCATE", "UNLISTEN", "UPDATE", "VACUUM", "VALUES",
@@ -5086,6 +5086,46 @@ match_previous_words(int pattern_id,
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
=20
+/* REPACK */
+ else if (Matches("REPACK"))
+ COMPLETE_WITH_SCHEMA_QUERY_PLUS(Query_for_list_of_clusterables,
+ "(", "USING INDEX");
+ else if (Matches("REPACK", "(*)"))
+ COMPLETE_WITH_SCHEMA_QUERY_PLUS(Query_for_list_of_clusterables,
+ "USING INDEX");
+ else if (Matches("REPACK", MatchAnyExcept("(")))
+ COMPLETE_WITH("USING INDEX");
+ else if (Matches("REPACK", "(*)", MatchAnyExcept("(")))
+ COMPLETE_WITH("USING INDEX");
+ else if (Matches("REPACK", MatchAny, "USING", "INDEX") ||
+ Matches("REPACK", "(*)", MatchAny, "USING", "INDEX"))
+ {
+ set_completion_reference(prev3_wd);
+ COMPLETE_WITH_SCHEMA_QUERY(Query_for_index_of_table);
+ }
+ /*
+ * Complete ... [ (*) ] <sth> USING INDEX, with a list of indexes for
+ * <sth>.
+ */
+ else if (TailMatches(MatchAny, "USING", "INDEX"))
+ {
+ set_completion_reference(prev3_wd);
+ COMPLETE_WITH_SCHEMA_QUERY(Query_for_index_of_table);
+ }
+ else if (HeadMatches("REPACK", "(*") &&
+ !HeadMatches("REPACK", "(*)"))
+ {
+ /*
+ * This fires if we're in an unfinished parenthesized option list.
+ * get_previous_words treats a completed parenthesized option list as
+ * one word, so the above test is correct.
+ */
+ if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
+ COMPLETE_WITH("ANALYZE", "VERBOSE");
+ else if (TailMatches("ANALYZE", "VERBOSE"))
+ COMPLETE_WITH("ON", "OFF");
+ }
+
/* SECURITY LABEL */
else if (Matches("SECURITY"))
COMPLETE_WITH("LABEL");
diff --git a/src/bin/scripts/Makefile b/src/bin/scripts/Makefile
index e6cd9ef4af5..f964a5803ca 100644
--- a/src/bin/scripts/Makefile
+++ b/src/bin/scripts/Makefile
@@ -23,6 +23,7 @@ PROGRAMS =3D \
dropdb \
dropuser \
pg_isready \
+ pg_repackdb \
reindexdb \
vacuumdb
=20
@@ -39,6 +40,7 @@ clusterdb: clusterdb.o common.o $(WIN32RES) | submake-lib=
pq submake-libpgport su
vacuumdb: vacuumdb.o vacuuming.o common.o $(WIN32RES) | submake-libpq subm=
ake-libpgport submake-libpgfeutils
reindexdb: reindexdb.o common.o $(WIN32RES) | submake-libpq submake-libpgp=
ort submake-libpgfeutils
pg_isready: pg_isready.o common.o $(WIN32RES) | submake-libpq submake-libp=
gport submake-libpgfeutils
+pg_repackdb: pg_repackdb.o vacuuming.o common.o $(WIN32RES) | submake-libp=
q submake-libpgport submake-libpgfeutils
=20
install: all installdirs
$(INSTALL_PROGRAM) createdb$(X) '$(DESTDIR)$(bindir)'/createdb$(X)
@@ -49,6 +51,7 @@ install: all installdirs
$(INSTALL_PROGRAM) vacuumdb$(X) '$(DESTDIR)$(bindir)'/vacuumdb$(X)
$(INSTALL_PROGRAM) reindexdb$(X) '$(DESTDIR)$(bindir)'/reindexdb$(X)
$(INSTALL_PROGRAM) pg_isready$(X) '$(DESTDIR)$(bindir)'/pg_isready$(X)
+ $(INSTALL_PROGRAM) pg_repackdb$(X) '$(DESTDIR)$(bindir)'/pg_repackdb$(X)
=20
installdirs:
$(MKDIR_P) '$(DESTDIR)$(bindir)'
diff --git a/src/bin/scripts/meson.build b/src/bin/scripts/meson.build
index c083ec38099..1e88bacebba 100644
--- a/src/bin/scripts/meson.build
+++ b/src/bin/scripts/meson.build
@@ -42,6 +42,7 @@ vacuuming_common =3D static_library('libvacuuming_common',
=20
binaries =3D [
'vacuumdb',
+ 'pg_repackdb',
]
foreach binary : binaries
binary_sources =3D files('@[email protected]'.format(binary))
@@ -80,6 +81,7 @@ tests +=3D {
't/100_vacuumdb.pl',
't/101_vacuumdb_all.pl',
't/102_vacuumdb_stages.pl',
+ 't/103_repackdb.pl',
't/200_connstr.pl',
],
},
diff --git a/src/bin/scripts/pg_repackdb.c b/src/bin/scripts/pg_repackdb.c
new file mode 100644
index 00000000000..2765d1e97b8
--- /dev/null
+++ b/src/bin/scripts/pg_repackdb.c
@@ -0,0 +1,240 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_repackdb
+ * An utility to run REPACK
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * FIXME: this is missing a way to specify the index to use to repack one
+ * table, or whether to pass a WITH INDEX clause when multiple tables are
+ * used. Something like --index[=3Dindexname]. Adding that bleeds into
+ * vacuuming.c as well.
+ *
+ * src/bin/scripts/pg_repackdb.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include <limits.h>
+
+#include "common.h"
+#include "common/logging.h"
+#include "fe_utils/option_utils.h"
+#include "vacuuming.h"
+
+static void help(const char *progname);
+static void check_objfilter(bits32 objfilter);
+
+int
+main(int argc, char *argv[])
+{
+ static struct option long_options[] =3D {
+ {"host", required_argument, NULL, 'h'},
+ {"port", required_argument, NULL, 'p'},
+ {"username", required_argument, NULL, 'U'},
+ {"no-password", no_argument, NULL, 'w'},
+ {"password", no_argument, NULL, 'W'},
+ {"echo", no_argument, NULL, 'e'},
+ {"quiet", no_argument, NULL, 'q'},
+ {"dbname", required_argument, NULL, 'd'},
+ {"analyze", no_argument, NULL, 'z'},
+ {"all", no_argument, NULL, 'a'},
+ /* XXX this could be 'i', but optional_arg is messy */
+ {"index", optional_argument, NULL, 1},
+ {"table", required_argument, NULL, 't'},
+ {"verbose", no_argument, NULL, 'v'},
+ {"jobs", required_argument, NULL, 'j'},
+ {"schema", required_argument, NULL, 'n'},
+ {"exclude-schema", required_argument, NULL, 'N'},
+ {"maintenance-db", required_argument, NULL, 2},
+ {NULL, 0, NULL, 0}
+ };
+
+ const char *progname;
+ int optindex;
+ int c;
+ const char *dbname =3D NULL;
+ const char *maintenance_db =3D NULL;
+ ConnParams cparams;
+ vacuumingOptions vacopts;
+ SimpleStringList objects =3D {NULL, NULL};
+ int concurrentCons =3D 1;
+ int tbl_count =3D 0;
+ int ret;
+
+ /* initialize options */
+ memset(&vacopts, 0, sizeof(vacopts));
+ vacopts.mode =3D MODE_REPACK;
+
+ /* the same for connection parameters */
+ memset(&cparams, 0, sizeof(cparams));
+ cparams.prompt_password =3D TRI_DEFAULT;
+
+ pg_logging_init(argv[0]);
+ progname =3D get_progname(argv[0]);
+ set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pgscripts"));
+
+ handle_help_version_opts(argc, argv, progname, help);
+
+ while ((c =3D getopt_long(argc, argv, "ad:eh:j:n:N:p:qt:U:vwWz",
+ long_options, &optindex)) !=3D -1)
+ {
+ switch (c)
+ {
+ case 'a':
+ vacopts.objfilter |=3D OBJFILTER_ALL_DBS;
+ break;
+ case 'd':
+ vacopts.objfilter |=3D OBJFILTER_DATABASE;
+ dbname =3D pg_strdup(optarg);
+ break;
+ case 'e':
+ vacopts.echo =3D true;
+ break;
+ case 'h':
+ cparams.pghost =3D pg_strdup(optarg);
+ break;
+ case 'j':
+ if (!option_parse_int(optarg, "-j/--jobs", 1, INT_MAX,
+ &concurrentCons))
+ exit(1);
+ break;
+ case 'n':
+ vacopts.objfilter |=3D OBJFILTER_SCHEMA;
+ simple_string_list_append(&objects, optarg);
+ break;
+ case 'N':
+ vacopts.objfilter |=3D OBJFILTER_SCHEMA_EXCLUDE;
+ simple_string_list_append(&objects, optarg);
+ break;
+ case 'p':
+ cparams.pgport =3D pg_strdup(optarg);
+ break;
+ case 'q':
+ vacopts.quiet =3D true;
+ break;
+ case 't':
+ vacopts.objfilter |=3D OBJFILTER_TABLE;
+ simple_string_list_append(&objects, optarg);
+ tbl_count++;
+ break;
+ case 'U':
+ cparams.pguser =3D pg_strdup(optarg);
+ break;
+ case 'v':
+ vacopts.verbose =3D true;
+ break;
+ case 'w':
+ cparams.prompt_password =3D TRI_NO;
+ break;
+ case 'W':
+ cparams.prompt_password =3D TRI_YES;
+ break;
+ case 'z':
+ vacopts.and_analyze =3D true;
+ break;
+ case 1:
+ vacopts.using_index =3D true;
+ if (optarg)
+ vacopts.indexname =3D pg_strdup(optarg);
+ else
+ vacopts.indexname =3D NULL;
+ break;
+ case 2:
+ maintenance_db =3D pg_strdup(optarg);
+ break;
+ default:
+ /* getopt_long already emitted a complaint */
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+ }
+
+ /*
+ * Non-option argument specifies database name as long as it wasn't
+ * already specified with -d / --dbname
+ */
+ if (optind < argc && dbname =3D=3D NULL)
+ {
+ vacopts.objfilter |=3D OBJFILTER_DATABASE;
+ dbname =3D argv[optind];
+ optind++;
+ }
+
+ if (optind < argc)
+ {
+ pg_log_error("too many command-line arguments (first is \"%s\")",
+ argv[optind]);
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit(1);
+ }
+
+ /*
+ * Validate the combination of filters specified in the command-line
+ * options.
+ */
+ check_objfilter(vacopts.objfilter);
+
+ ret =3D vacuuming_main(&cparams, dbname, maintenance_db, &vacopts,
+ &objects, tbl_count, concurrentCons,
+ progname);
+ exit(ret);
+}
+
+/*
+ * Verify that the filters used at command line are compatible.
+ */
+void
+check_objfilter(bits32 objfilter)
+{
+ if ((objfilter & OBJFILTER_ALL_DBS) &&
+ (objfilter & OBJFILTER_DATABASE))
+ pg_fatal("cannot repack all databases and a specific one at the same tim=
e");
+
+ if ((objfilter & OBJFILTER_TABLE) &&
+ (objfilter & OBJFILTER_SCHEMA))
+ pg_fatal("cannot repack all tables in schema(s) and specific table(s) at=
the same time");
+
+ if ((objfilter & OBJFILTER_TABLE) &&
+ (objfilter & OBJFILTER_SCHEMA_EXCLUDE))
+ pg_fatal("cannot repack specific table(s) and exclude schema(s) at the s=
ame time");
+
+ if ((objfilter & OBJFILTER_SCHEMA) &&
+ (objfilter & OBJFILTER_SCHEMA_EXCLUDE))
+ pg_fatal("cannot repack all tables in schema(s) and exclude schema(s) at=
the same time");
+}
+
+static void
+help(const char *progname)
+{
+ printf(_("%s repacks a PostgreSQL database.\n\n"), progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION]... [DBNAME]\n"), progname);
+ printf(_("\nOptions:\n"));
+ printf(_(" -a, --all repack all databases\n"));
+ printf(_(" -d, --dbname=3DDBNAME database to repack\n"));
+ printf(_(" -e, --echo show the commands being sent =
to the server\n"));
+ printf(_(" --index[=3DINDEX] repack following an index\n=
"));
+ printf(_(" -j, --jobs=3DNUM use this many concurrent co=
nnections to repack\n"));
+ printf(_(" -n, --schema=3DSCHEMA repack tables in the specif=
ied schema(s) only\n"));
+ printf(_(" -N, --exclude-schema=3DSCHEMA do not repack tables in the=
specified schema(s)\n"));
+ printf(_(" -q, --quiet don't write any messages\n"));
+ printf(_(" -t, --table=3D'TABLE[(COLUMNS)]' repack specific table(s) on=
ly\n"));
+ printf(_(" -v, --verbose write a lot of output\n"));
+ printf(_(" -V, --version output version information, t=
hen exit\n"));
+ printf(_(" -z, --analyze update optimizer statistics\n=
"));
+ printf(_(" -?, --help show this help, then exit\n")=
);
+ printf(_("\nConnection options:\n"));
+ printf(_(" -h, --host=3DHOSTNAME database server host or socket di=
rectory\n"));
+ printf(_(" -p, --port=3DPORT database server port\n"));
+ printf(_(" -U, --username=3DUSERNAME user name to connect as\n"));
+ printf(_(" -w, --no-password never prompt for password\n"));
+ printf(_(" -W, --password force password prompt\n"));
+ printf(_(" --maintenance-db=3DDBNAME alternate maintenance database\n"=
));
+ printf(_("\nRead the description of the SQL command REPACK for details.\n=
"));
+ printf(_("\nReport bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+ printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
diff --git a/src/bin/scripts/t/103_repackdb.pl b/src/bin/scripts/t/103_repa=
ckdb.pl
new file mode 100644
index 00000000000..44113367a61
--- /dev/null
+++ b/src/bin/scripts/t/103_repackdb.pl
@@ -0,0 +1,47 @@
+# Copyright (c) 2021-2026, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL =3D> 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+program_help_ok('pg_repackdb');
+program_version_ok('pg_repackdb');
+program_options_handling_ok('pg_repackdb');
+
+my $node =3D PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->start;
+
+$node->issues_sql_like(
+ [ 'pg_repackdb', 'postgres', '-t', 'pg_class'],
+ qr/statement: REPACK.*pg_class;/,
+ 'pg_repackdb processes a single table');
+
+$node->safe_psql('postgres', 'CREATE USER testusr;
+ GRANT CREATE ON SCHEMA public TO testusr');
+$node->safe_psql('postgres',
+ 'CREATE TABLE cluster_1 (a int primary key);
+ ALTER TABLE cluster_1 CLUSTER ON cluster_1_pkey;
+ CREATE TABLE cluster_2 (a int unique);
+ ALTER TABLE cluster_2 CLUSTER ON cluster_2_a_key;',
+ extra_params =3D> ['-U' =3D> 'testusr']);
+
+$node->issues_sql_like(
+ [ 'pg_repackdb', 'postgres', '-U', 'testusr' ],
+ qr/statement: REPACK.*;/,
+ 'SQL REPACK run');
+
+$node->issues_sql_like(
+ [ 'pg_repackdb', 'postgres', '--index'],
+ qr/statement: REPACK.*cluster_1 USING INDEX.*statement: REPACK.*cluster_2=
USING INDEX/ms,
+ 'pg_repackdb --index chooses multiple tables');
+
+$node->issues_sql_like(
+ [ 'pg_repackdb', 'postgres', '--analyze', '-t', 'cluster_1'],
+ qr/statement: REPACK \(ANALYZE\) public.cluster_1/,
+ 'pg_repackdb --analyze works');
+
+done_testing();
diff --git a/src/bin/scripts/vacuuming.c b/src/bin/scripts/vacuuming.c
index faac9089a01..0fa7be3c0c3 100644
--- a/src/bin/scripts/vacuuming.c
+++ b/src/bin/scripts/vacuuming.c
@@ -1,6 +1,6 @@
/*-------------------------------------------------------------------------
* vacuuming.c
- * Helper routines for vacuumdb
+ * Helper routines for vacuumdb and pg_repackdb
*
* Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -194,6 +194,14 @@ vacuum_one_database(ConnParams *cparams,
=20
conn =3D connectDatabase(cparams, progname, vacopts->echo, false, true);
=20
+ if (vacopts->mode =3D=3D MODE_REPACK && PQserverVersion(conn) < 190000)
+ {
+ /* XXX arguably, here we should use VACUUM FULL instead of failing */
+ PQfinish(conn);
+ pg_fatal("cannot use the \"%s\" command on server versions older than Po=
stgreSQL %s",
+ "REPACK", "19");
+ }
+
if (vacopts->disable_page_skipping && PQserverVersion(conn) < 90600)
{
PQfinish(conn);
@@ -286,9 +294,18 @@ vacuum_one_database(ConnParams *cparams,
if (vacopts->mode =3D=3D MODE_ANALYZE_IN_STAGES)
printf(_("%s: processing database \"%s\": %s\n"),
progname, PQdb(conn), _(stage_messages[stage]));
- else
+ else if (vacopts->mode =3D=3D MODE_ANALYZE)
+ printf(_("%s: analyzing database \"%s\"\n"),
+ progname, PQdb(conn));
+ else if (vacopts->mode =3D=3D MODE_VACUUM)
printf(_("%s: vacuuming database \"%s\"\n"),
progname, PQdb(conn));
+ else
+ {
+ Assert(vacopts->mode =3D=3D MODE_REPACK);
+ printf(_("%s: repacking database \"%s\"\n"),
+ progname, PQdb(conn));
+ }
fflush(stdout);
}
=20
@@ -640,6 +657,35 @@ retrieve_objects(PGconn *conn, vacuumingOptions *vacop=
ts,
" AND listed_objects.object_oid IS NOT NULL\n");
}
=20
+ /*
+ * In REPACK mode, if the 'using_index' option was given but no index
+ * name, filter only tables that have an index with indisclustered set.
+ * (If an index name is given, we trust the user to pass a reasonable list
+ * of tables.)
+ *
+ * XXX it may be worth printing an error if an index name is given with no
+ * list of tables.
+ */
+ if (vacopts->mode =3D=3D MODE_REPACK &&
+ vacopts->using_index && !vacopts->indexname)
+ {
+ appendPQExpBufferStr(&catalog_query,
+ " AND EXISTS (SELECT 1 FROM pg_catalog.pg_index\n"
+ " WHERE indrelid =3D c.oid AND indisclustered)\n");
+ }
+
+ /*
+ * In REPACK mode, only consider the tables that the current user has
+ * MAINTAIN privileges on. XXX maybe we should do this in all cases, not
+ * just REPACK. The vacuumdb output is too noisy for no reason.
+ */
+ if (vacopts->mode =3D=3D MODE_REPACK)
+ {
+ appendPQExpBufferStr(&catalog_query,
+ " AND pg_catalog.has_table_privilege(current_user, "
+ "c.oid, 'MAINTAIN')\n");
+ }
+
/*
* If no tables were listed, filter for the relevant relation types. If
* tables were given via --table, don't bother filtering by relation type.
@@ -878,8 +924,10 @@ prepare_vacuum_command(PGconn *conn, PQExpBuffer sql,
if (vacopts->verbose)
appendPQExpBufferStr(sql, " VERBOSE");
}
+
+ appendPQExpBuffer(sql, " %s", table);
}
- else
+ else if (vacopts->mode =3D=3D MODE_VACUUM)
{
appendPQExpBufferStr(sql, "VACUUM");
=20
@@ -993,9 +1041,39 @@ prepare_vacuum_command(PGconn *conn, PQExpBuffer sql,
if (vacopts->and_analyze)
appendPQExpBufferStr(sql, " ANALYZE");
}
+
+ appendPQExpBuffer(sql, " %s", table);
}
+ else if (vacopts->mode =3D=3D MODE_REPACK)
+ {
+ appendPQExpBufferStr(sql, "REPACK");
=20
- appendPQExpBuffer(sql, " %s;", table);
+ if (vacopts->verbose)
+ {
+ appendPQExpBuffer(sql, "%sVERBOSE", sep);
+ sep =3D comma;
+ }
+ if (vacopts->and_analyze)
+ {
+ appendPQExpBuffer(sql, "%sANALYZE", sep);
+ sep =3D comma;
+ }
+
+ if (sep !=3D paren)
+ appendPQExpBufferChar(sql, ')');
+
+ appendPQExpBuffer(sql, " %s", table);
+
+ if (vacopts->using_index)
+ {
+ appendPQExpBuffer(sql, " USING INDEX");
+ if (vacopts->indexname)
+ appendPQExpBuffer(sql, " %s", fmtIdEnc(vacopts->indexname,
+ PQclientEncoding(conn)));
+ }
+ }
+
+ appendPQExpBufferChar(sql, ';');
}
=20
/*
@@ -1024,13 +1102,21 @@ run_vacuum_command(ParallelSlot *free_slot, vacuumi=
ngOptions *vacopts,
{
if (table)
{
- pg_log_error("vacuuming of table \"%s\" in database \"%s\" failed: %s",
- table, PQdb(conn), PQerrorMessage(conn));
+ if (vacopts->mode =3D=3D MODE_VACUUM)
+ pg_log_error("vacuuming of table \"%s\" in database \"%s\" failed: %s",
+ table, PQdb(conn), PQerrorMessage(conn));
+ else
+ pg_log_error("repacking of table \"%s\" in database \"%s\" failed: %s",
+ table, PQdb(conn), PQerrorMessage(conn));
}
else
{
- pg_log_error("vacuuming of database \"%s\" failed: %s",
- PQdb(conn), PQerrorMessage(conn));
+ if (vacopts->mode =3D=3D MODE_VACUUM)
+ pg_log_error("vacuuming of database \"%s\" failed: %s",
+ PQdb(conn), PQerrorMessage(conn));
+ else
+ pg_log_error("repacking of database \"%s\" failed: %s",
+ PQdb(conn), PQerrorMessage(conn));
}
}
}
diff --git a/src/bin/scripts/vacuuming.h b/src/bin/scripts/vacuuming.h
index be4a75ef2f4..6e8abe00a4f 100644
--- a/src/bin/scripts/vacuuming.h
+++ b/src/bin/scripts/vacuuming.h
@@ -20,6 +20,7 @@
typedef enum
{
MODE_VACUUM,
+ MODE_REPACK,
MODE_ANALYZE,
MODE_ANALYZE_IN_STAGES
} RunMode;
@@ -37,6 +38,8 @@ typedef struct vacuumingOptions
bool and_analyze;
bool full;
bool freeze;
+ bool using_index;
+ char *indexname;
bool disable_page_skipping;
bool skip_locked;
int min_xid_age;
diff --git a/src/include/commands/cluster.h b/src/include/commands/cluster.h
index 8ea81622f9d..28741988478 100644
--- a/src/include/commands/cluster.h
+++ b/src/include/commands/cluster.h
@@ -24,6 +24,7 @@
#define CLUOPT_RECHECK 0x02 /* recheck relation state */
#define CLUOPT_RECHECK_ISCLUSTERED 0x04 /* recheck relation state for
* indisclustered */
+#define CLUOPT_ANALYZE 0x08 /* do an ANALYZE */
=20
/* options for CLUSTER */
typedef struct ClusterParams
@@ -31,8 +32,11 @@ typedef struct ClusterParams
bits32 options; /* bitmask of CLUOPT_* */
} ClusterParams;
=20
-extern void cluster(ParseState *pstate, ClusterStmt *stmt, bool isTopLevel=
);
-extern void cluster_rel(Relation OldHeap, Oid indexOid, ClusterParams *par=
ams);
+
+extern void ExecRepack(ParseState *pstate, RepackStmt *stmt, bool isTopLev=
el);
+
+extern void cluster_rel(RepackCommand command, Relation OldHeap, Oid index=
Oid,
+ ClusterParams *params);
extern void check_index_is_clusterable(Relation OldHeap, Oid indexOid,
LOCKMODE lockmode);
extern void mark_index_clustered(Relation rel, Oid indexOid, bool is_inter=
nal);
diff --git a/src/include/commands/progress.h b/src/include/commands/progres=
s.h
index 359221dc296..f00e39b937d 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -73,28 +73,34 @@
#define PROGRESS_ANALYZE_STARTED_BY_MANUAL 1
#define PROGRESS_ANALYZE_STARTED_BY_AUTOVACUUM 2
=20
-/* Progress parameters for cluster */
-#define PROGRESS_CLUSTER_COMMAND 0
-#define PROGRESS_CLUSTER_PHASE 1
-#define PROGRESS_CLUSTER_INDEX_RELID 2
-#define PROGRESS_CLUSTER_HEAP_TUPLES_SCANNED 3
-#define PROGRESS_CLUSTER_HEAP_TUPLES_WRITTEN 4
-#define PROGRESS_CLUSTER_TOTAL_HEAP_BLKS 5
-#define PROGRESS_CLUSTER_HEAP_BLKS_SCANNED 6
-#define PROGRESS_CLUSTER_INDEX_REBUILD_COUNT 7
-
-/* Phases of cluster (as advertised via PROGRESS_CLUSTER_PHASE) */
-#define PROGRESS_CLUSTER_PHASE_SEQ_SCAN_HEAP 1
-#define PROGRESS_CLUSTER_PHASE_INDEX_SCAN_HEAP 2
-#define PROGRESS_CLUSTER_PHASE_SORT_TUPLES 3
-#define PROGRESS_CLUSTER_PHASE_WRITE_NEW_HEAP 4
-#define PROGRESS_CLUSTER_PHASE_SWAP_REL_FILES 5
-#define PROGRESS_CLUSTER_PHASE_REBUILD_INDEX 6
-#define PROGRESS_CLUSTER_PHASE_FINAL_CLEANUP 7
-
-/* Commands of PROGRESS_CLUSTER */
-#define PROGRESS_CLUSTER_COMMAND_CLUSTER 1
-#define PROGRESS_CLUSTER_COMMAND_VACUUM_FULL 2
+/*
+ * Progress parameters for REPACK.
+ *
+ * Values for PROGRESS_REPACK_COMMAND are defined as in RepackCommand.
+ *
+ * Note: Since REPACK shares code with CLUSTER, these values are also
+ * used by CLUSTER. (CLUSTER is now deprecated, so it makes little sense to
+ * introduce a separate set of constants.)
+ */
+#define PROGRESS_REPACK_COMMAND 0
+#define PROGRESS_REPACK_PHASE 1
+#define PROGRESS_REPACK_INDEX_RELID 2
+#define PROGRESS_REPACK_HEAP_TUPLES_SCANNED 3
+#define PROGRESS_REPACK_HEAP_TUPLES_WRITTEN 4
+#define PROGRESS_REPACK_TOTAL_HEAP_BLKS 5
+#define PROGRESS_REPACK_HEAP_BLKS_SCANNED 6
+#define PROGRESS_REPACK_INDEX_REBUILD_COUNT 7
+
+/*
+ * Phases of repack (as advertised via PROGRESS_REPACK_PHASE).
+ */
+#define PROGRESS_REPACK_PHASE_SEQ_SCAN_HEAP 1
+#define PROGRESS_REPACK_PHASE_INDEX_SCAN_HEAP 2
+#define PROGRESS_REPACK_PHASE_SORT_TUPLES 3
+#define PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP 4
+#define PROGRESS_REPACK_PHASE_SWAP_REL_FILES 5
+#define PROGRESS_REPACK_PHASE_REBUILD_INDEX 6
+#define PROGRESS_REPACK_PHASE_FINAL_CLEANUP 7
=20
/* Progress parameters for CREATE INDEX */
/* 3, 4 and 5 reserved for "waitfor" metrics */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 646d6ced763..c15ba5e6f29 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3980,18 +3980,6 @@ typedef struct AlterSystemStmt
VariableSetStmt *setstmt; /* SET subcommand */
} AlterSystemStmt;
=20
-/* ----------------------
- * Cluster Statement (support pbrown's cluster index implementation)
- * ----------------------
- */
-typedef struct ClusterStmt
-{
- NodeTag type;
- RangeVar *relation; /* relation being indexed, or NULL if all */
- char *indexname; /* original index defined */
- List *params; /* list of DefElem nodes */
-} ClusterStmt;
-
/* ----------------------
* Vacuum and Analyze Statements
*
@@ -4004,7 +3992,7 @@ typedef struct VacuumStmt
NodeTag type;
List *options; /* list of DefElem nodes */
List *rels; /* list of VacuumRelation, or NIL for all */
- bool is_vacuumcmd; /* true for VACUUM, false for ANALYZE */
+ bool is_vacuumcmd; /* true for VACUUM, false otherwise */
} VacuumStmt;
=20
/*
@@ -4022,6 +4010,27 @@ typedef struct VacuumRelation
List *va_cols; /* list of column names, or NIL for all */
} VacuumRelation;
=20
+/* ----------------------
+ * Repack Statement
+ * ----------------------
+ */
+typedef enum RepackCommand
+{
+ REPACK_COMMAND_CLUSTER =3D 1,
+ REPACK_COMMAND_REPACK,
+ REPACK_COMMAND_VACUUMFULL,
+} RepackCommand;
+
+typedef struct RepackStmt
+{
+ NodeTag type;
+ RepackCommand command; /* type of command being run */
+ VacuumRelation *relation; /* relation being repacked */
+ char *indexname; /* order tuples by this index */
+ bool usingindex; /* whether USING INDEX is specified */
+ List *params; /* list of DefElem nodes */
+} RepackStmt;
+
/* ----------------------
* Explain Statement
*
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f7753c5c8a8..6f74a8c05c7 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -377,6 +377,7 @@ PG_KEYWORD("reindex", REINDEX, UNRESERVED_KEYWORD, BARE=
_LABEL)
PG_KEYWORD("relative", RELATIVE_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("release", RELEASE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("rename", RENAME, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("repack", REPACK, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/tcop/cmdtaglist.h b/src/include/tcop/cmdtaglist.h
index 1290c9bab68..652dc61b834 100644
--- a/src/include/tcop/cmdtaglist.h
+++ b/src/include/tcop/cmdtaglist.h
@@ -196,6 +196,7 @@ PG_CMDTAG(CMDTAG_REASSIGN_OWNED, "REASSIGN OWNED", fals=
e, false, false)
PG_CMDTAG(CMDTAG_REFRESH_MATERIALIZED_VIEW, "REFRESH MATERIALIZED VIEW", t=
rue, false, false)
PG_CMDTAG(CMDTAG_REINDEX, "REINDEX", true, false, false)
PG_CMDTAG(CMDTAG_RELEASE, "RELEASE", false, false, false)
+PG_CMDTAG(CMDTAG_REPACK, "REPACK", false, false, false)
PG_CMDTAG(CMDTAG_RESET, "RESET", false, false, false)
PG_CMDTAG(CMDTAG_REVOKE, "REVOKE", true, false, false)
PG_CMDTAG(CMDTAG_REVOKE_ROLE, "REVOKE ROLE", false, false, false)
diff --git a/src/include/utils/backend_progress.h b/src/include/utils/backe=
nd_progress.h
index 19f63b41431..6300dbd15d5 100644
--- a/src/include/utils/backend_progress.h
+++ b/src/include/utils/backend_progress.h
@@ -24,10 +24,10 @@ typedef enum ProgressCommandType
PROGRESS_COMMAND_INVALID,
PROGRESS_COMMAND_VACUUM,
PROGRESS_COMMAND_ANALYZE,
- PROGRESS_COMMAND_CLUSTER,
PROGRESS_COMMAND_CREATE_INDEX,
PROGRESS_COMMAND_BASEBACKUP,
PROGRESS_COMMAND_COPY,
+ PROGRESS_COMMAND_REPACK,
} ProgressCommandType;
=20
#define PGSTAT_NUM_PROGRESS_PARAM 20
diff --git a/src/test/regress/expected/cluster.out b/src/test/regress/expec=
ted/cluster.out
index 4d40a6809ab..277854418fa 100644
--- a/src/test/regress/expected/cluster.out
+++ b/src/test/regress/expected/cluster.out
@@ -495,6 +495,43 @@ ALTER TABLE clstrpart SET WITHOUT CLUSTER;
ERROR: cannot mark index clustered in partitioned table
ALTER TABLE clstrpart CLUSTER ON clstrpart_idx;
ERROR: cannot mark index clustered in partitioned table
+-- Check that REPACK sets new relfilenodes: it should process exactly the =
same
+-- tables as CLUSTER did.
+DROP TABLE old_cluster_info;
+DROP TABLE new_cluster_info;
+CREATE TEMP TABLE old_cluster_info AS SELECT relname, level, relfilenode, =
relkind FROM pg_partition_tree('clstrpart'::regclass) AS tree JOIN pg_class=
c ON c.oid=3Dtree.relid ;
+REPACK clstrpart USING INDEX clstrpart_idx;
+CREATE TEMP TABLE new_cluster_info AS SELECT relname, level, relfilenode, =
relkind FROM pg_partition_tree('clstrpart'::regclass) AS tree JOIN pg_class=
c ON c.oid=3Dtree.relid ;
+SELECT relname, old.level, old.relkind, old.relfilenode =3D new.relfilenod=
e FROM old_cluster_info AS old JOIN new_cluster_info AS new USING (relname)=
ORDER BY relname COLLATE "C";
+ relname | level | relkind | ?column?=20
+-------------+-------+---------+----------
+ clstrpart | 0 | p | t
+ clstrpart1 | 1 | p | t
+ clstrpart11 | 2 | r | f
+ clstrpart12 | 2 | p | t
+ clstrpart2 | 1 | r | f
+ clstrpart3 | 1 | p | t
+ clstrpart33 | 2 | r | f
+(7 rows)
+
+-- And finally the same for REPACK w/o index.
+DROP TABLE old_cluster_info;
+DROP TABLE new_cluster_info;
+CREATE TEMP TABLE old_cluster_info AS SELECT relname, level, relfilenode, =
relkind FROM pg_partition_tree('clstrpart'::regclass) AS tree JOIN pg_class=
c ON c.oid=3Dtree.relid ;
+REPACK clstrpart;
+CREATE TEMP TABLE new_cluster_info AS SELECT relname, level, relfilenode, =
relkind FROM pg_partition_tree('clstrpart'::regclass) AS tree JOIN pg_class=
c ON c.oid=3Dtree.relid ;
+SELECT relname, old.level, old.relkind, old.relfilenode =3D new.relfilenod=
e FROM old_cluster_info AS old JOIN new_cluster_info AS new USING (relname)=
ORDER BY relname COLLATE "C";
+ relname | level | relkind | ?column?=20
+-------------+-------+---------+----------
+ clstrpart | 0 | p | t
+ clstrpart1 | 1 | p | t
+ clstrpart11 | 2 | r | f
+ clstrpart12 | 2 | p | t
+ clstrpart2 | 1 | r | f
+ clstrpart3 | 1 | p | t
+ clstrpart33 | 2 | r | f
+(7 rows)
+
DROP TABLE clstrpart;
-- Ownership of partitions is checked
CREATE TABLE ptnowner(i int unique) PARTITION BY LIST (i);
@@ -513,7 +550,7 @@ CREATE TEMP TABLE ptnowner_oldnodes AS
JOIN pg_class AS c ON c.oid=3Dtree.relid;
SET SESSION AUTHORIZATION regress_ptnowner;
CLUSTER ptnowner USING ptnowner_i_idx;
-WARNING: permission denied to cluster "ptnowner2", skipping it
+WARNING: permission denied to execute CLUSTER on "ptnowner2", skipping it
RESET SESSION AUTHORIZATION;
SELECT a.relname, a.relfilenode=3Db.relfilenode FROM pg_class a
JOIN ptnowner_oldnodes b USING (oid) ORDER BY a.relname COLLATE "C";
@@ -665,6 +702,101 @@ SELECT * FROM clstr_expression WHERE -a =3D -3 ORDER =
BY -a, b;
(4 rows)
=20
COMMIT;
+----------------------------------------------------------------------
+--
+-- REPACK
+--
+----------------------------------------------------------------------
+-- REPACK handles individual tables identically to CLUSTER, but it's worth
+-- checking if it handles table hierarchies identically as well.
+REPACK clstr_tst USING INDEX clstr_tst_c;
+-- Verify that inheritance link still works
+INSERT INTO clstr_tst_inh VALUES (0, 100, 'in child table 2');
+SELECT a,b,c,substring(d for 30), length(d) from clstr_tst;
+ a | b | c | substring | length=20
+----+-----+------------------+--------------------------------+--------
+ 10 | 14 | catorce | |=20=20=20=
=20=20=20=20
+ 18 | 5 | cinco | |=20=20=20=
=20=20=20=20
+ 9 | 4 | cuatro | |=20=20=20=
=20=20=20=20
+ 26 | 19 | diecinueve | |=20=20=20=
=20=20=20=20
+ 12 | 18 | dieciocho | |=20=20=20=
=20=20=20=20
+ 30 | 16 | dieciseis | |=20=20=20=
=20=20=20=20
+ 24 | 17 | diecisiete | |=20=20=20=
=20=20=20=20
+ 2 | 10 | diez | |=20=20=20=
=20=20=20=20
+ 23 | 12 | doce | |=20=20=20=
=20=20=20=20
+ 11 | 2 | dos | |=20=20=20=
=20=20=20=20
+ 25 | 9 | nueve | |=20=20=20=
=20=20=20=20
+ 31 | 8 | ocho | |=20=20=20=
=20=20=20=20
+ 1 | 11 | once | |=20=20=20=
=20=20=20=20
+ 28 | 15 | quince | |=20=20=20=
=20=20=20=20
+ 32 | 6 | seis | xyzzyxyzzyxyzzyxyzzyxyzzyxyzzy | 500000
+ 29 | 7 | siete | |=20=20=20=
=20=20=20=20
+ 15 | 13 | trece | |=20=20=20=
=20=20=20=20
+ 22 | 30 | treinta | |=20=20=20=
=20=20=20=20
+ 17 | 32 | treinta y dos | |=20=20=20=
=20=20=20=20
+ 3 | 31 | treinta y uno | |=20=20=20=
=20=20=20=20
+ 5 | 3 | tres | |=20=20=20=
=20=20=20=20
+ 20 | 1 | uno | |=20=20=20=
=20=20=20=20
+ 6 | 20 | veinte | |=20=20=20=
=20=20=20=20
+ 14 | 25 | veinticinco | |=20=20=20=
=20=20=20=20
+ 21 | 24 | veinticuatro | |=20=20=20=
=20=20=20=20
+ 4 | 22 | veintidos | |=20=20=20=
=20=20=20=20
+ 19 | 29 | veintinueve | |=20=20=20=
=20=20=20=20
+ 16 | 28 | veintiocho | |=20=20=20=
=20=20=20=20
+ 27 | 26 | veintiseis | |=20=20=20=
=20=20=20=20
+ 13 | 27 | veintisiete | |=20=20=20=
=20=20=20=20
+ 7 | 23 | veintitres | |=20=20=20=
=20=20=20=20
+ 8 | 21 | veintiuno | |=20=20=20=
=20=20=20=20
+ 0 | 100 | in child table | |=20=20=20=
=20=20=20=20
+ 0 | 100 | in child table 2 | |=20=20=20=
=20=20=20=20
+(34 rows)
+
+-- Verify that foreign key link still works
+INSERT INTO clstr_tst (b, c) VALUES (1111, 'this should fail');
+ERROR: insert or update on table "clstr_tst" violates foreign key constra=
int "clstr_tst_con"
+DETAIL: Key (b)=3D(1111) is not present in table "clstr_tst_s".
+SELECT conname FROM pg_constraint WHERE conrelid =3D 'clstr_tst'::regclass
+ORDER BY 1;
+ conname=20=20=20=20=20=20=20=20
+----------------------
+ clstr_tst_a_not_null
+ clstr_tst_con
+ clstr_tst_pkey
+(3 rows)
+
+-- Verify partial analyze works
+REPACK (ANALYZE) clstr_tst (a);
+REPACK (ANALYZE) clstr_tst;
+REPACK (VERBOSE) clstr_tst (a);
+ERROR: ANALYZE option must be specified when a column list is provided
+-- REPACK w/o argument performs no ordering, so we can only check which ta=
bles
+-- have the relfilenode changed.
+RESET SESSION AUTHORIZATION;
+CREATE TEMP TABLE relnodes_old AS
+(SELECT relname, relfilenode
+FROM pg_class
+WHERE relname IN ('clstr_1', 'clstr_2', 'clstr_3'));
+SET SESSION AUTHORIZATION regress_clstr_user;
+SET client_min_messages =3D ERROR; -- order of "skipping" warnings may va=
ry
+REPACK;
+RESET client_min_messages;
+RESET SESSION AUTHORIZATION;
+CREATE TEMP TABLE relnodes_new AS
+(SELECT relname, relfilenode
+FROM pg_class
+WHERE relname IN ('clstr_1', 'clstr_2', 'clstr_3'));
+-- Do the actual comparison. Unlike CLUSTER, clstr_3 should have been
+-- processed because there is nothing like clustering index here.
+SELECT o.relname FROM relnodes_old o
+JOIN relnodes_new n ON o.relname =3D n.relname
+WHERE o.relfilenode <> n.relfilenode
+ORDER BY o.relname;
+ relname=20
+---------
+ clstr_1
+ clstr_3
+(2 rows)
+
-- clean up
DROP TABLE clustertest;
DROP TABLE clstr_1;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expecte=
d/rules.out
index f4ee2bd7459..48461550636 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2002,34 +2002,23 @@ pg_stat_progress_basebackup| SELECT pid,
ELSE NULL::text
END AS backup_type
FROM pg_stat_get_progress_info('BASEBACKUP'::text) s(pid, datid, relid,=
param1, param2, param3, param4, param5, param6, param7, param8, param9, pa=
ram10, param11, param12, param13, param14, param15, param16, param17, param=
18, param19, param20);
-pg_stat_progress_cluster| SELECT s.pid,
- s.datid,
- d.datname,
- s.relid,
- CASE s.param1
- WHEN 1 THEN 'CLUSTER'::text
- WHEN 2 THEN 'VACUUM FULL'::text
- ELSE NULL::text
+pg_stat_progress_cluster| SELECT pid,
+ datid,
+ datname,
+ relid,
+ CASE
+ WHEN (command =3D ANY (ARRAY['CLUSTER'::text, 'VACUUM FULL'::t=
ext])) THEN command
+ WHEN (repack_index_relid =3D (0)::oid) THEN 'VACUUM FULL'::text
+ ELSE 'CLUSTER'::text
END AS command,
- CASE s.param2
- WHEN 0 THEN 'initializing'::text
- WHEN 1 THEN 'seq scanning heap'::text
- WHEN 2 THEN 'index scanning heap'::text
- WHEN 3 THEN 'sorting tuples'::text
- WHEN 4 THEN 'writing new heap'::text
- WHEN 5 THEN 'swapping relation files'::text
- WHEN 6 THEN 'rebuilding index'::text
- WHEN 7 THEN 'performing final cleanup'::text
- ELSE NULL::text
- END AS phase,
- (s.param3)::oid AS cluster_index_relid,
- s.param4 AS heap_tuples_scanned,
- s.param5 AS heap_tuples_written,
- s.param6 AS heap_blks_total,
- s.param7 AS heap_blks_scanned,
- s.param8 AS index_rebuild_count
- FROM (pg_stat_get_progress_info('CLUSTER'::text) s(pid, datid, relid, p=
aram1, param2, param3, param4, param5, param6, param7, param8, param9, para=
m10, param11, param12, param13, param14, param15, param16, param17, param18=
, param19, param20)
- LEFT JOIN pg_database d ON ((s.datid =3D d.oid)));
+ phase,
+ repack_index_relid AS cluster_index_relid,
+ heap_tuples_scanned,
+ heap_tuples_written,
+ heap_blks_total,
+ heap_blks_scanned,
+ index_rebuild_count
+ FROM pg_stat_progress_repack;
pg_stat_progress_copy| SELECT s.pid,
s.datid,
d.datname,
@@ -2089,6 +2078,35 @@ pg_stat_progress_create_index| SELECT s.pid,
s.param15 AS partitions_done
FROM (pg_stat_get_progress_info('CREATE INDEX'::text) s(pid, datid, rel=
id, param1, param2, param3, param4, param5, param6, param7, param8, param9,=
param10, param11, param12, param13, param14, param15, param16, param17, pa=
ram18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid =3D d.oid)));
+pg_stat_progress_repack| SELECT s.pid,
+ s.datid,
+ d.datname,
+ s.relid,
+ CASE s.param1
+ WHEN 1 THEN 'CLUSTER'::text
+ WHEN 2 THEN 'REPACK'::text
+ WHEN 3 THEN 'VACUUM FULL'::text
+ ELSE NULL::text
+ END AS command,
+ CASE s.param2
+ WHEN 0 THEN 'initializing'::text
+ WHEN 1 THEN 'seq scanning heap'::text
+ WHEN 2 THEN 'index scanning heap'::text
+ WHEN 3 THEN 'sorting tuples'::text
+ WHEN 4 THEN 'writing new heap'::text
+ WHEN 5 THEN 'swapping relation files'::text
+ WHEN 6 THEN 'rebuilding index'::text
+ WHEN 7 THEN 'performing final cleanup'::text
+ ELSE NULL::text
+ END AS phase,
+ (s.param3)::oid AS repack_index_relid,
+ s.param4 AS heap_tuples_scanned,
+ s.param5 AS heap_tuples_written,
+ s.param6 AS heap_blks_total,
+ s.param7 AS heap_blks_scanned,
+ s.param8 AS index_rebuild_count
+ FROM (pg_stat_get_progress_info('REPACK'::text) s(pid, datid, relid, pa=
ram1, param2, param3, param4, param5, param6, param7, param8, param9, param=
10, param11, param12, param13, param14, param15, param16, param17, param18,=
param19, param20)
+ LEFT JOIN pg_database d ON ((s.datid =3D d.oid)));
pg_stat_progress_vacuum| SELECT s.pid,
s.datid,
d.datname,
diff --git a/src/test/regress/sql/cluster.sql b/src/test/regress/sql/cluste=
r.sql
index b7115f86104..c976823a3cb 100644
--- a/src/test/regress/sql/cluster.sql
+++ b/src/test/regress/sql/cluster.sql
@@ -76,7 +76,6 @@ INSERT INTO clstr_tst (b, c) VALUES (1111, 'this should f=
ail');
SELECT conname FROM pg_constraint WHERE conrelid =3D 'clstr_tst'::regclass
ORDER BY 1;
=20
-
SELECT relname, relkind,
EXISTS(SELECT 1 FROM pg_class WHERE oid =3D c.reltoastrelid) AS hastoa=
st
FROM pg_class c WHERE relname LIKE 'clstr_tst%' ORDER BY relname;
@@ -229,6 +228,24 @@ SELECT relname, old.level, old.relkind, old.relfilenod=
e =3D new.relfilenode FROM o
CLUSTER clstrpart;
ALTER TABLE clstrpart SET WITHOUT CLUSTER;
ALTER TABLE clstrpart CLUSTER ON clstrpart_idx;
+
+-- Check that REPACK sets new relfilenodes: it should process exactly the =
same
+-- tables as CLUSTER did.
+DROP TABLE old_cluster_info;
+DROP TABLE new_cluster_info;
+CREATE TEMP TABLE old_cluster_info AS SELECT relname, level, relfilenode, =
relkind FROM pg_partition_tree('clstrpart'::regclass) AS tree JOIN pg_class=
c ON c.oid=3Dtree.relid ;
+REPACK clstrpart USING INDEX clstrpart_idx;
+CREATE TEMP TABLE new_cluster_info AS SELECT relname, level, relfilenode, =
relkind FROM pg_partition_tree('clstrpart'::regclass) AS tree JOIN pg_class=
c ON c.oid=3Dtree.relid ;
+SELECT relname, old.level, old.relkind, old.relfilenode =3D new.relfilenod=
e FROM old_cluster_info AS old JOIN new_cluster_info AS new USING (relname)=
ORDER BY relname COLLATE "C";
+
+-- And finally the same for REPACK w/o index.
+DROP TABLE old_cluster_info;
+DROP TABLE new_cluster_info;
+CREATE TEMP TABLE old_cluster_info AS SELECT relname, level, relfilenode, =
relkind FROM pg_partition_tree('clstrpart'::regclass) AS tree JOIN pg_class=
c ON c.oid=3Dtree.relid ;
+REPACK clstrpart;
+CREATE TEMP TABLE new_cluster_info AS SELECT relname, level, relfilenode, =
relkind FROM pg_partition_tree('clstrpart'::regclass) AS tree JOIN pg_class=
c ON c.oid=3Dtree.relid ;
+SELECT relname, old.level, old.relkind, old.relfilenode =3D new.relfilenod=
e FROM old_cluster_info AS old JOIN new_cluster_info AS new USING (relname)=
ORDER BY relname COLLATE "C";
+
DROP TABLE clstrpart;
=20
-- Ownership of partitions is checked
@@ -313,6 +330,57 @@ EXPLAIN (COSTS OFF) SELECT * FROM clstr_expression WHE=
RE -a =3D -3 ORDER BY -a, b;
SELECT * FROM clstr_expression WHERE -a =3D -3 ORDER BY -a, b;
COMMIT;
=20
+----------------------------------------------------------------------
+--
+-- REPACK
+--
+----------------------------------------------------------------------
+
+-- REPACK handles individual tables identically to CLUSTER, but it's worth
+-- checking if it handles table hierarchies identically as well.
+REPACK clstr_tst USING INDEX clstr_tst_c;
+
+-- Verify that inheritance link still works
+INSERT INTO clstr_tst_inh VALUES (0, 100, 'in child table 2');
+SELECT a,b,c,substring(d for 30), length(d) from clstr_tst;
+
+-- Verify that foreign key link still works
+INSERT INTO clstr_tst (b, c) VALUES (1111, 'this should fail');
+
+SELECT conname FROM pg_constraint WHERE conrelid =3D 'clstr_tst'::regclass
+ORDER BY 1;
+
+-- Verify partial analyze works
+REPACK (ANALYZE) clstr_tst (a);
+REPACK (ANALYZE) clstr_tst;
+REPACK (VERBOSE) clstr_tst (a);
+
+-- REPACK w/o argument performs no ordering, so we can only check which ta=
bles
+-- have the relfilenode changed.
+RESET SESSION AUTHORIZATION;
+CREATE TEMP TABLE relnodes_old AS
+(SELECT relname, relfilenode
+FROM pg_class
+WHERE relname IN ('clstr_1', 'clstr_2', 'clstr_3'));
+
+SET SESSION AUTHORIZATION regress_clstr_user;
+SET client_min_messages =3D ERROR; -- order of "skipping" warnings may va=
ry
+REPACK;
+RESET client_min_messages;
+
+RESET SESSION AUTHORIZATION;
+CREATE TEMP TABLE relnodes_new AS
+(SELECT relname, relfilenode
+FROM pg_class
+WHERE relname IN ('clstr_1', 'clstr_2', 'clstr_3'));
+
+-- Do the actual comparison. Unlike CLUSTER, clstr_3 should have been
+-- processed because there is nothing like clustering index here.
+SELECT o.relname FROM relnodes_old o
+JOIN relnodes_new n ON o.relname =3D n.relname
+WHERE o.relfilenode <> n.relfilenode
+ORDER BY o.relname;
+
-- clean up
DROP TABLE clustertest;
DROP TABLE clstr_1;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs=
.list
index 1c8610fd46c..da0df6e89d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2576,6 +2576,8 @@ ReorderBufferTupleCidKey
ReorderBufferUpdateProgressTxnCB
ReorderTuple
RepOriginId
+RepackCommand
+RepackStmt
ReparameterizeForeignPathByChild_function
ReplaceVarsFromTargetList_context
ReplaceVarsNoMatchOption
--=20
2.47.3
--=-=-=
Content-Type: text/x-diff
Content-Disposition: attachment;
filename=v31-0002-Refactor-index_concurrently_create_copy-for-use-with.patch
^ permalink raw reply [nested|flat] 38+ messages in thread
end of thread, other threads:[~2026-01-22 08:23 UTC | newest]
Thread overview: 38+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-01-08 23:41 Re: Why is src/test/modules/committs/t/002_standby.pl flaky? Tom Lane <[email protected]>
2022-01-09 01:17 ` Tom Lane <[email protected]>
2022-01-09 11:00 ` Alexander Lakhin <[email protected]>
2022-01-09 16:49 ` Tom Lane <[email protected]>
2022-01-09 19:06 ` Thomas Munro <[email protected]>
2022-01-10 02:00 ` Thomas Munro <[email protected]>
2022-01-10 07:00 ` Alexander Lakhin <[email protected]>
2022-01-10 09:20 ` Thomas Munro <[email protected]>
2022-01-10 09:40 ` Thomas Munro <[email protected]>
2022-01-10 17:00 ` Alexander Lakhin <[email protected]>
2022-01-10 20:52 ` Thomas Munro <[email protected]>
2022-01-10 21:21 ` Andrew Dunstan <[email protected]>
2022-01-10 21:25 ` Tom Lane <[email protected]>
2022-03-19 08:47 ` Noah Misch <[email protected]>
2022-03-22 02:16 ` Thomas Munro <[email protected]>
2022-03-22 03:13 ` Tom Lane <[email protected]>
2022-01-11 15:00 ` Alexander Lakhin <[email protected]>
2022-01-11 20:10 ` Thomas Munro <[email protected]>
2022-01-11 20:16 ` Tom Lane <[email protected]>
2022-01-12 07:00 ` Alexander Lakhin <[email protected]>
2022-01-13 06:36 ` Thomas Munro <[email protected]>
2022-01-13 07:00 ` Alexander Lakhin <[email protected]>
2022-01-14 07:31 ` Thomas Munro <[email protected]>
2022-01-14 20:28 ` Andres Freund <[email protected]>
2022-01-14 20:47 ` Andres Freund <[email protected]>
2022-01-15 00:40 ` Thomas Munro <[email protected]>
2022-01-14 21:59 ` Thomas Munro <[email protected]>
2022-01-14 22:01 ` Thomas Munro <[email protected]>
2022-01-14 22:44 ` Andres Freund <[email protected]>
2022-01-15 00:19 ` Thomas Munro <[email protected]>
2022-01-15 00:51 ` Andres Freund <[email protected]>
2022-01-16 11:00 ` Alexander Lakhin <[email protected]>
2022-01-14 22:51 ` Tom Lane <[email protected]>
2022-01-24 20:02 ` Andres Freund <[email protected]>
2022-01-24 20:35 ` Tom Lane <[email protected]>
2022-01-24 21:28 ` Andres Freund <[email protected]>
2022-01-14 08:00 ` Alexander Lakhin <[email protected]>
2026-01-22 08:23 [PATCH 1/6] Add REPACK command Antonin Houska <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox