public inbox for [email protected]
help / color / mirror / Atom feed[PATCH 08/10] Add some missing newlines after function definitions
59+ messages / 11 participants
[nested] [flat]
* [PATCH 02/33] Add some missing newlines after function definitions
@ 2021-07-05 16:16 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Justin Pryzby @ 2021-07-05 16:16 UTC (permalink / raw)
---
src/backend/access/gin/ginlogic.c | 1 +
src/backend/access/rmgrdesc/heapdesc.c | 1 +
src/backend/utils/mmgr/freepage.c | 1 +
3 files changed, 3 insertions(+)
diff --git a/src/backend/access/gin/ginlogic.c b/src/backend/access/gin/ginlogic.c
index c38c27fbae2..62f98f9cfaf 100644
--- a/src/backend/access/gin/ginlogic.c
+++ b/src/backend/access/gin/ginlogic.c
@@ -58,6 +58,7 @@ trueConsistentFn(GinScanKey key)
key->recheckCurItem = false;
return true;
}
+
static GinTernaryValue
trueTriConsistentFn(GinScanKey key)
{
diff --git a/src/backend/access/rmgrdesc/heapdesc.c b/src/backend/access/rmgrdesc/heapdesc.c
index 6238085d65e..342fb95d869 100644
--- a/src/backend/access/rmgrdesc/heapdesc.c
+++ b/src/backend/access/rmgrdesc/heapdesc.c
@@ -114,6 +114,7 @@ heap_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "off %u", xlrec->offnum);
}
}
+
void
heap2_desc(StringInfo buf, XLogReaderState *record)
{
diff --git a/src/backend/utils/mmgr/freepage.c b/src/backend/utils/mmgr/freepage.c
index b26a295a4ea..eb281823bc2 100644
--- a/src/backend/utils/mmgr/freepage.c
+++ b/src/backend/utils/mmgr/freepage.c
@@ -270,6 +270,7 @@ sum_free_pages_recurse(FreePageManager *fpm, FreePageBtree *btp, Size *sum)
}
}
}
+
static Size
sum_free_pages(FreePageManager *fpm)
{
--
2.17.1
--924gEkU1VlJlwnwX
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0003-Remove-extranous-blank-in-function-header-comment-3e.patch"
^ permalink raw reply [nested|flat] 59+ messages in thread
* [PATCH 08/10] Add some missing newlines after function definitions
@ 2021-07-05 16:16 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Justin Pryzby @ 2021-07-05 16:16 UTC (permalink / raw)
---
src/backend/access/gin/ginlogic.c | 1 +
src/backend/access/rmgrdesc/heapdesc.c | 1 +
src/backend/utils/mmgr/freepage.c | 1 +
3 files changed, 3 insertions(+)
diff --git a/src/backend/access/gin/ginlogic.c b/src/backend/access/gin/ginlogic.c
index 6bf3288f5b..3c6a782b19 100644
--- a/src/backend/access/gin/ginlogic.c
+++ b/src/backend/access/gin/ginlogic.c
@@ -58,6 +58,7 @@ trueConsistentFn(GinScanKey key)
key->recheckCurItem = false;
return true;
}
+
static GinTernaryValue
trueTriConsistentFn(GinScanKey key)
{
diff --git a/src/backend/access/rmgrdesc/heapdesc.c b/src/backend/access/rmgrdesc/heapdesc.c
index 5c29fd9eae..1a5b8f7902 100644
--- a/src/backend/access/rmgrdesc/heapdesc.c
+++ b/src/backend/access/rmgrdesc/heapdesc.c
@@ -114,6 +114,7 @@ heap_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "off %u", xlrec->offnum);
}
}
+
void
heap2_desc(StringInfo buf, XLogReaderState *record)
{
diff --git a/src/backend/utils/mmgr/freepage.c b/src/backend/utils/mmgr/freepage.c
index e4ee1aab97..d253108d90 100644
--- a/src/backend/utils/mmgr/freepage.c
+++ b/src/backend/utils/mmgr/freepage.c
@@ -270,6 +270,7 @@ sum_free_pages_recurse(FreePageManager *fpm, FreePageBtree *btp, Size *sum)
}
}
}
+
static Size
sum_free_pages(FreePageManager *fpm)
{
--
2.17.0
--udcq9yAoWb9A4FsZ
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0009-Wrong-function-name-in-header-comment.patch"
^ permalink raw reply [nested|flat] 59+ messages in thread
* [PATCH v2 05/12] Add some missing newlines after function definitions
@ 2021-07-05 16:16 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Justin Pryzby @ 2021-07-05 16:16 UTC (permalink / raw)
---
src/backend/access/gin/ginlogic.c | 1 +
src/backend/access/rmgrdesc/heapdesc.c | 1 +
src/backend/utils/mmgr/freepage.c | 1 +
3 files changed, 3 insertions(+)
diff --git a/src/backend/access/gin/ginlogic.c b/src/backend/access/gin/ginlogic.c
index 6bf3288f5b..3c6a782b19 100644
--- a/src/backend/access/gin/ginlogic.c
+++ b/src/backend/access/gin/ginlogic.c
@@ -58,6 +58,7 @@ trueConsistentFn(GinScanKey key)
key->recheckCurItem = false;
return true;
}
+
static GinTernaryValue
trueTriConsistentFn(GinScanKey key)
{
diff --git a/src/backend/access/rmgrdesc/heapdesc.c b/src/backend/access/rmgrdesc/heapdesc.c
index 5c29fd9eae..1a5b8f7902 100644
--- a/src/backend/access/rmgrdesc/heapdesc.c
+++ b/src/backend/access/rmgrdesc/heapdesc.c
@@ -114,6 +114,7 @@ heap_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "off %u", xlrec->offnum);
}
}
+
void
heap2_desc(StringInfo buf, XLogReaderState *record)
{
diff --git a/src/backend/utils/mmgr/freepage.c b/src/backend/utils/mmgr/freepage.c
index e4ee1aab97..d253108d90 100644
--- a/src/backend/utils/mmgr/freepage.c
+++ b/src/backend/utils/mmgr/freepage.c
@@ -270,6 +270,7 @@ sum_free_pages_recurse(FreePageManager *fpm, FreePageBtree *btp, Size *sum)
}
}
}
+
static Size
sum_free_pages(FreePageManager *fpm)
{
--
2.17.0
--EMQjp+MvU6EBGjHc
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v2-0006-Wrong-function-name-in-header-comment.patch"
^ permalink raw reply [nested|flat] 59+ messages in thread
* [PATCH 16/16] Add some missing newlines after function definitions
@ 2021-07-05 16:16 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Justin Pryzby @ 2021-07-05 16:16 UTC (permalink / raw)
---
src/backend/access/gin/ginlogic.c | 1 +
src/backend/access/rmgrdesc/heapdesc.c | 1 +
src/backend/utils/mmgr/freepage.c | 1 +
3 files changed, 3 insertions(+)
diff --git a/src/backend/access/gin/ginlogic.c b/src/backend/access/gin/ginlogic.c
index c38c27fbae2..62f98f9cfaf 100644
--- a/src/backend/access/gin/ginlogic.c
+++ b/src/backend/access/gin/ginlogic.c
@@ -58,6 +58,7 @@ trueConsistentFn(GinScanKey key)
key->recheckCurItem = false;
return true;
}
+
static GinTernaryValue
trueTriConsistentFn(GinScanKey key)
{
diff --git a/src/backend/access/rmgrdesc/heapdesc.c b/src/backend/access/rmgrdesc/heapdesc.c
index 325aee93ff8..7556fc2546d 100644
--- a/src/backend/access/rmgrdesc/heapdesc.c
+++ b/src/backend/access/rmgrdesc/heapdesc.c
@@ -114,6 +114,7 @@ heap_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "off %u", xlrec->offnum);
}
}
+
void
heap2_desc(StringInfo buf, XLogReaderState *record)
{
diff --git a/src/backend/utils/mmgr/freepage.c b/src/backend/utils/mmgr/freepage.c
index dcf246faf11..1ba424c2298 100644
--- a/src/backend/utils/mmgr/freepage.c
+++ b/src/backend/utils/mmgr/freepage.c
@@ -270,6 +270,7 @@ sum_free_pages_recurse(FreePageManager *fpm, FreePageBtree *btp, Size *sum)
}
}
}
+
static Size
sum_free_pages(FreePageManager *fpm)
{
--
2.25.1
--TBNym+cBXeFsS4Vs--
^ permalink raw reply [nested|flat] 59+ messages in thread
* O(n) tasks cause lengthy startups and checkpoints
@ 2021-12-01 20:24 Bossart, Nathan <[email protected]>
2021-12-01 21:35 ` Re: O(n) tasks cause lengthy startups and checkpoints SATYANARAYANA NARLAPURAM <[email protected]>
2021-12-01 22:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2021-12-02 02:47 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
0 siblings, 3 replies; 59+ messages in thread
From: Bossart, Nathan @ 2021-12-01 20:24 UTC (permalink / raw)
To: pgsql-hackers
Hi hackers,
Thanks to 61752af, SyncDataDirectory() can make use of syncfs() to
avoid individually syncing all database files after a crash. However,
as noted earlier this year [0], there are still a number of O(n) tasks
that affect startup and checkpointing that I'd like to improve.
Below, I've attempted to summarize each task and to offer ideas for
improving matters. I'll likely split each of these into its own
thread, given there is community interest for such changes.
1) CheckPointSnapBuild(): This function loops through
pg_logical/snapshots to remove all snapshots that are no longer
needed. If there are many entries in this directory, this can take
a long time. The note above this function indicates that this is
done during checkpoints simply because it is convenient. IIUC
there is no requirement that this function actually completes for a
given checkpoint. My current idea is to move this to a new
maintenance worker.
2) CheckPointLogicalRewriteHeap(): This function loops through
pg_logical/mappings to remove old mappings and flush all remaining
ones. IIUC there is no requirement that the "remove old mappings"
part must complete for a given checkpoint, but the "flush all
remaining" portion allows replay after a checkpoint to only "deal
with the parts of a mapping that have been written out after the
checkpoint started." Therefore, I think we should move the "remove
old mappings" part to a new maintenance worker (probably the same
one as for 1), and we should consider using syncfs() for the "flush
all remaining" part. (I suspect the main argument against the
latter will be that it could cause IO spikes.)
3) RemovePgTempFiles(): This step can delay startup if there are many
temporary files to individually remove. This step is already
optionally done after a crash via the remove_temp_files_after_crash
GUC. I propose that we have startup move the temporary file
directories aside and create new ones, and then a separate worker
(probably the same one from 1 and 2) could clean up the old files.
4) StartupReorderBuffer(): This step deletes logical slot data that
has been spilled to disk. This code appears to be written to avoid
deleting different types of files in these directories, but AFAICT
there shouldn't be any other files. Therefore, I think we could do
something similar to 3 (i.e., move the directories aside during
startup and clean them up via a new maintenance worker).
I realize adding a new maintenance worker might be a bit heavy-handed,
but I think it would be nice to have somewhere to offload tasks that
really shouldn't impact startup and checkpointing. I imagine such a
process would come in handy down the road, too. WDYT?
Nathan
[0] https://postgr.es/m/32B59582-AA6C-4609-B08F-2256A271F7A5%40amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
@ 2021-12-01 21:35 ` SATYANARAYANA NARLAPURAM <[email protected]>
2 siblings, 0 replies; 59+ messages in thread
From: SATYANARAYANA NARLAPURAM @ 2021-12-01 21:35 UTC (permalink / raw)
To: Bossart, Nathan <[email protected]>; +Cc: pgsql-hackers
+1 to the idea. I don't see a reason why checkpointer has to do all of
that. Keeping checkpoint to minimal essential work helps servers recover
faster in the event of a crash.
RemoveOldXlogFiles is also an O(N) operation that can at least be avoided
during the end of recovery (CHECKPOINT_END_OF_RECOVERY) checkpoint. When a
sufficient number of WAL files accumulated and the previous checkpoint did
not get a chance to cleanup, this can increase the unavailability of the
server.
RemoveOldXlogFiles(_logSegNo, RedoRecPtr, recptr);
On Wed, Dec 1, 2021 at 12:24 PM Bossart, Nathan <[email protected]> wrote:
> Hi hackers,
>
> Thanks to 61752af, SyncDataDirectory() can make use of syncfs() to
> avoid individually syncing all database files after a crash. However,
> as noted earlier this year [0], there are still a number of O(n) tasks
> that affect startup and checkpointing that I'd like to improve.
> Below, I've attempted to summarize each task and to offer ideas for
> improving matters. I'll likely split each of these into its own
> thread, given there is community interest for such changes.
>
> 1) CheckPointSnapBuild(): This function loops through
> pg_logical/snapshots to remove all snapshots that are no longer
> needed. If there are many entries in this directory, this can take
> a long time. The note above this function indicates that this is
> done during checkpoints simply because it is convenient. IIUC
> there is no requirement that this function actually completes for a
> given checkpoint. My current idea is to move this to a new
> maintenance worker.
> 2) CheckPointLogicalRewriteHeap(): This function loops through
> pg_logical/mappings to remove old mappings and flush all remaining
> ones. IIUC there is no requirement that the "remove old mappings"
> part must complete for a given checkpoint, but the "flush all
> remaining" portion allows replay after a checkpoint to only "deal
> with the parts of a mapping that have been written out after the
> checkpoint started." Therefore, I think we should move the "remove
> old mappings" part to a new maintenance worker (probably the same
> one as for 1), and we should consider using syncfs() for the "flush
> all remaining" part. (I suspect the main argument against the
> latter will be that it could cause IO spikes.)
> 3) RemovePgTempFiles(): This step can delay startup if there are many
> temporary files to individually remove. This step is already
> optionally done after a crash via the remove_temp_files_after_crash
> GUC. I propose that we have startup move the temporary file
> directories aside and create new ones, and then a separate worker
> (probably the same one from 1 and 2) could clean up the old files.
> 4) StartupReorderBuffer(): This step deletes logical slot data that
> has been spilled to disk. This code appears to be written to avoid
> deleting different types of files in these directories, but AFAICT
> there shouldn't be any other files. Therefore, I think we could do
> something similar to 3 (i.e., move the directories aside during
> startup and clean them up via a new maintenance worker).
>
> I realize adding a new maintenance worker might be a bit heavy-handed,
> but I think it would be nice to have somewhere to offload tasks that
> really shouldn't impact startup and checkpointing. I imagine such a
> process would come in handy down the road, too. WDYT?
>
> Nathan
>
> [0] https://postgr.es/m/32B59582-AA6C-4609-B08F-2256A271F7A5%40amazon.com
>
>
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
@ 2021-12-01 22:56 ` Andres Freund <[email protected]>
2021-12-02 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2 siblings, 1 reply; 59+ messages in thread
From: Andres Freund @ 2021-12-01 22:56 UTC (permalink / raw)
To: Bossart, Nathan <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-12-01 20:24:25 +0000, Bossart, Nathan wrote:
> I realize adding a new maintenance worker might be a bit heavy-handed,
> but I think it would be nice to have somewhere to offload tasks that
> really shouldn't impact startup and checkpointing. I imagine such a
> process would come in handy down the road, too. WDYT?
-1. I think the overhead of an additional worker is disproportional here. And
there's simplicity benefits in having a predictable cleanup interlock as well.
I think particularly for the snapshot stuff it'd be better to optimize away
unnecessary snapshot files, rather than making the cleanup more asynchronous.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-01 22:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
@ 2021-12-02 00:19 ` Bossart, Nathan <[email protected]>
2021-12-02 02:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Euler Taveira <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Bossart, Nathan @ 2021-12-02 00:19 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On 12/1/21, 2:56 PM, "Andres Freund" <[email protected]> wrote:
> On 2021-12-01 20:24:25 +0000, Bossart, Nathan wrote:
>> I realize adding a new maintenance worker might be a bit heavy-handed,
>> but I think it would be nice to have somewhere to offload tasks that
>> really shouldn't impact startup and checkpointing. I imagine such a
>> process would come in handy down the road, too. WDYT?
>
> -1. I think the overhead of an additional worker is disproportional here. And
> there's simplicity benefits in having a predictable cleanup interlock as well.
Another idea I had was to put some upper limit on how much time is
spent on such tasks. For example, a checkpoint would only spend X
minutes on CheckPointSnapBuild() before giving up until the next one.
I think the main downside of that approach is that it could lead to
unbounded growth, so perhaps we would limit (or even skip) such tasks
only for end-of-recovery and shutdown checkpoints. Perhaps the
startup tasks could be limited in a similar fashion.
> I think particularly for the snapshot stuff it'd be better to optimize away
> unnecessary snapshot files, rather than making the cleanup more asynchronous.
I can look into this. Any pointers would be much appreciated.
Nathan
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-01 22:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2021-12-02 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
@ 2021-12-02 02:05 ` Euler Taveira <[email protected]>
2021-12-02 21:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Euler Taveira @ 2021-12-02 02:05 UTC (permalink / raw)
To: Bossart, Nathan <[email protected]>; Andres Freund <[email protected]>; +Cc: pgsql-hackers
On Wed, Dec 1, 2021, at 9:19 PM, Bossart, Nathan wrote:
> On 12/1/21, 2:56 PM, "Andres Freund" <[email protected]> wrote:
> > On 2021-12-01 20:24:25 +0000, Bossart, Nathan wrote:
> >> I realize adding a new maintenance worker might be a bit heavy-handed,
> >> but I think it would be nice to have somewhere to offload tasks that
> >> really shouldn't impact startup and checkpointing. I imagine such a
> >> process would come in handy down the road, too. WDYT?
> >
> > -1. I think the overhead of an additional worker is disproportional here. And
> > there's simplicity benefits in having a predictable cleanup interlock as well.
>
> Another idea I had was to put some upper limit on how much time is
> spent on such tasks. For example, a checkpoint would only spend X
> minutes on CheckPointSnapBuild() before giving up until the next one.
> I think the main downside of that approach is that it could lead to
> unbounded growth, so perhaps we would limit (or even skip) such tasks
> only for end-of-recovery and shutdown checkpoints. Perhaps the
> startup tasks could be limited in a similar fashion.
Saying that a certain task is O(n) doesn't mean it needs a separate process to
handle it. Did you have a use case or even better numbers (% of checkpoint /
startup time) that makes your proposal worthwhile?
I would try to optimize (1) and (2). However, delayed removal can be a
long-term issue if the new routine cannot keep up with the pace of file
creation (specially if the checkpoints are far apart).
For (3), there is already a GUC that would avoid the slowdown during startup.
Use it if you think the startup time is more important that disk space occupied
by useless files.
For (4), you are forgetting that the on-disk state of replication slots is
stored in the pg_replslot/SLOTNAME/state. It seems you cannot just rename the
replication slot directory and copy the state file. What happen if there is a
crash before copying the state file?
While we are talking about items (1), (2) and (4), we could probably have an
option to create some ephemeral logical decoding files into ramdisk (similar to
statistics directory). I wouldn't like to hijack this thread but this proposal
could alleviate the possible issues that you pointed out. If people are
interested in this proposal, I can start a new thread about it.
--
Euler Taveira
EDB https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-01 22:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2021-12-02 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-02 02:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Euler Taveira <[email protected]>
@ 2021-12-02 21:19 ` Bossart, Nathan <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Bossart, Nathan @ 2021-12-02 21:19 UTC (permalink / raw)
To: Euler Taveira <[email protected]>; Andres Freund <[email protected]>; +Cc: pgsql-hackers
On 12/1/21, 6:06 PM, "Euler Taveira" <[email protected]> wrote:
> Saying that a certain task is O(n) doesn't mean it needs a separate process to
> handle it. Did you have a use case or even better numbers (% of checkpoint /
> startup time) that makes your proposal worthwhile?
I don't have specific numbers on hand, but each of the four functions
I listed is something I routinely see impacting customers.
> For (3), there is already a GUC that would avoid the slowdown during startup.
> Use it if you think the startup time is more important that disk space occupied
> by useless files.
Setting remove_temp_files_after_crash to false only prevents temp file
cleanup during restart after a backend crash. It is always called for
other startups.
> For (4), you are forgetting that the on-disk state of replication slots is
> stored in the pg_replslot/SLOTNAME/state. It seems you cannot just rename the
> replication slot directory and copy the state file. What happen if there is a
> crash before copying the state file?
Good point. I think it's possible to deal with this, though. Perhaps
the files that should be deleted on startup should go in a separate
directory, or maybe we could devise a way to ensure the state file is
copied even if there is a crash at an inconvenient time.
Nathan
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
@ 2021-12-02 02:47 ` Bharath Rupireddy <[email protected]>
2021-12-02 21:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2 siblings, 1 reply; 59+ messages in thread
From: Bharath Rupireddy @ 2021-12-02 02:47 UTC (permalink / raw)
To: Bossart, Nathan <[email protected]>; +Cc: pgsql-hackers
On Thu, Dec 2, 2021 at 1:54 AM Bossart, Nathan <[email protected]> wrote:
>
> Hi hackers,
>
> Thanks to 61752af, SyncDataDirectory() can make use of syncfs() to
> avoid individually syncing all database files after a crash. However,
> as noted earlier this year [0], there are still a number of O(n) tasks
> that affect startup and checkpointing that I'd like to improve.
> Below, I've attempted to summarize each task and to offer ideas for
> improving matters. I'll likely split each of these into its own
> thread, given there is community interest for such changes.
>
> 1) CheckPointSnapBuild(): This function loops through
> pg_logical/snapshots to remove all snapshots that are no longer
> needed. If there are many entries in this directory, this can take
> a long time. The note above this function indicates that this is
> done during checkpoints simply because it is convenient. IIUC
> there is no requirement that this function actually completes for a
> given checkpoint. My current idea is to move this to a new
> maintenance worker.
> 2) CheckPointLogicalRewriteHeap(): This function loops through
> pg_logical/mappings to remove old mappings and flush all remaining
> ones. IIUC there is no requirement that the "remove old mappings"
> part must complete for a given checkpoint, but the "flush all
> remaining" portion allows replay after a checkpoint to only "deal
> with the parts of a mapping that have been written out after the
> checkpoint started." Therefore, I think we should move the "remove
> old mappings" part to a new maintenance worker (probably the same
> one as for 1), and we should consider using syncfs() for the "flush
> all remaining" part. (I suspect the main argument against the
> latter will be that it could cause IO spikes.)
> 3) RemovePgTempFiles(): This step can delay startup if there are many
> temporary files to individually remove. This step is already
> optionally done after a crash via the remove_temp_files_after_crash
> GUC. I propose that we have startup move the temporary file
> directories aside and create new ones, and then a separate worker
> (probably the same one from 1 and 2) could clean up the old files.
> 4) StartupReorderBuffer(): This step deletes logical slot data that
> has been spilled to disk. This code appears to be written to avoid
> deleting different types of files in these directories, but AFAICT
> there shouldn't be any other files. Therefore, I think we could do
> something similar to 3 (i.e., move the directories aside during
> startup and clean them up via a new maintenance worker).
>
> I realize adding a new maintenance worker might be a bit heavy-handed,
> but I think it would be nice to have somewhere to offload tasks that
> really shouldn't impact startup and checkpointing. I imagine such a
> process would come in handy down the road, too. WDYT?
+1 for the overall idea of making the checkpoint faster. In fact, we
here at our team have been thinking about this problem for a while. If
there are a lot of files that checkpoint has to loop over and remove,
IMO, that task can be delegated to someone else (maybe a background
worker called background cleaner or bg cleaner, of course, we can have
a GUC to enable or disable it). The checkpoint can just write some
marker files (for instance, it can write snapshot_<cutofflsn> files
with file name itself representing the cutoff lsn so that the new bg
cleaner can remove the snapshot files, similarly it can write marker
files for other file removals). Having said that, a new bg cleaner
deleting the files asynchronously on behalf of checkpoint can look an
overkill until we have some numbers that we could save with this
approach. For this purpose, I did a small experiment to figure out how
much usually file deletion takes [1] on a SSD, for 1million files
8seconds, I'm sure it will be much more on HDD.
The bg cleaner can also be used for RemovePgTempFiles, probably the
postmaster just renaming the pgsql_temp to something
pgsql_temp_delete, then proceeding with the server startup, the bg
cleaner can then delete the files.
Also, we could do something similar for removing/recycling old xlog
files and StartupReorderBuffer.
Another idea could be to parallelize the checkpoint i.e. IIUC, the
tasks that checkpoint do in CheckPointGuts are independent and if we
have some counters like (how many snapshot/mapping files that the
server generated)
[1] on SSD:
deletion of 1000000 files took 7.930380 seconds
deletion of 500000 files took 3.921676 seconds
deletion of 100000 files took 0.768772 seconds
deletion of 50000 files took 0.400623 seconds
deletion of 10000 files took 0.077565 seconds
deletion of 1000 files took 0.006232 seconds
Regards,
Bharath Rupireddy.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-02 02:47 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
@ 2021-12-02 21:31 ` Bossart, Nathan <[email protected]>
2021-12-03 13:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Bossart, Nathan @ 2021-12-02 21:31 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: pgsql-hackers
On 12/1/21, 6:48 PM, "Bharath Rupireddy" <[email protected]> wrote:
> +1 for the overall idea of making the checkpoint faster. In fact, we
> here at our team have been thinking about this problem for a while. If
> there are a lot of files that checkpoint has to loop over and remove,
> IMO, that task can be delegated to someone else (maybe a background
> worker called background cleaner or bg cleaner, of course, we can have
> a GUC to enable or disable it). The checkpoint can just write some
Right. IMO it isn't optimal to have critical things like startup and
checkpointing depend on somewhat-unrelated tasks. I understand the
desire to avoid adding additional processes, and maybe it is a bigger
hammer than what is necessary to reduce the impact, but it seemed like
a natural solution for this problem. That being said, I'm all for
exploring other ways to handle this.
> Another idea could be to parallelize the checkpoint i.e. IIUC, the
> tasks that checkpoint do in CheckPointGuts are independent and if we
> have some counters like (how many snapshot/mapping files that the
> server generated)
Could you elaborate on this? Is your idea that the checkpointer would
create worker processes like autovacuum does?
Nathan
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-02 02:47 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-02 21:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
@ 2021-12-03 13:56 ` Bharath Rupireddy <[email protected]>
2021-12-03 18:20 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Bharath Rupireddy @ 2021-12-03 13:56 UTC (permalink / raw)
To: Bossart, Nathan <[email protected]>; +Cc: pgsql-hackers
On Fri, Dec 3, 2021 at 3:01 AM Bossart, Nathan <[email protected]> wrote:
>
> On 12/1/21, 6:48 PM, "Bharath Rupireddy" <[email protected]> wrote:
> > +1 for the overall idea of making the checkpoint faster. In fact, we
> > here at our team have been thinking about this problem for a while. If
> > there are a lot of files that checkpoint has to loop over and remove,
> > IMO, that task can be delegated to someone else (maybe a background
> > worker called background cleaner or bg cleaner, of course, we can have
> > a GUC to enable or disable it). The checkpoint can just write some
>
> Right. IMO it isn't optimal to have critical things like startup and
> checkpointing depend on somewhat-unrelated tasks. I understand the
> desire to avoid adding additional processes, and maybe it is a bigger
> hammer than what is necessary to reduce the impact, but it seemed like
> a natural solution for this problem. That being said, I'm all for
> exploring other ways to handle this.
Having a generic background cleaner process (controllable via a few
GUCs), which can delete a bunch of files (snapshot, mapping, old WAL,
temp files etc.) or some other task on behalf of the checkpointer,
seems to be the easiest solution.
I'm too open for other ideas.
> > Another idea could be to parallelize the checkpoint i.e. IIUC, the
> > tasks that checkpoint do in CheckPointGuts are independent and if we
> > have some counters like (how many snapshot/mapping files that the
> > server generated)
>
> Could you elaborate on this? Is your idea that the checkpointer would
> create worker processes like autovacuum does?
Yes, I was thinking that the checkpointer creates one or more dynamic
background workers (we can assume one background worker for now) to
delete the files. If a threshold of files crosses (snapshot files
count is more than this threshold), the new worker gets spawned which
would then enumerate the files and delete the unneeded ones, the
checkpointer can proceed with the other tasks and finish the
checkpointing. Having said this, I prefer the background cleaner
approach over the dynamic background worker. The advantage with the
background cleaner being that it can do other tasks (like other kinds
of file deletion).
Another idea could be that, use the existing background writer to do
the file deletion while the checkpoint is happening. But again, this
might cause problems because the bg writer flushing dirty buffers will
get delayed.
Regards,
Bharath Rupireddy.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-02 02:47 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-02 21:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-03 13:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
@ 2021-12-03 18:20 ` Bossart, Nathan <[email protected]>
2021-12-06 11:43 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Bossart, Nathan @ 2021-12-03 18:20 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: pgsql-hackers
On 12/3/21, 5:57 AM, "Bharath Rupireddy" <[email protected]> wrote:
> On Fri, Dec 3, 2021 at 3:01 AM Bossart, Nathan <[email protected]> wrote:
>>
>> On 12/1/21, 6:48 PM, "Bharath Rupireddy" <[email protected]> wrote:
>> > +1 for the overall idea of making the checkpoint faster. In fact, we
>> > here at our team have been thinking about this problem for a while. If
>> > there are a lot of files that checkpoint has to loop over and remove,
>> > IMO, that task can be delegated to someone else (maybe a background
>> > worker called background cleaner or bg cleaner, of course, we can have
>> > a GUC to enable or disable it). The checkpoint can just write some
>>
>> Right. IMO it isn't optimal to have critical things like startup and
>> checkpointing depend on somewhat-unrelated tasks. I understand the
>> desire to avoid adding additional processes, and maybe it is a bigger
>> hammer than what is necessary to reduce the impact, but it seemed like
>> a natural solution for this problem. That being said, I'm all for
>> exploring other ways to handle this.
>
> Having a generic background cleaner process (controllable via a few
> GUCs), which can delete a bunch of files (snapshot, mapping, old WAL,
> temp files etc.) or some other task on behalf of the checkpointer,
> seems to be the easiest solution.
>
> I'm too open for other ideas.
I might hack something together for the separate worker approach, if
for no other reason than to make sure I really understand how these
functions work. If/when a better idea emerges, we can alter course.
Nathan
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-02 02:47 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-02 21:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-03 13:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-03 18:20 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
@ 2021-12-06 11:43 ` Bharath Rupireddy <[email protected]>
2021-12-06 19:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Bharath Rupireddy @ 2021-12-06 11:43 UTC (permalink / raw)
To: Bossart, Nathan <[email protected]>; +Cc: pgsql-hackers
On Fri, Dec 3, 2021 at 11:50 PM Bossart, Nathan <[email protected]> wrote:
>
> On 12/3/21, 5:57 AM, "Bharath Rupireddy" <[email protected]> wrote:
> > On Fri, Dec 3, 2021 at 3:01 AM Bossart, Nathan <[email protected]> wrote:
> >>
> >> On 12/1/21, 6:48 PM, "Bharath Rupireddy" <[email protected]> wrote:
> >> > +1 for the overall idea of making the checkpoint faster. In fact, we
> >> > here at our team have been thinking about this problem for a while. If
> >> > there are a lot of files that checkpoint has to loop over and remove,
> >> > IMO, that task can be delegated to someone else (maybe a background
> >> > worker called background cleaner or bg cleaner, of course, we can have
> >> > a GUC to enable or disable it). The checkpoint can just write some
> >>
> >> Right. IMO it isn't optimal to have critical things like startup and
> >> checkpointing depend on somewhat-unrelated tasks. I understand the
> >> desire to avoid adding additional processes, and maybe it is a bigger
> >> hammer than what is necessary to reduce the impact, but it seemed like
> >> a natural solution for this problem. That being said, I'm all for
> >> exploring other ways to handle this.
> >
> > Having a generic background cleaner process (controllable via a few
> > GUCs), which can delete a bunch of files (snapshot, mapping, old WAL,
> > temp files etc.) or some other task on behalf of the checkpointer,
> > seems to be the easiest solution.
> >
> > I'm too open for other ideas.
>
> I might hack something together for the separate worker approach, if
> for no other reason than to make sure I really understand how these
> functions work. If/when a better idea emerges, we can alter course.
Thanks. As I said upthread we've been discussing the approach of
offloading some of the checkpoint tasks like (deleting snapshot files)
internally for quite some time and I would like to share a patch that
adds a new background cleaner process (currently able to delete the
logical replication snapshot files, if required can be extended to do
other tasks as well). I don't mind if it gets rejected. Please have a
look.
Regards,
Bharath Rupireddy.
Attachments:
[application/octet-stream] v1-0001-background-cleaner-to-offload-checkpoint-tasks.patch (22.1K, ../../CALj2ACUJ8LGwvc=ZzMrW7X6O5nWA11AqPHFjUdDL66kv3qQgpQ@mail.gmail.com/2-v1-0001-background-cleaner-to-offload-checkpoint-tasks.patch)
download | inline diff:
From 4735db532aa818a9e3958ccc79229044fdfc7069 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Mon, 6 Dec 2021 11:39:36 +0000
Subject: [PATCH v1] background cleaner to offload checkpoint tasks
---
src/backend/postmaster/Makefile | 1 +
src/backend/postmaster/bgcleaner.c | 415 ++++++++++++++++++++
src/backend/postmaster/postmaster.c | 34 +-
src/backend/replication/logical/logical.c | 40 ++
src/backend/replication/logical/snapbuild.c | 8 +
src/backend/utils/activity/wait_event.c | 3 +
src/backend/utils/init/miscinit.c | 3 +
src/backend/utils/misc/guc.c | 37 +-
src/include/miscadmin.h | 1 +
src/include/postmaster/bgcleaner.h | 32 ++
src/include/replication/logical.h | 6 +
src/include/utils/wait_event.h | 1 +
12 files changed, 579 insertions(+), 2 deletions(-)
create mode 100644 src/backend/postmaster/bgcleaner.c
create mode 100644 src/include/postmaster/bgcleaner.h
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile
index 787c6a2c3b..f55903dd1a 100644
--- a/src/backend/postmaster/Makefile
+++ b/src/backend/postmaster/Makefile
@@ -15,6 +15,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
autovacuum.o \
auxprocess.o \
+ bgcleaner.o \
bgworker.o \
bgwriter.o \
checkpointer.o \
diff --git a/src/backend/postmaster/bgcleaner.c b/src/backend/postmaster/bgcleaner.c
new file mode 100644
index 0000000000..14d98e48eb
--- /dev/null
+++ b/src/backend/postmaster/bgcleaner.c
@@ -0,0 +1,415 @@
+/*-------------------------------------------------------------------------
+ *
+ * bgcleaner.c
+ *
+ * The background cleaner (bgcleaner) process removes unneeded replication
+ * slot files (.snap). This is to offload the checkpoint responsibility so
+ * that the checkpoint (and so the recovery) can be faster.
+ *
+ *
+ * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/postmaster/bgcleaner.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include <signal.h>
+#include <sys/stat.h>
+#include <sys/time.h>
+#include <unistd.h>
+
+#include "libpq/pqsignal.h"
+#include "miscadmin.h"
+#include "pgstat.h"
+#include "postmaster/bgcleaner.h"
+#include "postmaster/fork_process.h"
+#include "postmaster/interrupt.h"
+#include "postmaster/postmaster.h"
+#include "replication/logical.h"
+#include "storage/dsm.h"
+#include "storage/fd.h"
+#include "storage/pg_shmem.h"
+#include "utils/guc.h"
+#include "utils/ps_status.h"
+
+/*
+ * GUC parameters
+ */
+bool BgCleanerEnable = true;
+bool BgCleanerStopProcessingFiles = false;
+int BgCleanerDelay = 180;
+
+#ifdef EXEC_BACKEND
+static pid_t bgcleaner_forkexec(void);
+#endif
+
+NON_EXEC_STATIC void BackgroundCleanerMain(int argc, char *argv[]) pg_attribute_noreturn();
+
+static void CheckForBgCleanerInterrupts(void);
+static void ProcessSnapshotCutoffFiles(void);
+static int RemoveSnapShotFiles(XLogRecPtr cutoff);
+
+#ifdef EXEC_BACKEND
+/*
+ * bgcleaner_forkexec() -
+ *
+ * Format up the arglist for, then fork and exec, bgcleaner process
+ */
+static pid_t
+bgcleaner_forkexec(void)
+{
+ char *av[10];
+ int ac = 0;
+
+ av[ac++] = "postgres";
+ av[ac++] = "--forkbgcleaner";
+ av[ac++] = NULL; /* filled in by postmaster_forkexec */
+
+ av[ac] = NULL;
+ Assert(ac < lengthof(av));
+
+ return postmaster_forkexec(ac, av);
+}
+#endif /* EXEC_BACKEND */
+
+/*
+ * Called from postmaster at startup or after an existing bgcleaner died.
+ * Attempt to fire up a fresh bgcleaner.
+ *
+ * Returns PID of child process, or 0 if fail.
+ *
+ * Note: if fail, we will be called again from the postmaster main loop.
+ */
+int
+BgCleanerStart(void)
+{
+ pid_t pid;
+
+#ifdef EXEC_BACKEND
+ switch ((pid = bgcleaner_forkexec()))
+#else
+ switch ((pid = fork_process()))
+#endif
+ {
+ case -1:
+ ereport(LOG,
+ (errmsg("could not fork background cleaner: %m")));
+ return 0;
+
+#ifndef EXEC_BACKEND
+ case 0:
+ /* in postmaster child ... */
+ InitPostmasterChild();
+
+ /* Close the postmaster's sockets */
+ ClosePostmasterPorts(false);
+
+ /* Drop our connection to postmaster's shared memory, as well */
+ dsm_detach_all();
+ PGSharedMemoryDetach();
+
+ BackgroundCleanerMain(0, NULL);
+ break;
+#endif
+
+ default:
+ return (int) pid;
+ }
+
+ /* shouldn't get here */
+ return 0;
+}
+
+/*
+ * Main entry point for bgcleaner process
+ *
+ * argc/argv parameters are valid only in EXEC_BACKEND case.
+ */
+NON_EXEC_STATIC void
+BackgroundCleanerMain(int argc, char *argv[])
+{
+ int n_wait = 0;
+ bool msg_logged = false;
+
+ /*
+ * Ignore all signals usually bound to some action in the postmaster,
+ * except SIGHUP, SIGTERM and SIGQUIT. Note we don't need a SIGUSR1
+ * handler to support latch operations, because we only use a local latch.
+ */
+ pqsignal(SIGHUP, SignalHandlerForConfigReload);
+ pqsignal(SIGINT, SIG_IGN);
+ pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
+ pqsignal(SIGQUIT, SignalHandlerForShutdownRequest);
+ pqsignal(SIGALRM, SIG_IGN);
+ pqsignal(SIGPIPE, SIG_IGN);
+ pqsignal(SIGUSR1, SIG_IGN);
+ pqsignal(SIGUSR2, SIG_IGN);
+ pqsignal(SIGCHLD, SIG_DFL);
+ pqsignal(SIGTTIN, SIG_DFL);
+ pqsignal(SIGTTOU, SIG_DFL);
+ pqsignal(SIGCONT, SIG_DFL);
+ pqsignal(SIGWINCH, SIG_DFL);
+ /*
+ * Unblock signals (they were blocked when the postmaster forked us)
+ */
+ PG_SETMASK(&UnBlockSig);
+
+ MyBackendType = B_BG_CLEANER;
+ init_ps_display(NULL);
+
+ /*
+ * Loop until we get SIGQUIT, SIGTERM or detect ungraceful death of
+ * parent postmaster.
+ */
+ for (;;)
+ {
+ int rc;
+
+ /* Clear any already-pending wakeups */
+ ResetLatch(MyLatch);
+
+ CheckForBgCleanerInterrupts();
+
+ /* Do the main work */
+ if (!BgCleanerStopProcessingFiles)
+ {
+ if (msg_logged)
+ {
+ /* we were told to start processing files */
+ elog(LOG, "background cleaner started file processing as parameter \"%s\" is set to off",
+ "bgcleaner_stop_processing_files");
+ msg_logged = false;
+ }
+
+ ProcessSnapshotCutoffFiles();
+ }
+ else if (BgCleanerStopProcessingFiles && !msg_logged)
+ {
+ /* we were told to stop processing files */
+ elog(LOG, "background cleaner stopped file processing as parameter \"%s\" is set to on",
+ "bgcleaner_stop_processing_files");
+ msg_logged = true;
+ }
+
+ /*
+ * Sleep until we are signaled or BgCleanerDelay has elapsed.
+ */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
+ BgCleanerDelay * 1000L /* convert to ms */ ,
+ WAIT_EVENT_BGCLEANER_MAIN);
+
+ /*
+ * Emergency bailout if postmaster has died. This is to avoid the
+ * necessity for manual cleanup of all postmaster children.
+ */
+ if (rc & WL_POSTMASTER_DEATH)
+ break;
+
+ n_wait++;
+
+ /*
+ * Emit a log message after every 5 rounds of sleep to indicate the
+ * process is active.
+ */
+ if (n_wait == 5 && !BgCleanerStopProcessingFiles)
+ {
+ elog(LOG, "background cleaner is running with %d seconds of sleep time between rounds",
+ BgCleanerDelay);
+
+ n_wait = 0;
+ }
+ }
+
+ exit(0);
+}
+
+/*
+ * Read snapshot cutoff file names written by checkpointer to get the cutoff
+ * LSN and remove the unneded snapshot files.
+ */
+static void
+ProcessSnapshotCutoffFiles(void)
+{
+ DIR *dir;
+ struct dirent *cutoff_de;
+
+ dir = AllocateDir("pg_logical");
+ while ((cutoff_de = ReadDir(dir, "pg_logical")) != NULL)
+ {
+ char path[MAXPGPATH + 11];
+ uint32 hi;
+ uint32 lo;
+ XLogRecPtr cutoff = InvalidXLogRecPtr;
+ XLogRecPtr prev_cutoff = InvalidXLogRecPtr;
+ struct stat statbuf;
+ int res;
+
+ CheckForBgCleanerInterrupts();
+
+ /* see if we were told to stop processing files */
+ if (BgCleanerStopProcessingFiles)
+ {
+ elog(LOG, "background cleaner is stopping file processing at cutoff LSN: %X/%X as parameter \"%s\" is set to on",
+ LSN_FORMAT_ARGS(prev_cutoff),
+ "bgcleaner_stop_processing_files");
+ return;
+ }
+
+ if (strcmp(cutoff_de->d_name, ".") == 0 ||
+ strcmp(cutoff_de->d_name, "..") == 0)
+ continue;
+
+ snprintf(path, sizeof(path), "pg_logical/%s", cutoff_de->d_name);
+
+ if (lstat(path, &statbuf) == 0 && !S_ISREG(statbuf.st_mode))
+ {
+ elog(DEBUG2, "only regular files expected: %s", cutoff_de->d_name);
+ continue;
+ }
+
+ /*
+ * We just log a message if a file doesn't fit the pattern, it's
+ * probably some editors lock/state file or similar...
+ */
+ if (sscanf(cutoff_de->d_name, "snapshot_cutoff_%X-%X", &hi, &lo) != 2)
+ {
+ elog(DEBUG2, "could not parse file name: %s", cutoff_de->d_name);
+ continue;
+ }
+
+ prev_cutoff = cutoff;
+ cutoff = ((uint64) hi) << 32 | lo;
+ elog(DEBUG2, "replication slots cutoff LSN: %X/%X",
+ LSN_FORMAT_ARGS(cutoff));
+
+ res = RemoveSnapShotFiles(cutoff);
+
+ if (res == 0)
+ {
+ /* remove the file */
+ if (unlink(path) < 0)
+ {
+ elog(LOG, "could not remove snapshot cutoff file: %s",
+ cutoff_de->d_name);
+ continue;
+ }
+
+ elog(DEBUG1, "removed snapshot cutoff file: %s", cutoff_de->d_name);
+ }
+ else if (res < 0)
+ {
+ elog(DEBUG2, "retained snapshot cutoff file: %s", cutoff_de->d_name);
+ continue;
+ }
+ }
+ FreeDir(dir);
+}
+
+/*
+ * Remove unneded snapshot files i.e. files with LSN < cutoff LSN.
+ *
+ * Return value -1 indicates that the caller can not remove the snapshot file.
+ *
+ * Return value 0 indicates that the caller can delete the snapshot cutoff
+ * file.
+ */
+static int
+RemoveSnapShotFiles(XLogRecPtr cutoff)
+{
+ DIR *dir;
+ struct dirent *snap_de;
+ int res = 0;
+ uint32 snap_files_deleted = 0;
+
+ dir = AllocateDir("pg_logical/snapshots");
+ while ((snap_de = ReadDir(dir, "pg_logical/snapshots")) != NULL)
+ {
+ char path[MAXPGPATH + 21];
+ uint32 hi;
+ uint32 lo;
+ XLogRecPtr lsn;
+ struct stat statbuf;
+
+ CheckForBgCleanerInterrupts();
+
+ /* see if we were told to stop processing files */
+ if (BgCleanerStopProcessingFiles)
+ {
+ elog(LOG, "background cleaner is stopping file processing at cutoff LSN: %X/%X as parameter \"%s\" is set to on",
+ LSN_FORMAT_ARGS(cutoff),
+ "bgcleaner_stop_processing_files");
+ return -1;
+ }
+
+ if (strcmp(snap_de->d_name, ".") == 0 ||
+ strcmp(snap_de->d_name, "..") == 0)
+ continue;
+
+ snprintf(path, sizeof(path), "pg_logical/snapshots/%s", snap_de->d_name);
+
+ if (lstat(path, &statbuf) == 0 && !S_ISREG(statbuf.st_mode))
+ continue;
+
+ /*
+ * We just log a message if a file doesn't fit the pattern, it's
+ * probably some editors lock/state file or similar...
+ */
+ if (sscanf(snap_de->d_name, "%X-%X.snap", &hi, &lo) != 2)
+ {
+ elog(DEBUG2, "could not parse file name: %s", snap_de->d_name);
+ continue;
+ }
+
+ lsn = ((uint64) hi) << 32 | lo;
+
+ /* check whether we still need it */
+ if (lsn < cutoff || cutoff == InvalidXLogRecPtr)
+ {
+ /* remove the file */
+ if (unlink(path) < 0)
+ {
+ elog(LOG, "could not remove snapshot file: %s",
+ snap_de->d_name);
+ res = -1;
+ continue;
+ }
+
+ snap_files_deleted++;
+ elog(DEBUG1, "removed snapshot file: %s", snap_de->d_name);
+ }
+ else
+ {
+ elog(DEBUG2, "retained snapshot file: %s", snap_de->d_name);
+ continue;
+ }
+ }
+ FreeDir(dir);
+
+ if (snap_files_deleted > 0)
+ ereport(LOG,
+ (errmsg_plural("removed %u snapshot file with cutoff LSN %X/%X",
+ "removed %u snapshot files with cutoff LSN %X/%X",
+ snap_files_deleted,
+ snap_files_deleted,
+ LSN_FORMAT_ARGS(cutoff))));
+ return res;
+}
+
+static void
+CheckForBgCleanerInterrupts(void)
+{
+ if (ShutdownRequestPending)
+ exit(0);
+
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 328ecafa8c..49325570e5 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -110,6 +110,7 @@
#include "port/pg_bswap.h"
#include "postmaster/autovacuum.h"
#include "postmaster/auxprocess.h"
+#include "postmaster/bgcleaner.h"
#include "postmaster/bgworker_internals.h"
#include "postmaster/fork_process.h"
#include "postmaster/interrupt.h"
@@ -255,7 +256,8 @@ static pid_t StartupPID = 0,
AutoVacPID = 0,
PgArchPID = 0,
PgStatPID = 0,
- SysLoggerPID = 0;
+ SysLoggerPID = 0,
+ BgCleanerPID = 0;
/* Startup process's status */
typedef enum
@@ -1459,6 +1461,8 @@ PostmasterMain(int argc, char *argv[])
CheckpointerPID = StartCheckpointer();
if (BgWriterPID == 0)
BgWriterPID = StartBackgroundWriter();
+ if (BgCleanerPID == 0 && BgCleanerEnable)
+ BgCleanerPID = BgCleanerStart();
/*
* We're ready to rock and roll...
@@ -1828,6 +1832,8 @@ ServerLoop(void)
CheckpointerPID = StartCheckpointer();
if (BgWriterPID == 0)
BgWriterPID = StartBackgroundWriter();
+ if (BgCleanerPID == 0 && BgCleanerEnable)
+ BgCleanerPID = BgCleanerStart();
}
/*
@@ -2794,6 +2800,8 @@ SIGHUP_handler(SIGNAL_ARGS)
signal_child(SysLoggerPID, SIGHUP);
if (PgStatPID != 0)
signal_child(PgStatPID, SIGHUP);
+ if (BgCleanerPID != 0)
+ signal_child(BgCleanerPID, SIGHUP);
/* Reload authentication config files too */
if (!load_hba())
@@ -3111,6 +3119,8 @@ reaper(SIGNAL_ARGS)
CheckpointerPID = StartCheckpointer();
if (BgWriterPID == 0)
BgWriterPID = StartBackgroundWriter();
+ if (BgCleanerPID == 0 && BgCleanerEnable)
+ BgCleanerPID = BgCleanerStart();
if (WalWriterPID == 0)
WalWriterPID = StartWalWriter();
@@ -3225,6 +3235,22 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the bgcleaner? If so, just try to start a new one; no need
+ * to force reset of the rest of the system. (If fail, we'll try again
+ * in future cycles of the main loop.)
+ */
+ if (pid == BgCleanerPID)
+ {
+ BgCleanerPID = 0;
+ if (!EXIT_STATUS_0(exitstatus))
+ LogChildExit(LOG, _("background cleaner process"),
+ pid, exitstatus);
+ if (BgCleanerPID == 0 && BgCleanerEnable)
+ BgCleanerPID = BgCleanerStart();
+ continue;
+ }
+
/*
* Was it the wal receiver? If exit status is zero (normal) or one
* (FATAL exit), we assume everything is all right just like normal
@@ -5175,6 +5201,12 @@ SubPostmasterMain(int argc, char *argv[])
SysLoggerMain(argc, argv); /* does not return */
}
+ if (strcmp(argv[1], "--forkbgcleaner") == 0)
+ {
+ /* Do not want to attach to shared memory */
+
+ BackgroundCleanerMain(argc, argv); /* does not return */
+ }
abort(); /* shouldn't get here */
}
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 10cbdea124..7b947a7428 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -28,6 +28,8 @@
#include "postgres.h"
+#include <unistd.h>
+
#include "access/xact.h"
#include "access/xlog_internal.h"
#include "fmgr.h"
@@ -1842,3 +1844,41 @@ UpdateDecodingStats(LogicalDecodingContext *ctx)
rb->totalTxns = 0;
rb->totalBytes = 0;
}
+
+void
+CreateReplicationCleanupFile(ReplCleanupFileKind kind, XLogRecPtr cutoff_lsn)
+{
+ int fd;
+ /*
+ * 27 is size of the fixed path name "pg_logical/snapshot_cutoff_".
+ * XXX: Dynamically allocate memory for the path variable, if at all, this
+ * function is changed to deal with other kinds of files with differnt
+ * fixed path names.
+ */
+ char path[MAXPGPATH + 27];
+
+ Assert(kind == REPL_CLEANUP_FILE_SNAPSHOT);
+
+ MemSet(path, '\0', sizeof(path));
+
+ /* create a file named snapshot_cutoff_<hi>-<lo> */
+ snprintf(path, sizeof(path), "pg_logical/snapshot_cutoff_%X-%X",
+ LSN_FORMAT_ARGS(cutoff_lsn));
+
+ /*
+ * We don't need O_EXCL flag as it might cause FATAL error if the file
+ * already exists. It is the responsibility of the clean up prgoram to
+ * delete the previous files.
+ */
+ fd = BasicOpenFile(path, O_RDWR | O_CREAT);
+ if (fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m", path)));
+
+ /* make sure we persist */
+ fsync_fname(path, false);
+ fsync_fname("pg_logical", true);
+
+ close(fd);
+}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index dbdc172a2b..3933923d52 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -125,6 +125,7 @@
#include "access/xact.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/bgcleaner.h"
#include "replication/logical.h"
#include "replication/reorderbuffer.h"
#include "replication/snapbuild.h"
@@ -1941,6 +1942,13 @@ CheckPointSnapBuild(void)
if (redo < cutoff)
cutoff = redo;
+ /* do this only if there exists a background cleaner */
+ if (BgCleanerEnable && !BgCleanerStopProcessingFiles)
+ {
+ CreateReplicationCleanupFile(REPL_CLEANUP_FILE_SNAPSHOT, cutoff);
+ return;
+ }
+
snap_dir = AllocateDir("pg_logical/snapshots");
while ((snap_de = ReadDir(snap_dir, "pg_logical/snapshots")) != NULL)
{
diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c
index 4d53f040e8..f0fc681e5f 100644
--- a/src/backend/utils/activity/wait_event.c
+++ b/src/backend/utils/activity/wait_event.c
@@ -215,6 +215,9 @@ pgstat_get_wait_activity(WaitEventActivity w)
case WAIT_EVENT_AUTOVACUUM_MAIN:
event_name = "AutoVacuumMain";
break;
+ case WAIT_EVENT_BGCLEANER_MAIN:
+ event_name = "BackgroundCleanerMain";
+ break;
case WAIT_EVENT_BGWRITER_HIBERNATE:
event_name = "BgWriterHibernate";
break;
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 88801374b5..f734c82229 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -264,6 +264,9 @@ GetBackendTypeDesc(BackendType backendType)
case B_BACKEND:
backendDesc = "client backend";
break;
+ case B_BG_CLEANER:
+ backendDesc = "background cleaner";
+ break;
case B_BG_WORKER:
backendDesc = "background worker";
break;
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index ee6a838b3a..29eef8f155 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -68,6 +68,9 @@
#include "parser/scansup.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
+#ifdef AZURE_SERVICE_FABRIC
+#include "postmaster/bgcleaner.h"
+#endif
#include "postmaster/bgworker_internals.h"
#include "postmaster/bgwriter.h"
#include "postmaster/postmaster.h"
@@ -1791,7 +1794,26 @@ static struct config_bool ConfigureNamesBool[] =
false,
NULL, NULL, NULL
},
-
+#ifdef AZURE_SERVICE_FABRIC
+ {
+ {"bgcleaner_enable", PGC_POSTMASTER, DEVELOPER_OPTIONS,
+ gettext_noop("Start a subprocess to remove unneeded replication slot snapshot files."),
+ NULL
+ },
+ &BgCleanerEnable,
+ true,
+ NULL, NULL, NULL
+ },
+ {
+ {"bgcleaner_stop_processing_files", PGC_SIGHUP, DEVELOPER_OPTIONS,
+ gettext_noop("Inform background cleaner to stop processing files."),
+ NULL
+ },
+ &BgCleanerStopProcessingFiles,
+ false,
+ NULL, NULL, NULL
+ },
+#endif
#ifdef TRACE_SORT
{
{"trace_sort", PGC_USERSET, DEVELOPER_OPTIONS,
@@ -3065,6 +3087,19 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+#ifdef AZURE_SERVICE_FABRIC
+ {
+ {"bgcleaner_delay", PGC_SIGHUP, DEVELOPER_OPTIONS,
+ gettext_noop("Background cleaner sleep time between rounds."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &BgCleanerDelay,
+ 180, 60, 86400,
+ NULL, NULL, NULL
+ },
+#endif
+
{
{"effective_io_concurrency",
PGC_USERSET,
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 90a3016065..5449f07a80 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -326,6 +326,7 @@ typedef enum BackendType
B_AUTOVAC_LAUNCHER,
B_AUTOVAC_WORKER,
B_BACKEND,
+ B_BG_CLEANER,
B_BG_WORKER,
B_BG_WRITER,
B_CHECKPOINTER,
diff --git a/src/include/postmaster/bgcleaner.h b/src/include/postmaster/bgcleaner.h
new file mode 100644
index 0000000000..5bba615ec8
--- /dev/null
+++ b/src/include/postmaster/bgcleaner.h
@@ -0,0 +1,32 @@
+/*-------------------------------------------------------------------------
+ *
+ * bgcleaner.h
+ * Exports from postmaster/bgcleaner.c.
+ *
+ * The bgcleaner process removes unneeded replication slot files (.snap).
+ * This is to offload the checkpoint responsibility so that the checkpoint
+ * (and so the recovery) can be faster.
+ *
+ * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
+ *
+ * src/include/postmaster/bgcleaner.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _BGCLEANER_H
+#define _BGCLEANER_H
+
+/*
+ * GUC parameters
+ */
+extern bool BgCleanerEnable;
+extern bool BgCleanerStopProcessingFiles;
+extern int BgCleanerDelay;
+
+extern int BgCleanerStart(void);
+
+#ifdef EXEC_BACKEND
+extern void BackgroundCleanerMain(int argc, char *argv[]) pg_attribute_noreturn();
+#endif
+
+#endif /* _BGCLEANER_H */
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index e0f513b773..9971dc81f7 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -16,6 +16,11 @@
struct LogicalDecodingContext;
+typedef enum ReplCleanupFileKind
+{
+ REPL_CLEANUP_FILE_SNAPSHOT = 1
+} ReplCleanupFileKind;
+
typedef void (*LogicalOutputPluginWriterWrite) (struct LogicalDecodingContext *lr,
XLogRecPtr Ptr,
TransactionId xid,
@@ -140,5 +145,6 @@ extern bool filter_prepare_cb_wrapper(LogicalDecodingContext *ctx,
extern bool filter_by_origin_cb_wrapper(LogicalDecodingContext *ctx, RepOriginId origin_id);
extern void ResetLogicalStreamingState(void);
extern void UpdateDecodingStats(LogicalDecodingContext *ctx);
+extern void CreateReplicationCleanupFile(ReplCleanupFileKind kind, XLogRecPtr cutoff_lsn);
#endif
diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h
index 8785a8e12c..be90cd0fe1 100644
--- a/src/include/utils/wait_event.h
+++ b/src/include/utils/wait_event.h
@@ -37,6 +37,7 @@ typedef enum
{
WAIT_EVENT_ARCHIVER_MAIN = PG_WAIT_ACTIVITY,
WAIT_EVENT_AUTOVACUUM_MAIN,
+ WAIT_EVENT_BGCLEANER_MAIN,
WAIT_EVENT_BGWRITER_HIBERNATE,
WAIT_EVENT_BGWRITER_MAIN,
WAIT_EVENT_CHECKPOINTER_MAIN,
--
2.25.1
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-02 02:47 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-02 21:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-03 13:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-03 18:20 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-06 11:43 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
@ 2021-12-06 19:22 ` Bossart, Nathan <[email protected]>
2021-12-10 19:03 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Bossart, Nathan @ 2021-12-06 19:22 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: pgsql-hackers
On 12/6/21, 3:44 AM, "Bharath Rupireddy" <[email protected]> wrote:
> On Fri, Dec 3, 2021 at 11:50 PM Bossart, Nathan <[email protected]> wrote:
>> I might hack something together for the separate worker approach, if
>> for no other reason than to make sure I really understand how these
>> functions work. If/when a better idea emerges, we can alter course.
>
> Thanks. As I said upthread we've been discussing the approach of
> offloading some of the checkpoint tasks like (deleting snapshot files)
> internally for quite some time and I would like to share a patch that
> adds a new background cleaner process (currently able to delete the
> logical replication snapshot files, if required can be extended to do
> other tasks as well). I don't mind if it gets rejected. Please have a
> look.
Thanks for sharing! I've also spent some time on a patch set, which I
intend to share once I have handling for all four tasks (so far I have
handling for CheckPointSnapBuild() and RemovePgTempFiles()). I'll
take a look at your patch as well.
Nathan
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-02 02:47 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-02 21:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-03 13:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-03 18:20 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-06 11:43 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-06 19:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
@ 2021-12-10 19:03 ` Bossart, Nathan <[email protected]>
2021-12-13 13:53 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Bossart, Nathan @ 2021-12-10 19:03 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: pgsql-hackers
On 12/6/21, 11:23 AM, "Bossart, Nathan" <[email protected]> wrote:
> On 12/6/21, 3:44 AM, "Bharath Rupireddy" <[email protected]> wrote:
>> Thanks. As I said upthread we've been discussing the approach of
>> offloading some of the checkpoint tasks like (deleting snapshot files)
>> internally for quite some time and I would like to share a patch that
>> adds a new background cleaner process (currently able to delete the
>> logical replication snapshot files, if required can be extended to do
>> other tasks as well). I don't mind if it gets rejected. Please have a
>> look.
>
> Thanks for sharing! I've also spent some time on a patch set, which I
> intend to share once I have handling for all four tasks (so far I have
> handling for CheckPointSnapBuild() and RemovePgTempFiles()). I'll
> take a look at your patch as well.
Well, I haven't had a chance to look at your patch, and my patch set
still only has handling for CheckPointSnapBuild() and
RemovePgTempFiles(), but I thought I'd share what I have anyway. I
split it into 5 patches:
0001 - Adds a new "custodian" auxiliary process that does nothing.
0002 - During startup, remove the pgsql_tmp directories instead of
only clearing the contents.
0003 - Split temporary file cleanup during startup into two stages.
The first renames the directories, and the second clears them.
0004 - Moves the second stage from 0003 to the custodian process.
0005 - Moves CheckPointSnapBuild() to the custodian process.
This is still very much a work in progress, and I've done minimal
testing so far.
Nathan
Attachments:
[application/octet-stream] v1-0005-Move-removal-of-old-serialized-snapshots-to-custo.patch (3.8K, ../../[email protected]/2-v1-0005-Move-removal-of-old-serialized-snapshots-to-custo.patch)
download | inline diff:
From b1014d699b82e5eb5bbe05a1b3b294577df1f77c Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Sun, 5 Dec 2021 22:02:40 -0800
Subject: [PATCH v1 5/5] Move removal of old serialized snapshots to custodian.
This was only done during checkpoints because it was a convenient
place to put it. However, if there are many snapshots to remove,
it can significantly extend checkpoint time. To avoid this, move
this work to the newly-introduced custodian process.
---
src/backend/access/transam/xlog.c | 2 --
src/backend/postmaster/custodian.c | 11 +++++++++++
src/backend/replication/logical/snapbuild.c | 9 ++++-----
src/include/replication/snapbuild.h | 2 +-
4 files changed, 16 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index d894af310a..2fcca38c23 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -56,7 +56,6 @@
#include "replication/logical.h"
#include "replication/origin.h"
#include "replication/slot.h"
-#include "replication/snapbuild.h"
#include "replication/walreceiver.h"
#include "replication/walsender.h"
#include "storage/bufmgr.h"
@@ -9602,7 +9601,6 @@ CheckPointGuts(XLogRecPtr checkPointRedo, int flags)
{
CheckPointRelationMap();
CheckPointReplicationSlots();
- CheckPointSnapBuild();
CheckPointLogicalRewriteHeap();
CheckPointReplicationOrigin();
diff --git a/src/backend/postmaster/custodian.c b/src/backend/postmaster/custodian.c
index a5443f9a21..b088cdb0b8 100644
--- a/src/backend/postmaster/custodian.c
+++ b/src/backend/postmaster/custodian.c
@@ -38,6 +38,7 @@
#include "pgstat.h"
#include "postmaster/custodian.h"
#include "postmaster/interrupt.h"
+#include "replication/snapbuild.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/proc.h"
@@ -204,6 +205,16 @@ CustodianMain(void)
*/
RemovePgTempFiles(false, false);
+ /*
+ * Remove serialized snapshots that are no longer required by any
+ * logical replication slot.
+ *
+ * It is not important for these to be removed in single-user mode, so
+ * we don't need any extra hndling outside of the custodian process for
+ * this.
+ */
+ RemoveOldSerializedSnapshots();
+
/* Calculate how long to sleep */
end_time = (pg_time_t) time(NULL);
elapsed_secs = end_time - start_time;
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index dbdc172a2b..0fa7d822e1 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1912,14 +1912,13 @@ snapshot_not_interesting:
/*
* Remove all serialized snapshots that are not required anymore because no
- * slot can need them. This doesn't actually have to run during a checkpoint,
- * but it's a convenient point to schedule this.
+ * slot can need them.
*
- * NB: We run this during checkpoints even if logical decoding is disabled so
- * we cleanup old slots at some point after it got disabled.
+ * NB: We run this even if logical decoding is disabled so we cleanup old slots
+ * at some point after it got disabled.
*/
void
-CheckPointSnapBuild(void)
+RemoveOldSerializedSnapshots(void)
{
XLogRecPtr cutoff;
XLogRecPtr redo;
diff --git a/src/include/replication/snapbuild.h b/src/include/replication/snapbuild.h
index 82aa86125b..ba7276058d 100644
--- a/src/include/replication/snapbuild.h
+++ b/src/include/replication/snapbuild.h
@@ -57,7 +57,7 @@ struct ReorderBuffer;
struct xl_heap_new_cid;
struct xl_running_xacts;
-extern void CheckPointSnapBuild(void);
+extern void RemoveOldSerializedSnapshots(void);
extern SnapBuild *AllocateSnapshotBuilder(struct ReorderBuffer *cache,
TransactionId xmin_horizon, XLogRecPtr start_lsn,
--
2.16.6
[application/octet-stream] v1-0004-Move-pgsql_tmp-file-removal-to-custodian-process.patch (5.1K, ../../[email protected]/3-v1-0004-Move-pgsql_tmp-file-removal-to-custodian-process.patch)
download | inline diff:
From bc755b58a982956e2a494fab91a49c0142f84030 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Sun, 5 Dec 2021 21:42:52 -0800
Subject: [PATCH v1 4/5] Move pgsql_tmp file removal to custodian process.
With this change, startup (and restart after a crash) simply
renames the pgsql_tmp directories, and the custodian process
actually removes all the files in the staged directories as well as
the staged directories themselves. This should help avoid long
startup delays due to many leftover temporary files.
---
src/backend/postmaster/custodian.c | 13 ++++++++++++-
src/backend/postmaster/postmaster.c | 14 +++++++++-----
src/backend/storage/file/fd.c | 22 ++++++++++++++++------
3 files changed, 37 insertions(+), 12 deletions(-)
diff --git a/src/backend/postmaster/custodian.c b/src/backend/postmaster/custodian.c
index 0ba59949bb..a5443f9a21 100644
--- a/src/backend/postmaster/custodian.c
+++ b/src/backend/postmaster/custodian.c
@@ -191,7 +191,18 @@ CustodianMain(void)
start_time = (pg_time_t) time(NULL);
- /* TODO: offloaded tasks go here */
+ /*
+ * Remove any pgsql_tmp directories that have been staged for deletion.
+ * Since pgsql_tmp directories can accumulate many files, removing all
+ * of the files during startup (which we used to do) can take a very
+ * long time. To avoid delaying startup, we simply have startup rename
+ * the temporary directories, and we clean them up here.
+ *
+ * pgsql_tmp directories are not staged or cleaned in single-user mode,
+ * so we don't need any extra handling outside of the custodian process
+ * for this.
+ */
+ RemovePgTempFiles(false, false);
/* Calculate how long to sleep */
end_time = (pg_time_t) time(NULL);
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 1ae2dc179e..b098482496 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -1391,9 +1391,11 @@ PostmasterMain(int argc, char *argv[])
/*
* Remove old temporary files. At this point there can be no other
* Postgres processes running in this directory, so this should be safe.
+ *
+ * Note that this just stages the pgsql_tmp directories for deletion. The
+ * custodian process is responsible for actually removing the files.
*/
RemovePgTempFiles(true, true);
- RemovePgTempFiles(false, false);
/*
* Initialize stats collection subsystem (this does NOT start the
@@ -4139,12 +4141,14 @@ PostmasterStateMachine(void)
ereport(LOG,
(errmsg("all server processes terminated; reinitializing")));
- /* remove leftover temporary files after a crash */
+ /*
+ * Remove leftover temporary files after a crash.
+ *
+ * Note that this just stages the pgsql_tmp directories for deletion.
+ * The custodian process is responsible for actually removing the files.
+ */
if (remove_temp_files_after_crash)
- {
RemovePgTempFiles(true, true);
- RemovePgTempFiles(false, false);
- }
/* allow background workers to immediately restart */
ResetBackgroundWorkerCrashTimes();
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 633c6eee18..0807f9c590 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -100,6 +100,8 @@
#include "postmaster/startup.h"
#include "storage/fd.h"
#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
#include "utils/guc.h"
#include "utils/resowner_private.h"
@@ -1640,9 +1642,9 @@ PathNameOpenFilePerm(const char *fileName, int fileFlags, mode_t fileMode)
*
* Directories created within the top-level temporary directory should begin
* with PG_TEMP_FILE_PREFIX, so that they can be identified as temporary and
- * deleted at startup by RemovePgTempFiles(). Further subdirectories below
- * that do not need any particular prefix.
-*/
+ * deleted by RemovePgTempFiles(). Further subdirectories below that do not
+ * need any particular prefix.
+ */
void
PathNameCreateTemporaryDir(const char *basedir, const char *directory)
{
@@ -1840,9 +1842,9 @@ OpenTemporaryFileInTablespace(Oid tblspcOid, bool rejectError)
*
* If the file is inside the top-level temporary directory, its name should
* begin with PG_TEMP_FILE_PREFIX so that it can be identified as temporary
- * and deleted at startup by RemovePgTempFiles(). Alternatively, it can be
- * inside a directory created with PathNameCreateTemporaryDir(), in which case
- * the prefix isn't needed.
+ * and deleted by RemovePgTempFiles(). Alternatively, it can be inside a
+ * directory created with PathNameCreateTemporaryDir(), in which case the prefix
+ * isn't needed.
*/
File
PathNameCreateTemporaryFile(const char *path, bool error_on_failure)
@@ -3211,6 +3213,14 @@ RemovePgTempFiles(bool stage, bool remove_relation_files)
* would create a race condition. It's done separately, earlier in
* postmaster startup.
*/
+
+ /*
+ * If we just staged some pgsql_tmp directories for removal, wake up the
+ * custodian process so that it deletes all the files in the staged
+ * directories as well as the directories themselves.
+ */
+ if (stage && ProcGlobal->custodianLatch)
+ SetLatch(ProcGlobal->custodianLatch);
}
/*
--
2.16.6
[application/octet-stream] v1-0003-Split-pgsql_tmp-cleanup-into-two-stages.patch (9.8K, ../../[email protected]/4-v1-0003-Split-pgsql_tmp-cleanup-into-two-stages.patch)
download | inline diff:
From 400d3ef141593e02c7703534e8c909f42fad3a2b Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Sun, 5 Dec 2021 21:16:44 -0800
Subject: [PATCH v1 3/5] Split pgsql_tmp cleanup into two stages.
First, pgsql_tmp directories will be renamed to stage them for
removal. Then, all files in pgsql_tmp are removed before removing
the staged directories themselves. This change is being made in
preparation for a follow-up change to offload most temporary file
cleanup to the new custodian process.
Note that temporary relation files cannot be cleaned up via the
aforementioned strategy and will not be offloaded to the custodian.
---
src/backend/postmaster/postmaster.c | 8 +-
src/backend/storage/file/fd.c | 176 +++++++++++++++++++++++++++++++-----
src/include/storage/fd.h | 2 +-
3 files changed, 162 insertions(+), 24 deletions(-)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 51613aaa2a..1ae2dc179e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -1392,7 +1392,8 @@ PostmasterMain(int argc, char *argv[])
* Remove old temporary files. At this point there can be no other
* Postgres processes running in this directory, so this should be safe.
*/
- RemovePgTempFiles();
+ RemovePgTempFiles(true, true);
+ RemovePgTempFiles(false, false);
/*
* Initialize stats collection subsystem (this does NOT start the
@@ -4140,7 +4141,10 @@ PostmasterStateMachine(void)
/* remove leftover temporary files after a crash */
if (remove_temp_files_after_crash)
- RemovePgTempFiles();
+ {
+ RemovePgTempFiles(true, true);
+ RemovePgTempFiles(false, false);
+ }
/* allow background workers to immediately restart */
ResetBackgroundWorkerCrashTimes();
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 545e91978c..633c6eee18 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -112,6 +112,8 @@
#define PG_FLUSH_DATA_WORKS 1
#endif
+#define PG_TEMP_DIR_TO_REMOVE_PREFIX (PG_TEMP_FILES_DIR "_to_remove_")
+
/*
* We must leave some file descriptors free for system(), the dynamic loader,
* and other code that tries to open files without consulting fd.c. This
@@ -338,6 +340,8 @@ static void BeforeShmemExit_Files(int code, Datum arg);
static void CleanupTempFiles(bool isCommit, bool isProcExit);
static void RemovePgTempRelationFiles(const char *tsdirname);
static void RemovePgTempRelationFilesInDbspace(const char *dbspacedirname);
+static void StagePgTempDirForRemoval(const char *tmp_dir);
+static void RemoveStagedPgTempDirs(const char *spc_dir);
static void walkdir(const char *path,
void (*action) (const char *fname, bool isdir, int elevel),
@@ -3133,24 +3137,20 @@ CleanupTempFiles(bool isCommit, bool isProcExit)
* Remove temporary and temporary relation files left over from a prior
* postmaster session
*
- * This should be called during postmaster startup. It will forcibly
- * remove any leftover files created by OpenTemporaryFile and any leftover
- * temporary relation files created by mdcreate.
+ * If stage is true, this function will simply rename all pgsql_tmp directories
+ * to stage them for removal at a later time. If stage is false, this function
+ * will delete all files in the staged directories as well as the directories
+ * themselves.
*
- * During post-backend-crash restart cycle, this routine is called when
- * remove_temp_files_after_crash GUC is enabled. Multiple crashes while
- * queries are using temp files could result in useless storage usage that can
- * only be reclaimed by a service restart. The argument against enabling it is
- * that someone might want to examine the temporary files for debugging
- * purposes. This does however mean that OpenTemporaryFile had better allow for
- * collision with an existing temp file name.
+ * If remove_relation_files is true, this function will remove the temporary
+ * relation files. Otherwise, this step is skipped.
*
* NOTE: this function and its subroutines generally report syscall failures
* with ereport(LOG) and keep going. Removing temp files is not so critical
* that we should fail to start the database when we can't do it.
*/
void
-RemovePgTempFiles(void)
+RemovePgTempFiles(bool stage, bool remove_relation_files)
{
char temp_path[MAXPGPATH + 10 + sizeof(TABLESPACE_VERSION_DIRECTORY) + sizeof(PG_TEMP_FILES_DIR)];
DIR *spc_dir;
@@ -3159,9 +3159,16 @@ RemovePgTempFiles(void)
/*
* First process temp files in pg_default ($PGDATA/base)
*/
- snprintf(temp_path, sizeof(temp_path), "base/%s", PG_TEMP_FILES_DIR);
- RemovePgTempDir(temp_path, true, false);
- RemovePgTempRelationFiles("base");
+ if (stage)
+ {
+ snprintf(temp_path, sizeof(temp_path), "base/%s", PG_TEMP_FILES_DIR);
+ StagePgTempDirForRemoval(temp_path);
+ }
+ else
+ RemoveStagedPgTempDirs("base");
+
+ if (remove_relation_files)
+ RemovePgTempRelationFiles("base");
/*
* Cycle through temp directories for all non-default tablespaces.
@@ -3174,13 +3181,26 @@ RemovePgTempFiles(void)
strcmp(spc_de->d_name, "..") == 0)
continue;
- snprintf(temp_path, sizeof(temp_path), "pg_tblspc/%s/%s/%s",
- spc_de->d_name, TABLESPACE_VERSION_DIRECTORY, PG_TEMP_FILES_DIR);
- RemovePgTempDir(temp_path, true, false);
+ if (stage)
+ {
+ snprintf(temp_path, sizeof(temp_path), "pg_tblspc/%s/%s/%s",
+ spc_de->d_name, TABLESPACE_VERSION_DIRECTORY,
+ PG_TEMP_FILES_DIR);
+ StagePgTempDirForRemoval(temp_path);
+ }
+ else
+ {
+ snprintf(temp_path, sizeof(temp_path), "pg_tblspc/%s/%s",
+ spc_de->d_name, TABLESPACE_VERSION_DIRECTORY);
+ RemoveStagedPgTempDirs(temp_path);
+ }
- snprintf(temp_path, sizeof(temp_path), "pg_tblspc/%s/%s",
- spc_de->d_name, TABLESPACE_VERSION_DIRECTORY);
- RemovePgTempRelationFiles(temp_path);
+ if (remove_relation_files)
+ {
+ snprintf(temp_path, sizeof(temp_path), "pg_tblspc/%s/%s",
+ spc_de->d_name, TABLESPACE_VERSION_DIRECTORY);
+ RemovePgTempRelationFiles(temp_path);
+ }
}
FreeDir(spc_dir);
@@ -3194,7 +3214,121 @@ RemovePgTempFiles(void)
}
/*
- * Process one pgsql_tmp directory for RemovePgTempFiles.
+ * StagePgTempDirForRemoval
+ *
+ * This function renames the given directory with a special prefix that
+ * RemoveStagedPgTempDirs() will know to look for. An integer is appended to
+ * the end of the new directory name in case previously staged pgsql_tmp
+ * directories have not yet been removed.
+ */
+static void
+StagePgTempDirForRemoval(const char *tmp_dir)
+{
+ DIR *dir;
+ char stage_path[MAXPGPATH * 2];
+ char parent_path[MAXPGPATH * 2];
+
+ /*
+ * If tmp_dir doesn't exist, there is nothing to stage.
+ */
+ dir = AllocateDir(tmp_dir);
+ if (dir == NULL)
+ {
+ if (errno != ENOENT)
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not open directory \"%s\": %m", tmp_dir)));
+ return;
+ }
+ FreeDir(dir);
+
+ strlcpy(parent_path, tmp_dir, MAXPGPATH * 2);
+ get_parent_directory(parent_path);
+
+ /*
+ * get_parent_directory() returns an empty string if the input argument is
+ * just a file name (see comments in path.c), so handle that as being the
+ * current directory.
+ */
+ if (strlen(parent_path) == 0)
+ strlcpy(parent_path, ".", MAXPGPATH * 2);
+
+ /*
+ * Find a name for the stage directory. We just increment an integer at the
+ * end of the name until we find one that doesn't exist.
+ */
+ for (int n = 0; n <= INT_MAX; n++)
+ {
+ snprintf(stage_path, sizeof(stage_path), "%s/%s%d", parent_path,
+ PG_TEMP_DIR_TO_REMOVE_PREFIX, n);
+
+ dir = AllocateDir(stage_path);
+ if (dir == NULL)
+ {
+ if (errno == ENOENT)
+ break;
+
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not open directory \"%s\": %m",
+ stage_path)));
+ return;
+ }
+ FreeDir(dir);
+
+ stage_path[0] = '\0';
+ }
+
+ /*
+ * In the unlikely event that we couldn't find a name for the stage
+ * directory, bail out.
+ */
+ if (stage_path[0] == '\0')
+ {
+ ereport(LOG,
+ (errmsg("could not stage \"%s\" for deletion",
+ tmp_dir)));
+ return;
+ }
+
+ /*
+ * Rename the temporary directory.
+ */
+ if (rename(tmp_dir, stage_path) != 0)
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not rename directory \"%s\" to \"%s\": %m",
+ tmp_dir, stage_path)));
+}
+
+/*
+ * RemoveStagedPgTempDirs
+ *
+ * This function removes all pgsql_tmp directories that have been staged for
+ * removal by StagePgTempDirForRemoval() in the given tablespace directory.
+ */
+static void
+RemoveStagedPgTempDirs(const char *spc_dir)
+{
+ char temp_path[MAXPGPATH * 2];
+ DIR *dir;
+ struct dirent *de;
+
+ dir = AllocateDir(spc_dir);
+ while ((de = ReadDirExtended(dir, spc_dir, LOG)) != NULL)
+ {
+ if (strncmp(de->d_name, PG_TEMP_DIR_TO_REMOVE_PREFIX,
+ strlen(PG_TEMP_DIR_TO_REMOVE_PREFIX)) != 0)
+ continue;
+
+ snprintf(temp_path, sizeof(temp_path), "%s/%s", spc_dir, de->d_name);
+ RemovePgTempDir(temp_path, true, false);
+ }
+ FreeDir(dir);
+}
+
+/*
+ * Process one pgsql_tmp directory for RemoveStagedPgTempDirs.
*
* If missing_ok is true, it's all right for the named directory to not exist.
* Any other problem results in a LOG message. (missing_ok should be true at
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index 762f6b46c1..85fa987aca 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -168,7 +168,7 @@ extern Oid GetNextTempTableSpace(void);
extern void AtEOXact_Files(bool isCommit);
extern void AtEOSubXact_Files(bool isCommit, SubTransactionId mySubid,
SubTransactionId parentSubid);
-extern void RemovePgTempFiles(void);
+extern void RemovePgTempFiles(bool stage, bool remove_relation_files);
extern void RemovePgTempDir(const char *tmpdirname, bool missing_ok,
bool unlink_all);
extern bool looks_like_temp_rel_name(const char *name);
--
2.16.6
[application/octet-stream] v1-0002-Also-remove-pgsql_tmp-directories-during-startup.patch (4.8K, ../../[email protected]/5-v1-0002-Also-remove-pgsql_tmp-directories-during-startup.patch)
download | inline diff:
From 9b4683c6d3ad798f02ba0d084d26ddc0283d1ef1 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Sun, 5 Dec 2021 19:38:20 -0800
Subject: [PATCH v1 2/5] Also remove pgsql_tmp directories during startup.
Presently, the server only removes the contents of the temporary
directories during startup, not the directory itself. This changes
that to prepare for future commits that will move temporary file
cleanup to a separate auxiliary process.
---
src/backend/postmaster/postmaster.c | 2 +-
src/backend/storage/file/fd.c | 20 ++++++++++----------
src/include/storage/fd.h | 4 ++--
src/test/recovery/t/022_crash_temp_files.pl | 6 ++++--
4 files changed, 17 insertions(+), 15 deletions(-)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 635313cdb7..51613aaa2a 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -1117,7 +1117,7 @@ PostmasterMain(int argc, char *argv[])
* safe to do so now, because we verified earlier that there are no
* conflicting Postgres processes in this data directory.
*/
- RemovePgTempFilesInDir(PG_TEMP_FILES_DIR, true, false);
+ RemovePgTempDir(PG_TEMP_FILES_DIR, true, false);
#endif
/*
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 263057841d..545e91978c 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -3160,7 +3160,7 @@ RemovePgTempFiles(void)
* First process temp files in pg_default ($PGDATA/base)
*/
snprintf(temp_path, sizeof(temp_path), "base/%s", PG_TEMP_FILES_DIR);
- RemovePgTempFilesInDir(temp_path, true, false);
+ RemovePgTempDir(temp_path, true, false);
RemovePgTempRelationFiles("base");
/*
@@ -3176,7 +3176,7 @@ RemovePgTempFiles(void)
snprintf(temp_path, sizeof(temp_path), "pg_tblspc/%s/%s/%s",
spc_de->d_name, TABLESPACE_VERSION_DIRECTORY, PG_TEMP_FILES_DIR);
- RemovePgTempFilesInDir(temp_path, true, false);
+ RemovePgTempDir(temp_path, true, false);
snprintf(temp_path, sizeof(temp_path), "pg_tblspc/%s/%s",
spc_de->d_name, TABLESPACE_VERSION_DIRECTORY);
@@ -3209,7 +3209,7 @@ RemovePgTempFiles(void)
* them separate.)
*/
void
-RemovePgTempFilesInDir(const char *tmpdirname, bool missing_ok, bool unlink_all)
+RemovePgTempDir(const char *tmpdirname, bool missing_ok, bool unlink_all)
{
DIR *temp_dir;
struct dirent *temp_de;
@@ -3247,13 +3247,7 @@ RemovePgTempFilesInDir(const char *tmpdirname, bool missing_ok, bool unlink_all)
if (S_ISDIR(statbuf.st_mode))
{
/* recursively remove contents, then directory itself */
- RemovePgTempFilesInDir(rm_path, false, true);
-
- if (rmdir(rm_path) < 0)
- ereport(LOG,
- (errcode_for_file_access(),
- errmsg("could not remove directory \"%s\": %m",
- rm_path)));
+ RemovePgTempDir(rm_path, false, true);
}
else
{
@@ -3271,6 +3265,12 @@ RemovePgTempFilesInDir(const char *tmpdirname, bool missing_ok, bool unlink_all)
}
FreeDir(temp_dir);
+
+ if (rmdir(tmpdirname) < 0)
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not remove directory \"%s\": %m",
+ tmpdirname)));
}
/* Process one tablespace directory, look for per-DB subdirectories */
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index 34602ae006..762f6b46c1 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -169,8 +169,8 @@ extern void AtEOXact_Files(bool isCommit);
extern void AtEOSubXact_Files(bool isCommit, SubTransactionId mySubid,
SubTransactionId parentSubid);
extern void RemovePgTempFiles(void);
-extern void RemovePgTempFilesInDir(const char *tmpdirname, bool missing_ok,
- bool unlink_all);
+extern void RemovePgTempDir(const char *tmpdirname, bool missing_ok,
+ bool unlink_all);
extern bool looks_like_temp_rel_name(const char *name);
extern int pg_fsync(int fd);
diff --git a/src/test/recovery/t/022_crash_temp_files.pl b/src/test/recovery/t/022_crash_temp_files.pl
index bf95a30761..481f1f23a2 100644
--- a/src/test/recovery/t/022_crash_temp_files.pl
+++ b/src/test/recovery/t/022_crash_temp_files.pl
@@ -143,7 +143,8 @@ $node->poll_query_until('postgres', undef, '');
# Check for temporary files
is( $node->safe_psql(
- 'postgres', 'SELECT COUNT(1) FROM pg_ls_dir($$base/pgsql_tmp$$)'),
+ 'postgres',
+ 'SELECT COUNT(1) FROM pg_ls_dir($$base$$) WHERE pg_ls_dir = \'pgsql_tmp\''),
qq(0),
'no temporary files');
@@ -241,7 +242,8 @@ $node->restart();
# Check the temporary files -- should be gone
is( $node->safe_psql(
- 'postgres', 'SELECT COUNT(1) FROM pg_ls_dir($$base/pgsql_tmp$$)'),
+ 'postgres',
+ 'SELECT COUNT(1) FROM pg_ls_dir($$base$$) WHERE pg_ls_dir = \'pgsql_tmp\''),
qq(0),
'temporary file was removed');
--
2.16.6
[application/octet-stream] v1-0001-Introduce-custodian.patch (18.6K, ../../[email protected]/6-v1-0001-Introduce-custodian.patch)
download | inline diff:
From e17e79bfe150fa137d345dc7c6848c4e596c2fa4 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Sun, 5 Dec 2021 15:07:28 -0800
Subject: [PATCH v1 1/5] Introduce custodian.
The custodian process is a new auxiliary process that is intended
to help offload tasks could otherwise delay startup and
checkpointing. This commit simply adds the new process; it does
not yet do anything useful.
---
src/backend/postmaster/Makefile | 1 +
src/backend/postmaster/auxprocess.c | 8 ++
src/backend/postmaster/custodian.c | 210 ++++++++++++++++++++++++++++++++
src/backend/postmaster/postmaster.c | 44 ++++++-
src/backend/storage/lmgr/proc.c | 1 +
src/backend/utils/activity/wait_event.c | 3 +
src/backend/utils/init/miscinit.c | 3 +
src/include/miscadmin.h | 3 +
src/include/postmaster/custodian.h | 17 +++
src/include/storage/proc.h | 11 +-
src/include/utils/wait_event.h | 1 +
11 files changed, 297 insertions(+), 5 deletions(-)
create mode 100644 src/backend/postmaster/custodian.c
create mode 100644 src/include/postmaster/custodian.h
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile
index 787c6a2c3b..7ec7b23467 100644
--- a/src/backend/postmaster/Makefile
+++ b/src/backend/postmaster/Makefile
@@ -18,6 +18,7 @@ OBJS = \
bgworker.o \
bgwriter.o \
checkpointer.o \
+ custodian.o \
fork_process.o \
interrupt.o \
pgarch.o \
diff --git a/src/backend/postmaster/auxprocess.c b/src/backend/postmaster/auxprocess.c
index 7452f908b2..c55cc84490 100644
--- a/src/backend/postmaster/auxprocess.c
+++ b/src/backend/postmaster/auxprocess.c
@@ -20,6 +20,7 @@
#include "pgstat.h"
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
+#include "postmaster/custodian.h"
#include "postmaster/startup.h"
#include "postmaster/walwriter.h"
#include "replication/walreceiver.h"
@@ -74,6 +75,9 @@ AuxiliaryProcessMain(AuxProcType auxtype)
case CheckpointerProcess:
MyBackendType = B_CHECKPOINTER;
break;
+ case CustodianProcess:
+ MyBackendType = B_CUSTODIAN;
+ break;
case WalWriterProcess:
MyBackendType = B_WAL_WRITER;
break;
@@ -153,6 +157,10 @@ AuxiliaryProcessMain(AuxProcType auxtype)
CheckpointerMain();
proc_exit(1);
+ case CustodianProcess:
+ CustodianMain();
+ proc_exit(1);
+
case WalWriterProcess:
InitXLOGAccess();
WalWriterMain();
diff --git a/src/backend/postmaster/custodian.c b/src/backend/postmaster/custodian.c
new file mode 100644
index 0000000000..0ba59949bb
--- /dev/null
+++ b/src/backend/postmaster/custodian.c
@@ -0,0 +1,210 @@
+/*-------------------------------------------------------------------------
+ *
+ * custodian.c
+ *
+ * The custodian process is new as of Postgres 15. It's main purpose is to
+ * offload tasks that could otherwise delay startup and checkpointing, but
+ * it needn't be restricted to just those things. Offloaded tasks should
+ * not be synchronous (e.g., checkpointing shouldn't need to wait for the
+ * custodian to complete a task before proceeding). Also, ensure that any
+ * offloaded tasks are either not required during single-user mode or are
+ * performed separately during single-user mode.
+ *
+ * The custodian is not an essential process and can shutdown quickly when
+ * requested. The custodian will wake up approximately once every 5
+ * minutes to perform its tasks, but backends can (and should) set its
+ * latch to wake it up sooner.
+ *
+ * Normal termination is by SIGTERM, which instructs the bgwriter to
+ * exit(0). Emergency termination is by SIGQUIT; like any backend, the
+ * custodian will simply abort and exit on SIGQUIT.
+ *
+ * If the custodian exits unexpectedly, the postmaster treats that the same
+ * as a backend crash: shared memory may be corrupted, so remaining
+ * backends should be killed by SIGQUIT and then a recovery cycle started.
+ *
+ *
+ * Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/postmaster/custodian.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "libpq/pqsignal.h"
+#include "pgstat.h"
+#include "postmaster/custodian.h"
+#include "postmaster/interrupt.h"
+#include "storage/bufmgr.h"
+#include "storage/condition_variable.h"
+#include "storage/proc.h"
+#include "storage/procsignal.h"
+#include "utils/memutils.h"
+
+#define CUSTODIAN_TIMEOUT_S (300) /* 5 minutes */
+
+/*
+ * Main entry point for custodian process
+ *
+ * This is invoked from AuxiliaryProcessMain, which has already created the
+ * basic execution environment, but not enabled signals yet.
+ */
+void
+CustodianMain(void)
+{
+ sigjmp_buf local_sigjmp_buf;
+ MemoryContext custodian_context;
+
+ /*
+ * Properly accept or ignore signals that might be sent to us.
+ */
+ pqsignal(SIGHUP, SignalHandlerForConfigReload);
+ pqsignal(SIGINT, SignalHandlerForShutdownRequest);
+ pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
+ /* SIGQUIT handler was already set up by InitPostmasterChild */
+ pqsignal(SIGALRM, SIG_IGN);
+ pqsignal(SIGPIPE, SIG_IGN);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
+ pqsignal(SIGUSR2, SIG_IGN);
+
+ /*
+ * Reset some signals that are accepted by postmaster but not here
+ */
+ pqsignal(SIGCHLD, SIG_DFL);
+
+ /*
+ * Create a memory context that we will do all our work in. We do this so
+ * that we can reset the context during error recovery and thereby avoid
+ * possible memory leaks.
+ */
+ custodian_context = AllocSetContextCreate(TopMemoryContext,
+ "Custodian",
+ ALLOCSET_DEFAULT_SIZES);
+ MemoryContextSwitchTo(custodian_context);
+
+ /*
+ * If an exception is encountered, processing resumes here.
+ *
+ * You might wonder why this isn't coded as an infinite loop around a
+ * PG_TRY construct. The reason is that this is the bottom of the
+ * exception stack, and so with PG_TRY there would be no exception handler
+ * in force at all during the CATCH part. By leaving the outermost setjmp
+ * always active, we have at least some chance of recovering from an error
+ * during error recovery. (If we get into an infinite loop thereby, it
+ * will soon be stopped by overflow of elog.c's internal state stack.)
+ *
+ * Note that we use sigsetjmp(..., 1), so that the prevailing signal mask
+ * (to wit, BlockSig) will be restored when longjmp'ing to here. Thus,
+ * signals other than SIGQUIT will be blocked until we complete error
+ * recovery. It might seem that this policy makes the HOLD_INTERRUPS()
+ * call redundant, but it is not since InterruptPending might be set
+ * already.
+ */
+ if (sigsetjmp(local_sigjmp_buf, 1) != 0)
+ {
+ /* Since not using PG_TRY, must reset error stack by hand */
+ error_context_stack = NULL;
+
+ /* Prevent interrupts while cleaning up */
+ HOLD_INTERRUPTS();
+
+ /* Report the error to the server log */
+ EmitErrorReport();
+
+ /* These operations are really just a minimal subset of
+ * AbortTransaction(). We don't have very many resources to worry
+ * about.
+ */
+ LWLockReleaseAll();
+ ConditionVariableCancelSleep();
+ pgstat_report_wait_end();
+ AbortBufferIO();
+ UnlockBuffers();
+ ReleaseAuxProcessResources(false);
+ AtEOXact_Buffers(false);
+ AtEOXact_SMgr();
+ AtEOXact_Files(false);
+ AtEOXact_HashTables(false);
+
+ /*
+ * Now return to normal top-level context and clear ErrorContext for
+ * next time.
+ */
+ MemoryContextSwitchTo(custodian_context);
+ FlushErrorState();
+
+ /* Flush any leaked data in the top-level context */
+ MemoryContextResetAndDeleteChildren(custodian_context);
+
+ /* Now we can allow interrupts again */
+ RESUME_INTERRUPTS();
+
+ /*
+ * Sleep at least 1 second after any error. A write error is likely
+ * to be repeated, and we don't want to be filling the error logs as
+ * fast as we can.
+ */
+ pg_usleep(1000000L);
+
+ /*
+ * Close all open files after any error. This is helpful on Windows,
+ * where holding deleted files open causes various strange errors.
+ * It's not clear we need it elsewhere, but shouldn't hurt.
+ */
+ smgrcloseall();
+
+ /* Report wait end here, when there is no further possibility of wait */
+ pgstat_report_wait_end();
+ }
+
+ /* We can now handle ereport(ERROR) */
+ PG_exception_stack = &local_sigjmp_buf;
+
+ /*
+ * Unblock signals (they were blocked when the postmaster forked us)
+ */
+ PG_SETMASK(&UnBlockSig);
+
+ /*
+ * Advertise out latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->custodianLatch = &MyProc->procLatch;
+
+ /*
+ * Loop forever
+ */
+ for (;;)
+ {
+ pg_time_t start_time;
+ pg_time_t end_time;
+ int elapsed_secs;
+ int cur_timeout;
+
+ /* Clear any already-pending wakeups */
+ ResetLatch(MyLatch);
+
+ HandleMainLoopInterrupts();
+
+ start_time = (pg_time_t) time(NULL);
+
+ /* TODO: offloaded tasks go here */
+
+ /* Calculate how long to sleep */
+ end_time = (pg_time_t) time(NULL);
+ elapsed_secs = end_time - start_time;
+ if (elapsed_secs >= CUSTODIAN_TIMEOUT_S)
+ continue; /* no sleep for us */
+ cur_timeout = CUSTODIAN_TIMEOUT_S - elapsed_secs;
+
+ (void) WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ cur_timeout * 1000L /* convert to ms */ ,
+ WAIT_EVENT_CUSTODIAN_MAIN);
+ }
+
+ pg_unreachable();
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 328ecafa8c..635313cdb7 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -250,6 +250,7 @@ bool remove_temp_files_after_crash = true;
static pid_t StartupPID = 0,
BgWriterPID = 0,
CheckpointerPID = 0,
+ CustodianPID = 0,
WalWriterPID = 0,
WalReceiverPID = 0,
AutoVacPID = 0,
@@ -556,6 +557,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
+#define StartCustodian() StartChildProcess(CustodianProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
#define StartWalReceiver() StartChildProcess(WalReceiverProcess)
@@ -1819,13 +1821,16 @@ ServerLoop(void)
/*
* If no background writer process is running, and we are not in a
* state that prevents it, start one. It doesn't matter if this
- * fails, we'll just try again later. Likewise for the checkpointer.
+ * fails, we'll just try again later. Likewise for the checkpointer
+ * and custodian.
*/
if (pmState == PM_RUN || pmState == PM_RECOVERY ||
pmState == PM_HOT_STANDBY || pmState == PM_STARTUP)
{
if (CheckpointerPID == 0)
CheckpointerPID = StartCheckpointer();
+ if (CustodianPID == 0)
+ CustodianPID = StartCustodian();
if (BgWriterPID == 0)
BgWriterPID = StartBackgroundWriter();
}
@@ -2782,6 +2787,8 @@ SIGHUP_handler(SIGNAL_ARGS)
signal_child(BgWriterPID, SIGHUP);
if (CheckpointerPID != 0)
signal_child(CheckpointerPID, SIGHUP);
+ if (CustodianPID != 0)
+ signal_child(CustodianPID, SIGHUP);
if (WalWriterPID != 0)
signal_child(WalWriterPID, SIGHUP);
if (WalReceiverPID != 0)
@@ -3109,6 +3116,8 @@ reaper(SIGNAL_ARGS)
*/
if (CheckpointerPID == 0)
CheckpointerPID = StartCheckpointer();
+ if (CustodianPID == 0)
+ CustodianPID = StartCustodian();
if (BgWriterPID == 0)
BgWriterPID = StartBackgroundWriter();
if (WalWriterPID == 0)
@@ -3211,6 +3220,20 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the custodian? Normal exit can be ignored; we'll start a
+ * new one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
+ */
+ if (pid == CustodianPID)
+ {
+ CustodianPID = 0;
+ if (!EXIT_STATUS_0(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("custodian process"));
+ continue;
+ }
+
/*
* Was it the wal writer? Normal exit can be ignored; we'll start a
* new one at the next iteration of the postmaster's main loop, if
@@ -3684,6 +3707,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(CheckpointerPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the custodian too */
+ if (pid == CustodianPID)
+ CustodianPID = 0;
+ else if (CustodianPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) CustodianPID)));
+ signal_child(CustodianPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/* Take care of the walwriter too */
if (pid == WalWriterPID)
WalWriterPID = 0;
@@ -3887,6 +3922,9 @@ PostmasterStateMachine(void)
/* and the bgwriter too */
if (BgWriterPID != 0)
signal_child(BgWriterPID, SIGTERM);
+ /* and the custodian too */
+ if (CustodianPID != 0)
+ signal_child(CustodianPID, SIGTERM);
/* and the walwriter too */
if (WalWriterPID != 0)
signal_child(WalWriterPID, SIGTERM);
@@ -3924,6 +3962,7 @@ PostmasterStateMachine(void)
BgWriterPID == 0 &&
(CheckpointerPID == 0 ||
(!FatalError && Shutdown < ImmediateShutdown)) &&
+ CustodianPID == 0 &&
WalWriterPID == 0 &&
AutoVacPID == 0)
{
@@ -4017,6 +4056,7 @@ PostmasterStateMachine(void)
Assert(WalReceiverPID == 0);
Assert(BgWriterPID == 0);
Assert(CheckpointerPID == 0);
+ Assert(CustodianPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
/* syslogger is not considered here */
@@ -4222,6 +4262,8 @@ TerminateChildren(int signal)
signal_child(BgWriterPID, signal);
if (CheckpointerPID != 0)
signal_child(CheckpointerPID, signal);
+ if (CustodianPID != 0)
+ signal_child(CustodianPID, signal);
if (WalWriterPID != 0)
signal_child(WalWriterPID, signal);
if (WalReceiverPID != 0)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index b7d9da0aa9..a86a05adb4 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->custodianLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c
index 4d53f040e8..530af294d9 100644
--- a/src/backend/utils/activity/wait_event.c
+++ b/src/backend/utils/activity/wait_event.c
@@ -224,6 +224,9 @@ pgstat_get_wait_activity(WaitEventActivity w)
case WAIT_EVENT_CHECKPOINTER_MAIN:
event_name = "CheckpointerMain";
break;
+ case WAIT_EVENT_CUSTODIAN_MAIN:
+ event_name = "CustodianMain";
+ break;
case WAIT_EVENT_LOGICAL_APPLY_MAIN:
event_name = "LogicalApplyMain";
break;
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 88801374b5..90c4160d42 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -273,6 +273,9 @@ GetBackendTypeDesc(BackendType backendType)
case B_CHECKPOINTER:
backendDesc = "checkpointer";
break;
+ case B_CUSTODIAN:
+ backendDesc = "custodian";
+ break;
case B_STARTUP:
backendDesc = "startup";
break;
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 90a3016065..83089d23ff 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -329,6 +329,7 @@ typedef enum BackendType
B_BG_WORKER,
B_BG_WRITER,
B_CHECKPOINTER,
+ B_CUSTODIAN,
B_STARTUP,
B_WAL_RECEIVER,
B_WAL_SENDER,
@@ -433,6 +434,7 @@ typedef enum
BgWriterProcess,
ArchiverProcess,
CheckpointerProcess,
+ CustodianProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -445,6 +447,7 @@ extern AuxProcType MyAuxProcType;
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
+#define AmCustodianProcess() (MyAuxProcType == CustodianProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/custodian.h b/src/include/postmaster/custodian.h
new file mode 100644
index 0000000000..e8ac2ad3dd
--- /dev/null
+++ b/src/include/postmaster/custodian.h
@@ -0,0 +1,17 @@
+/*-------------------------------------------------------------------------
+ *
+ * custodian.h
+ * Exports from postmaster/custodian.c.
+ *
+ * Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ * src/include/postmaster/custodian.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _CUSTODIAN_H
+#define _CUSTODIAN_H
+
+extern void CustodianMain(void) pg_attribute_noreturn();
+
+#endif /* _CUSTODIAN_H */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index cfabfdbedf..1fc4599941 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Custodian process's latch */
+ Latch *custodianLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -377,11 +379,12 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer, WAL writer and archiver run during normal
- * operation. Startup process and WAL receiver also consume 2 slots, but WAL
- * writer is launched only after startup has exited, so we only need 5 slots.
+ * Background writer, checkpointer, custodian, WAL writer and archiver run
+ * during normal operation. Startup process and WAL receiver also consume 2
+ * slots, but WAL writer is launched only after startup has exited, so we only
+ * need 6 slots.
*/
-#define NUM_AUXILIARY_PROCS 5
+#define NUM_AUXILIARY_PROCS 6
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h
index 8785a8e12c..08dc9d5caa 100644
--- a/src/include/utils/wait_event.h
+++ b/src/include/utils/wait_event.h
@@ -40,6 +40,7 @@ typedef enum
WAIT_EVENT_BGWRITER_HIBERNATE,
WAIT_EVENT_BGWRITER_MAIN,
WAIT_EVENT_CHECKPOINTER_MAIN,
+ WAIT_EVENT_CUSTODIAN_MAIN,
WAIT_EVENT_LOGICAL_APPLY_MAIN,
WAIT_EVENT_LOGICAL_LAUNCHER_MAIN,
WAIT_EVENT_PGSTAT_MAIN,
--
2.16.6
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-02 02:47 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-02 21:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-03 13:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-03 18:20 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-06 11:43 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-06 19:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-10 19:03 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
@ 2021-12-13 13:53 ` Robert Haas <[email protected]>
2021-12-13 17:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Justin Pryzby <[email protected]>
2021-12-13 18:21 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
0 siblings, 2 replies; 59+ messages in thread
From: Robert Haas @ 2021-12-13 13:53 UTC (permalink / raw)
To: Bossart, Nathan <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; pgsql-hackers
On Fri, Dec 10, 2021 at 2:03 PM Bossart, Nathan <[email protected]> wrote:
> Well, I haven't had a chance to look at your patch, and my patch set
> still only has handling for CheckPointSnapBuild() and
> RemovePgTempFiles(), but I thought I'd share what I have anyway. I
> split it into 5 patches:
>
> 0001 - Adds a new "custodian" auxiliary process that does nothing.
> 0002 - During startup, remove the pgsql_tmp directories instead of
> only clearing the contents.
> 0003 - Split temporary file cleanup during startup into two stages.
> The first renames the directories, and the second clears them.
> 0004 - Moves the second stage from 0003 to the custodian process.
> 0005 - Moves CheckPointSnapBuild() to the custodian process.
I don't know whether this kind of idea is good or not.
One thing we've seen a number of times now is that entrusting the same
process with multiple responsibilities often ends poorly. Sometimes
it's busy with one thing when another thing really needs to be done
RIGHT NOW. Perhaps that won't be an issue here since all of these
things are related to checkpointing, but then the process name should
reflect that rather than making it sound like we can just keep piling
more responsibilities onto this process indefinitely. At some point
that seems bound to become an issue.
Another issue is that we don't want to increase the number of
processes without bound. Processes use memory and CPU resources and if
we run too many of them it becomes a burden on the system. Low-end
systems may not have too many resources in total, and high-end systems
can struggle to fit demanding workloads within the resources that they
have. Maybe it would be cheaper to do more things at once if we were
using threads rather than processes, but that day still seems fairly
far off.
But against all that, if these tasks are slowing down checkpoints and
that's avoidable, that seems pretty important too. Interestingly, I
can't say that I've ever seen any of these things be a problem for
checkpoint or startup speed. I wonder why you've had a different
experience.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-02 02:47 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-02 21:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-03 13:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-03 18:20 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-06 11:43 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-06 19:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-10 19:03 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-13 13:53 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
@ 2021-12-13 17:19 ` Justin Pryzby <[email protected]>
2021-12-13 18:30 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
1 sibling, 1 reply; 59+ messages in thread
From: Justin Pryzby @ 2021-12-13 17:19 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; pgsql-hackers
On Mon, Dec 13, 2021 at 08:53:37AM -0500, Robert Haas wrote:
> On Fri, Dec 10, 2021 at 2:03 PM Bossart, Nathan <[email protected]> wrote:
> > Well, I haven't had a chance to look at your patch, and my patch set
> > still only has handling for CheckPointSnapBuild() and
> > RemovePgTempFiles(), but I thought I'd share what I have anyway. I
> > split it into 5 patches:
> >
> > 0001 - Adds a new "custodian" auxiliary process that does nothing.
...
>
> I don't know whether this kind of idea is good or not.
...
>
> Another issue is that we don't want to increase the number of
> processes without bound. Processes use memory and CPU resources and if
> we run too many of them it becomes a burden on the system. Low-end
> systems may not have too many resources in total, and high-end systems
> can struggle to fit demanding workloads within the resources that they
> have. Maybe it would be cheaper to do more things at once if we were
> using threads rather than processes, but that day still seems fairly
> far off.
Maybe that's an argument that this should be a dynamic background worker
instead of an auxilliary process. Then maybe it would be controlled by
max_parallel_maintenance_workers (or something similar). The checkpointer
would need to do these tasks itself if parallel workers were disabled or
couldn't be launched.
--
Justin
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-02 02:47 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-02 21:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-03 13:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-03 18:20 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-06 11:43 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-06 19:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-10 19:03 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-13 13:53 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2021-12-13 17:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Justin Pryzby <[email protected]>
@ 2021-12-13 18:30 ` Bossart, Nathan <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Bossart, Nathan @ 2021-12-13 18:30 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; Robert Haas <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; pgsql-hackers
On 12/13/21, 9:20 AM, "Justin Pryzby" <[email protected]> wrote:
> On Mon, Dec 13, 2021 at 08:53:37AM -0500, Robert Haas wrote:
>> Another issue is that we don't want to increase the number of
>> processes without bound. Processes use memory and CPU resources and if
>> we run too many of them it becomes a burden on the system. Low-end
>> systems may not have too many resources in total, and high-end systems
>> can struggle to fit demanding workloads within the resources that they
>> have. Maybe it would be cheaper to do more things at once if we were
>> using threads rather than processes, but that day still seems fairly
>> far off.
>
> Maybe that's an argument that this should be a dynamic background worker
> instead of an auxilliary process. Then maybe it would be controlled by
> max_parallel_maintenance_workers (or something similar). The checkpointer
> would need to do these tasks itself if parallel workers were disabled or
> couldn't be launched.
I think this is an interesting idea. I dislike the prospect of having
two code paths for all this stuff, but if it addresses the concerns
about resource usage, maybe it's worth it.
Nathan
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-02 02:47 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-02 21:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-03 13:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-03 18:20 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-06 11:43 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-06 19:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-10 19:03 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-13 13:53 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
@ 2021-12-13 18:21 ` Bossart, Nathan <[email protected]>
2021-12-13 20:36 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
1 sibling, 1 reply; 59+ messages in thread
From: Bossart, Nathan @ 2021-12-13 18:21 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; pgsql-hackers
On 12/13/21, 5:54 AM, "Robert Haas" <[email protected]> wrote:
> I don't know whether this kind of idea is good or not.
Thanks for chiming in. I have an almost-complete patch set that I'm
hoping to post to the lists in the next couple of days.
> One thing we've seen a number of times now is that entrusting the same
> process with multiple responsibilities often ends poorly. Sometimes
> it's busy with one thing when another thing really needs to be done
> RIGHT NOW. Perhaps that won't be an issue here since all of these
> things are related to checkpointing, but then the process name should
> reflect that rather than making it sound like we can just keep piling
> more responsibilities onto this process indefinitely. At some point
> that seems bound to become an issue.
Two of the tasks are cleanup tasks that checkpointing handles at the
moment, and two are cleanup tasks that are done at startup. For now,
all of these tasks are somewhat nonessential. There's no requirement
that any of these tasks complete in order to finish startup or
checkpointing. In fact, outside of preventing the server from running
out of disk space, I don't think there's any requirement that these
tasks run at all. IMO this would have to be a core tenet of a new
auxiliary process like this.
That being said, I totally understand your point. If there were a
dozen such tasks handled by a single auxiliary process, that could
cause a new set of problems. Your checkpointing and startup might be
fast, but you might run out of disk space because our cleanup process
can't handle it all. So a new worker could end up becoming an
availability risk as well.
> Another issue is that we don't want to increase the number of
> processes without bound. Processes use memory and CPU resources and if
> we run too many of them it becomes a burden on the system. Low-end
> systems may not have too many resources in total, and high-end systems
> can struggle to fit demanding workloads within the resources that they
> have. Maybe it would be cheaper to do more things at once if we were
> using threads rather than processes, but that day still seems fairly
> far off.
I do agree that it is important to be very careful about adding new
processes, and if a better idea for how to handle these tasks emerges,
I will readily abandon my current approach. Upthread, Andres
mentioned optimizing unnecessary snapshot files, and I mentioned
possibly limiting how much time startup and checkpoints spend on these
tasks. I don't have too many details for the former, and for the
latter, I'm worried about not being able to keep up. But if the
prospect of adding a new auxiliary process for this stuff is a non-
starter, perhaps I should explore that approach some more.
> But against all that, if these tasks are slowing down checkpoints and
> that's avoidable, that seems pretty important too. Interestingly, I
> can't say that I've ever seen any of these things be a problem for
> checkpoint or startup speed. I wonder why you've had a different
> experience.
Yeah, it's difficult for me to justify why users should suffer long
periods of downtime because startup or checkpointing is taking a very
long time doing things that are arguably unrelated to startup and
checkpointing.
Nathan
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-02 02:47 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-02 21:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-03 13:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-03 18:20 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-06 11:43 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-06 19:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-10 19:03 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-13 13:53 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2021-12-13 18:21 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
@ 2021-12-13 20:36 ` Robert Haas <[email protected]>
2021-12-13 23:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Robert Haas @ 2021-12-13 20:36 UTC (permalink / raw)
To: Bossart, Nathan <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; pgsql-hackers
On Mon, Dec 13, 2021 at 1:21 PM Bossart, Nathan <[email protected]> wrote:
> > But against all that, if these tasks are slowing down checkpoints and
> > that's avoidable, that seems pretty important too. Interestingly, I
> > can't say that I've ever seen any of these things be a problem for
> > checkpoint or startup speed. I wonder why you've had a different
> > experience.
>
> Yeah, it's difficult for me to justify why users should suffer long
> periods of downtime because startup or checkpointing is taking a very
> long time doing things that are arguably unrelated to startup and
> checkpointing.
Well sure. But I've never actually seen that happen.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-02 02:47 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-02 21:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-03 13:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-03 18:20 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-06 11:43 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-06 19:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-10 19:03 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-13 13:53 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2021-12-13 18:21 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-13 20:36 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
@ 2021-12-13 23:05 ` Bossart, Nathan <[email protected]>
2021-12-14 16:59 ` Re: O(n) tasks cause lengthy startups and checkpoints Bruce Momjian <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Bossart, Nathan @ 2021-12-13 23:05 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; pgsql-hackers
On 12/13/21, 12:37 PM, "Robert Haas" <[email protected]> wrote:
> On Mon, Dec 13, 2021 at 1:21 PM Bossart, Nathan <[email protected]> wrote:
>> > But against all that, if these tasks are slowing down checkpoints and
>> > that's avoidable, that seems pretty important too. Interestingly, I
>> > can't say that I've ever seen any of these things be a problem for
>> > checkpoint or startup speed. I wonder why you've had a different
>> > experience.
>>
>> Yeah, it's difficult for me to justify why users should suffer long
>> periods of downtime because startup or checkpointing is taking a very
>> long time doing things that are arguably unrelated to startup and
>> checkpointing.
>
> Well sure. But I've never actually seen that happen.
I'll admit that surprises me. As noted elsewhere [0], we were seeing
this enough with pgsql_tmp that we started moving the directory aside
before starting the server. Discussions about handling this usually
prompt questions about why there are so many temporary files in the
first place (which is fair). FWIW all four functions noted in my
original message [1] are things I've seen firsthand affecting users.
Nathan
[0] https://postgr.es/m/E7573D54-A8C9-40A8-89D7-0596A36ED124%40amazon.com
[1] https://postgr.es/m/C1EE64B0-D4DB-40F3-98C8-0CED324D34CB%40amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-02 02:47 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-02 21:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-03 13:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-03 18:20 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-06 11:43 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-06 19:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-10 19:03 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-13 13:53 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2021-12-13 18:21 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-13 20:36 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2021-12-13 23:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
@ 2021-12-14 16:59 ` Bruce Momjian <[email protected]>
2021-12-14 20:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Bruce Momjian @ 2021-12-14 16:59 UTC (permalink / raw)
To: Bossart, Nathan <[email protected]>; +Cc: Robert Haas <[email protected]>; Bharath Rupireddy <[email protected]>; pgsql-hackers
On Mon, Dec 13, 2021 at 11:05:46PM +0000, Bossart, Nathan wrote:
> On 12/13/21, 12:37 PM, "Robert Haas" <[email protected]> wrote:
> > On Mon, Dec 13, 2021 at 1:21 PM Bossart, Nathan <[email protected]> wrote:
> >> > But against all that, if these tasks are slowing down checkpoints and
> >> > that's avoidable, that seems pretty important too. Interestingly, I
> >> > can't say that I've ever seen any of these things be a problem for
> >> > checkpoint or startup speed. I wonder why you've had a different
> >> > experience.
> >>
> >> Yeah, it's difficult for me to justify why users should suffer long
> >> periods of downtime because startup or checkpointing is taking a very
> >> long time doing things that are arguably unrelated to startup and
> >> checkpointing.
> >
> > Well sure. But I've never actually seen that happen.
>
> I'll admit that surprises me. As noted elsewhere [0], we were seeing
> this enough with pgsql_tmp that we started moving the directory aside
> before starting the server. Discussions about handling this usually
> prompt questions about why there are so many temporary files in the
> first place (which is fair). FWIW all four functions noted in my
> original message [1] are things I've seen firsthand affecting users.
Have we changed temporary file handling in any recent major releases,
meaning is this a current problem or one already improved in PG 14.
--
Bruce Momjian <[email protected]> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-02 02:47 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-02 21:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-03 13:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-03 18:20 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-06 11:43 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-06 19:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-10 19:03 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-13 13:53 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2021-12-13 18:21 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-13 20:36 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2021-12-13 23:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-14 16:59 ` Re: O(n) tasks cause lengthy startups and checkpoints Bruce Momjian <[email protected]>
@ 2021-12-14 20:09 ` Bossart, Nathan <[email protected]>
2021-12-14 20:13 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Bossart, Nathan @ 2021-12-14 20:09 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; +Cc: Robert Haas <[email protected]>; Bharath Rupireddy <[email protected]>; pgsql-hackers
On 12/14/21, 9:00 AM, "Bruce Momjian" <[email protected]> wrote:
> Have we changed temporary file handling in any recent major releases,
> meaning is this a current problem or one already improved in PG 14.
I haven't noticed any recent improvements while working in this area.
Nathan
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-02 02:47 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-02 21:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-03 13:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-03 18:20 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-06 11:43 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-06 19:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-10 19:03 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-13 13:53 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2021-12-13 18:21 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-13 20:36 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2021-12-13 23:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-14 16:59 ` Re: O(n) tasks cause lengthy startups and checkpoints Bruce Momjian <[email protected]>
2021-12-14 20:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
@ 2021-12-14 20:13 ` Bossart, Nathan <[email protected]>
2021-12-14 20:23 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Bossart, Nathan @ 2021-12-14 20:13 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; +Cc: Robert Haas <[email protected]>; Bharath Rupireddy <[email protected]>; pgsql-hackers
On 12/14/21, 12:09 PM, "Bossart, Nathan" <[email protected]> wrote:
> On 12/14/21, 9:00 AM, "Bruce Momjian" <[email protected]> wrote:
>> Have we changed temporary file handling in any recent major releases,
>> meaning is this a current problem or one already improved in PG 14.
>
> I haven't noticed any recent improvements while working in this area.
On second thought, the addition of the remove_temp_files_after_crash
GUC is arguably an improvement since it could prevent files from
accumulating after repeated crashes.
Nathan
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-02 02:47 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-02 21:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-03 13:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-03 18:20 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-06 11:43 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-06 19:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-10 19:03 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-13 13:53 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2021-12-13 18:21 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-13 20:36 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2021-12-13 23:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-14 16:59 ` Re: O(n) tasks cause lengthy startups and checkpoints Bruce Momjian <[email protected]>
2021-12-14 20:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-14 20:13 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
@ 2021-12-14 20:23 ` Bossart, Nathan <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Bossart, Nathan @ 2021-12-14 20:23 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; +Cc: Robert Haas <[email protected]>; Bharath Rupireddy <[email protected]>; pgsql-hackers
On 12/13/21, 10:21 AM, "Bossart, Nathan" <[email protected]> wrote:
> Thanks for chiming in. I have an almost-complete patch set that I'm
> hoping to post to the lists in the next couple of days.
As promised, here is v2. This patch set includes handling for all
four tasks noted upthread. I'd still consider this a work-in-
progress, as I've done minimal testing. At the very least, it should
demonstrate what an auxiliary process approach might look like.
Nathan
Attachments:
[application/octet-stream] v2-0001-Introduce-custodian.patch (18.6K, ../../[email protected]/2-v2-0001-Introduce-custodian.patch)
download | inline diff:
From 5b8133d2707d9da843ac3bb0561b9535ed40675a Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Sun, 5 Dec 2021 15:07:28 -0800
Subject: [PATCH v2 1/8] Introduce custodian.
The custodian process is a new auxiliary process that is intended
to help offload tasks could otherwise delay startup and
checkpointing. This commit simply adds the new process; it does
not yet do anything useful.
---
src/backend/postmaster/Makefile | 1 +
src/backend/postmaster/auxprocess.c | 8 ++
src/backend/postmaster/custodian.c | 210 ++++++++++++++++++++++++++++++++
src/backend/postmaster/postmaster.c | 44 ++++++-
src/backend/storage/lmgr/proc.c | 1 +
src/backend/utils/activity/wait_event.c | 3 +
src/backend/utils/init/miscinit.c | 3 +
src/include/miscadmin.h | 3 +
src/include/postmaster/custodian.h | 17 +++
src/include/storage/proc.h | 11 +-
src/include/utils/wait_event.h | 1 +
11 files changed, 297 insertions(+), 5 deletions(-)
create mode 100644 src/backend/postmaster/custodian.c
create mode 100644 src/include/postmaster/custodian.h
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile
index 787c6a2c3b..7ec7b23467 100644
--- a/src/backend/postmaster/Makefile
+++ b/src/backend/postmaster/Makefile
@@ -18,6 +18,7 @@ OBJS = \
bgworker.o \
bgwriter.o \
checkpointer.o \
+ custodian.o \
fork_process.o \
interrupt.o \
pgarch.o \
diff --git a/src/backend/postmaster/auxprocess.c b/src/backend/postmaster/auxprocess.c
index 43497676ab..10626e7029 100644
--- a/src/backend/postmaster/auxprocess.c
+++ b/src/backend/postmaster/auxprocess.c
@@ -20,6 +20,7 @@
#include "pgstat.h"
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
+#include "postmaster/custodian.h"
#include "postmaster/startup.h"
#include "postmaster/walwriter.h"
#include "replication/walreceiver.h"
@@ -74,6 +75,9 @@ AuxiliaryProcessMain(AuxProcType auxtype)
case CheckpointerProcess:
MyBackendType = B_CHECKPOINTER;
break;
+ case CustodianProcess:
+ MyBackendType = B_CUSTODIAN;
+ break;
case WalWriterProcess:
MyBackendType = B_WAL_WRITER;
break;
@@ -153,6 +157,10 @@ AuxiliaryProcessMain(AuxProcType auxtype)
CheckpointerMain();
proc_exit(1);
+ case CustodianProcess:
+ CustodianMain();
+ proc_exit(1);
+
case WalWriterProcess:
WalWriterMain();
proc_exit(1);
diff --git a/src/backend/postmaster/custodian.c b/src/backend/postmaster/custodian.c
new file mode 100644
index 0000000000..0ba59949bb
--- /dev/null
+++ b/src/backend/postmaster/custodian.c
@@ -0,0 +1,210 @@
+/*-------------------------------------------------------------------------
+ *
+ * custodian.c
+ *
+ * The custodian process is new as of Postgres 15. It's main purpose is to
+ * offload tasks that could otherwise delay startup and checkpointing, but
+ * it needn't be restricted to just those things. Offloaded tasks should
+ * not be synchronous (e.g., checkpointing shouldn't need to wait for the
+ * custodian to complete a task before proceeding). Also, ensure that any
+ * offloaded tasks are either not required during single-user mode or are
+ * performed separately during single-user mode.
+ *
+ * The custodian is not an essential process and can shutdown quickly when
+ * requested. The custodian will wake up approximately once every 5
+ * minutes to perform its tasks, but backends can (and should) set its
+ * latch to wake it up sooner.
+ *
+ * Normal termination is by SIGTERM, which instructs the bgwriter to
+ * exit(0). Emergency termination is by SIGQUIT; like any backend, the
+ * custodian will simply abort and exit on SIGQUIT.
+ *
+ * If the custodian exits unexpectedly, the postmaster treats that the same
+ * as a backend crash: shared memory may be corrupted, so remaining
+ * backends should be killed by SIGQUIT and then a recovery cycle started.
+ *
+ *
+ * Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/postmaster/custodian.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "libpq/pqsignal.h"
+#include "pgstat.h"
+#include "postmaster/custodian.h"
+#include "postmaster/interrupt.h"
+#include "storage/bufmgr.h"
+#include "storage/condition_variable.h"
+#include "storage/proc.h"
+#include "storage/procsignal.h"
+#include "utils/memutils.h"
+
+#define CUSTODIAN_TIMEOUT_S (300) /* 5 minutes */
+
+/*
+ * Main entry point for custodian process
+ *
+ * This is invoked from AuxiliaryProcessMain, which has already created the
+ * basic execution environment, but not enabled signals yet.
+ */
+void
+CustodianMain(void)
+{
+ sigjmp_buf local_sigjmp_buf;
+ MemoryContext custodian_context;
+
+ /*
+ * Properly accept or ignore signals that might be sent to us.
+ */
+ pqsignal(SIGHUP, SignalHandlerForConfigReload);
+ pqsignal(SIGINT, SignalHandlerForShutdownRequest);
+ pqsignal(SIGTERM, SignalHandlerForShutdownRequest);
+ /* SIGQUIT handler was already set up by InitPostmasterChild */
+ pqsignal(SIGALRM, SIG_IGN);
+ pqsignal(SIGPIPE, SIG_IGN);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
+ pqsignal(SIGUSR2, SIG_IGN);
+
+ /*
+ * Reset some signals that are accepted by postmaster but not here
+ */
+ pqsignal(SIGCHLD, SIG_DFL);
+
+ /*
+ * Create a memory context that we will do all our work in. We do this so
+ * that we can reset the context during error recovery and thereby avoid
+ * possible memory leaks.
+ */
+ custodian_context = AllocSetContextCreate(TopMemoryContext,
+ "Custodian",
+ ALLOCSET_DEFAULT_SIZES);
+ MemoryContextSwitchTo(custodian_context);
+
+ /*
+ * If an exception is encountered, processing resumes here.
+ *
+ * You might wonder why this isn't coded as an infinite loop around a
+ * PG_TRY construct. The reason is that this is the bottom of the
+ * exception stack, and so with PG_TRY there would be no exception handler
+ * in force at all during the CATCH part. By leaving the outermost setjmp
+ * always active, we have at least some chance of recovering from an error
+ * during error recovery. (If we get into an infinite loop thereby, it
+ * will soon be stopped by overflow of elog.c's internal state stack.)
+ *
+ * Note that we use sigsetjmp(..., 1), so that the prevailing signal mask
+ * (to wit, BlockSig) will be restored when longjmp'ing to here. Thus,
+ * signals other than SIGQUIT will be blocked until we complete error
+ * recovery. It might seem that this policy makes the HOLD_INTERRUPS()
+ * call redundant, but it is not since InterruptPending might be set
+ * already.
+ */
+ if (sigsetjmp(local_sigjmp_buf, 1) != 0)
+ {
+ /* Since not using PG_TRY, must reset error stack by hand */
+ error_context_stack = NULL;
+
+ /* Prevent interrupts while cleaning up */
+ HOLD_INTERRUPTS();
+
+ /* Report the error to the server log */
+ EmitErrorReport();
+
+ /* These operations are really just a minimal subset of
+ * AbortTransaction(). We don't have very many resources to worry
+ * about.
+ */
+ LWLockReleaseAll();
+ ConditionVariableCancelSleep();
+ pgstat_report_wait_end();
+ AbortBufferIO();
+ UnlockBuffers();
+ ReleaseAuxProcessResources(false);
+ AtEOXact_Buffers(false);
+ AtEOXact_SMgr();
+ AtEOXact_Files(false);
+ AtEOXact_HashTables(false);
+
+ /*
+ * Now return to normal top-level context and clear ErrorContext for
+ * next time.
+ */
+ MemoryContextSwitchTo(custodian_context);
+ FlushErrorState();
+
+ /* Flush any leaked data in the top-level context */
+ MemoryContextResetAndDeleteChildren(custodian_context);
+
+ /* Now we can allow interrupts again */
+ RESUME_INTERRUPTS();
+
+ /*
+ * Sleep at least 1 second after any error. A write error is likely
+ * to be repeated, and we don't want to be filling the error logs as
+ * fast as we can.
+ */
+ pg_usleep(1000000L);
+
+ /*
+ * Close all open files after any error. This is helpful on Windows,
+ * where holding deleted files open causes various strange errors.
+ * It's not clear we need it elsewhere, but shouldn't hurt.
+ */
+ smgrcloseall();
+
+ /* Report wait end here, when there is no further possibility of wait */
+ pgstat_report_wait_end();
+ }
+
+ /* We can now handle ereport(ERROR) */
+ PG_exception_stack = &local_sigjmp_buf;
+
+ /*
+ * Unblock signals (they were blocked when the postmaster forked us)
+ */
+ PG_SETMASK(&UnBlockSig);
+
+ /*
+ * Advertise out latch that backends can use to wake us up while we're
+ * sleeping.
+ */
+ ProcGlobal->custodianLatch = &MyProc->procLatch;
+
+ /*
+ * Loop forever
+ */
+ for (;;)
+ {
+ pg_time_t start_time;
+ pg_time_t end_time;
+ int elapsed_secs;
+ int cur_timeout;
+
+ /* Clear any already-pending wakeups */
+ ResetLatch(MyLatch);
+
+ HandleMainLoopInterrupts();
+
+ start_time = (pg_time_t) time(NULL);
+
+ /* TODO: offloaded tasks go here */
+
+ /* Calculate how long to sleep */
+ end_time = (pg_time_t) time(NULL);
+ elapsed_secs = end_time - start_time;
+ if (elapsed_secs >= CUSTODIAN_TIMEOUT_S)
+ continue; /* no sleep for us */
+ cur_timeout = CUSTODIAN_TIMEOUT_S - elapsed_secs;
+
+ (void) WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ cur_timeout * 1000L /* convert to ms */ ,
+ WAIT_EVENT_CUSTODIAN_MAIN);
+ }
+
+ pg_unreachable();
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 328ecafa8c..635313cdb7 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -250,6 +250,7 @@ bool remove_temp_files_after_crash = true;
static pid_t StartupPID = 0,
BgWriterPID = 0,
CheckpointerPID = 0,
+ CustodianPID = 0,
WalWriterPID = 0,
WalReceiverPID = 0,
AutoVacPID = 0,
@@ -556,6 +557,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#define StartArchiver() StartChildProcess(ArchiverProcess)
#define StartBackgroundWriter() StartChildProcess(BgWriterProcess)
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
+#define StartCustodian() StartChildProcess(CustodianProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
#define StartWalReceiver() StartChildProcess(WalReceiverProcess)
@@ -1819,13 +1821,16 @@ ServerLoop(void)
/*
* If no background writer process is running, and we are not in a
* state that prevents it, start one. It doesn't matter if this
- * fails, we'll just try again later. Likewise for the checkpointer.
+ * fails, we'll just try again later. Likewise for the checkpointer
+ * and custodian.
*/
if (pmState == PM_RUN || pmState == PM_RECOVERY ||
pmState == PM_HOT_STANDBY || pmState == PM_STARTUP)
{
if (CheckpointerPID == 0)
CheckpointerPID = StartCheckpointer();
+ if (CustodianPID == 0)
+ CustodianPID = StartCustodian();
if (BgWriterPID == 0)
BgWriterPID = StartBackgroundWriter();
}
@@ -2782,6 +2787,8 @@ SIGHUP_handler(SIGNAL_ARGS)
signal_child(BgWriterPID, SIGHUP);
if (CheckpointerPID != 0)
signal_child(CheckpointerPID, SIGHUP);
+ if (CustodianPID != 0)
+ signal_child(CustodianPID, SIGHUP);
if (WalWriterPID != 0)
signal_child(WalWriterPID, SIGHUP);
if (WalReceiverPID != 0)
@@ -3109,6 +3116,8 @@ reaper(SIGNAL_ARGS)
*/
if (CheckpointerPID == 0)
CheckpointerPID = StartCheckpointer();
+ if (CustodianPID == 0)
+ CustodianPID = StartCustodian();
if (BgWriterPID == 0)
BgWriterPID = StartBackgroundWriter();
if (WalWriterPID == 0)
@@ -3211,6 +3220,20 @@ reaper(SIGNAL_ARGS)
continue;
}
+ /*
+ * Was it the custodian? Normal exit can be ignored; we'll start a
+ * new one at the next iteration of the postmaster's main loop, if
+ * necessary. Any other exit condition is treated as a crash.
+ */
+ if (pid == CustodianPID)
+ {
+ CustodianPID = 0;
+ if (!EXIT_STATUS_0(exitstatus))
+ HandleChildCrash(pid, exitstatus,
+ _("custodian process"));
+ continue;
+ }
+
/*
* Was it the wal writer? Normal exit can be ignored; we'll start a
* new one at the next iteration of the postmaster's main loop, if
@@ -3684,6 +3707,18 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
signal_child(CheckpointerPID, (SendStop ? SIGSTOP : SIGQUIT));
}
+ /* Take care of the custodian too */
+ if (pid == CustodianPID)
+ CustodianPID = 0;
+ else if (CustodianPID != 0 && take_action)
+ {
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (SendStop ? "SIGSTOP" : "SIGQUIT"),
+ (int) CustodianPID)));
+ signal_child(CustodianPID, (SendStop ? SIGSTOP : SIGQUIT));
+ }
+
/* Take care of the walwriter too */
if (pid == WalWriterPID)
WalWriterPID = 0;
@@ -3887,6 +3922,9 @@ PostmasterStateMachine(void)
/* and the bgwriter too */
if (BgWriterPID != 0)
signal_child(BgWriterPID, SIGTERM);
+ /* and the custodian too */
+ if (CustodianPID != 0)
+ signal_child(CustodianPID, SIGTERM);
/* and the walwriter too */
if (WalWriterPID != 0)
signal_child(WalWriterPID, SIGTERM);
@@ -3924,6 +3962,7 @@ PostmasterStateMachine(void)
BgWriterPID == 0 &&
(CheckpointerPID == 0 ||
(!FatalError && Shutdown < ImmediateShutdown)) &&
+ CustodianPID == 0 &&
WalWriterPID == 0 &&
AutoVacPID == 0)
{
@@ -4017,6 +4056,7 @@ PostmasterStateMachine(void)
Assert(WalReceiverPID == 0);
Assert(BgWriterPID == 0);
Assert(CheckpointerPID == 0);
+ Assert(CustodianPID == 0);
Assert(WalWriterPID == 0);
Assert(AutoVacPID == 0);
/* syslogger is not considered here */
@@ -4222,6 +4262,8 @@ TerminateChildren(int signal)
signal_child(BgWriterPID, signal);
if (CheckpointerPID != 0)
signal_child(CheckpointerPID, signal);
+ if (CustodianPID != 0)
+ signal_child(CustodianPID, signal);
if (WalWriterPID != 0)
signal_child(WalWriterPID, signal);
if (WalReceiverPID != 0)
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index b7d9da0aa9..a86a05adb4 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -182,6 +182,7 @@ InitProcGlobal(void)
ProcGlobal->startupBufferPinWaitBufId = -1;
ProcGlobal->walwriterLatch = NULL;
ProcGlobal->checkpointerLatch = NULL;
+ ProcGlobal->custodianLatch = NULL;
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PGPROCNO);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PGPROCNO);
diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c
index 4d53f040e8..530af294d9 100644
--- a/src/backend/utils/activity/wait_event.c
+++ b/src/backend/utils/activity/wait_event.c
@@ -224,6 +224,9 @@ pgstat_get_wait_activity(WaitEventActivity w)
case WAIT_EVENT_CHECKPOINTER_MAIN:
event_name = "CheckpointerMain";
break;
+ case WAIT_EVENT_CUSTODIAN_MAIN:
+ event_name = "CustodianMain";
+ break;
case WAIT_EVENT_LOGICAL_APPLY_MAIN:
event_name = "LogicalApplyMain";
break;
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 88801374b5..90c4160d42 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -273,6 +273,9 @@ GetBackendTypeDesc(BackendType backendType)
case B_CHECKPOINTER:
backendDesc = "checkpointer";
break;
+ case B_CUSTODIAN:
+ backendDesc = "custodian";
+ break;
case B_STARTUP:
backendDesc = "startup";
break;
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 90a3016065..83089d23ff 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -329,6 +329,7 @@ typedef enum BackendType
B_BG_WORKER,
B_BG_WRITER,
B_CHECKPOINTER,
+ B_CUSTODIAN,
B_STARTUP,
B_WAL_RECEIVER,
B_WAL_SENDER,
@@ -433,6 +434,7 @@ typedef enum
BgWriterProcess,
ArchiverProcess,
CheckpointerProcess,
+ CustodianProcess,
WalWriterProcess,
WalReceiverProcess,
@@ -445,6 +447,7 @@ extern AuxProcType MyAuxProcType;
#define AmBackgroundWriterProcess() (MyAuxProcType == BgWriterProcess)
#define AmArchiverProcess() (MyAuxProcType == ArchiverProcess)
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
+#define AmCustodianProcess() (MyAuxProcType == CustodianProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
diff --git a/src/include/postmaster/custodian.h b/src/include/postmaster/custodian.h
new file mode 100644
index 0000000000..e8ac2ad3dd
--- /dev/null
+++ b/src/include/postmaster/custodian.h
@@ -0,0 +1,17 @@
+/*-------------------------------------------------------------------------
+ *
+ * custodian.h
+ * Exports from postmaster/custodian.c.
+ *
+ * Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ * src/include/postmaster/custodian.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _CUSTODIAN_H
+#define _CUSTODIAN_H
+
+extern void CustodianMain(void) pg_attribute_noreturn();
+
+#endif /* _CUSTODIAN_H */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index cfabfdbedf..1fc4599941 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -357,6 +357,8 @@ typedef struct PROC_HDR
Latch *walwriterLatch;
/* Checkpointer process's latch */
Latch *checkpointerLatch;
+ /* Custodian process's latch */
+ Latch *custodianLatch;
/* Current shared estimate of appropriate spins_per_delay value */
int spins_per_delay;
/* The proc of the Startup process, since not in ProcArray */
@@ -377,11 +379,12 @@ extern PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
- * Background writer, checkpointer, WAL writer and archiver run during normal
- * operation. Startup process and WAL receiver also consume 2 slots, but WAL
- * writer is launched only after startup has exited, so we only need 5 slots.
+ * Background writer, checkpointer, custodian, WAL writer and archiver run
+ * during normal operation. Startup process and WAL receiver also consume 2
+ * slots, but WAL writer is launched only after startup has exited, so we only
+ * need 6 slots.
*/
-#define NUM_AUXILIARY_PROCS 5
+#define NUM_AUXILIARY_PROCS 6
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h
index 8785a8e12c..08dc9d5caa 100644
--- a/src/include/utils/wait_event.h
+++ b/src/include/utils/wait_event.h
@@ -40,6 +40,7 @@ typedef enum
WAIT_EVENT_BGWRITER_HIBERNATE,
WAIT_EVENT_BGWRITER_MAIN,
WAIT_EVENT_CHECKPOINTER_MAIN,
+ WAIT_EVENT_CUSTODIAN_MAIN,
WAIT_EVENT_LOGICAL_APPLY_MAIN,
WAIT_EVENT_LOGICAL_LAUNCHER_MAIN,
WAIT_EVENT_PGSTAT_MAIN,
--
2.16.6
[application/octet-stream] v2-0008-Move-removal-of-spilled-logical-slot-data-to-cust.patch (14.2K, ../../[email protected]/3-v2-0008-Move-removal-of-spilled-logical-slot-data-to-cust.patch)
download | inline diff:
From 590fc3d96320fa701662a5e4af7713a8daeb9a46 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Tue, 14 Dec 2021 18:40:12 +0000
Subject: [PATCH v2 8/8] Move removal of spilled logical slot data to
custodian.
If there are many such files, startup can take much longer than
necessary. To handle this, startup creates a new slot directory,
copies the state file, and swaps the new directory with the old
one. The custodian then asynchronously cleans up the old slot
directory.
---
src/backend/access/transam/xlog.c | 15 +-
src/backend/postmaster/custodian.c | 14 ++
src/backend/replication/logical/reorderbuffer.c | 292 +++++++++++++++++++++++-
src/backend/replication/slot.c | 4 +
src/include/replication/reorderbuffer.h | 1 +
5 files changed, 317 insertions(+), 9 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 5fb8ac9483..bd3c671988 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -7156,18 +7156,21 @@ StartupXLOG(void)
checkPoint.newestCommitTsXid);
XLogCtl->ckptFullXid = checkPoint.nextXid;
- /*
- * Initialize replication slots, before there's a chance to remove
- * required resources.
- */
- StartupReplicationSlots();
-
/*
* Startup logical state, needs to be setup now so we have proper data
* during crash recovery.
+ *
+ * NB: This also performs some important cleanup that must be done prior to
+ * other replication slot steps (e.g., StartupReplicationSlots()).
*/
StartupReorderBuffer();
+ /*
+ * Initialize replication slots, before there's a chance to remove
+ * required resources.
+ */
+ StartupReplicationSlots();
+
/*
* Startup CLOG. This must be done after ShmemVariableCache->nextXid has
* been initialized and before we accept connections or begin WAL replay.
diff --git a/src/backend/postmaster/custodian.c b/src/backend/postmaster/custodian.c
index e3318878d0..9bfad80794 100644
--- a/src/backend/postmaster/custodian.c
+++ b/src/backend/postmaster/custodian.c
@@ -39,6 +39,7 @@
#include "pgstat.h"
#include "postmaster/custodian.h"
#include "postmaster/interrupt.h"
+#include "replication/reorderbuffer.h"
#include "replication/snapbuild.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
@@ -206,6 +207,19 @@ CustodianMain(void)
*/
RemovePgTempFiles(false, false);
+ /*
+ * Remove any replication slot directories that have been staged for
+ * deletion. Since slot directories can accumulate many files, removing
+ * all of the files during startup (which we used to do) can take a very
+ * long time. To avoid delaying startup, we simply have startup rename
+ * the slot directories, and we clean them up here.
+ *
+ * Replication slot directories are not staged or cleaned in single-user
+ * mode, so we don't need any extra handling outside of the custodian
+ * process for this.
+ */
+ RemoveStagedSlotDirectories();
+
/*
* Remove serialized snapshots that are no longer required by any
* logical replication slot.
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 7aa5647a2c..a43e98d848 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -91,15 +91,19 @@
#include "access/xact.h"
#include "access/xlog_internal.h"
#include "catalog/catalog.h"
+#include "common/string.h"
#include "lib/binaryheap.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/interrupt.h"
#include "replication/logical.h"
#include "replication/reorderbuffer.h"
#include "replication/slot.h"
#include "replication/snapbuild.h" /* just for SnapBuildSnapDecRefcount */
#include "storage/bufmgr.h"
+#include "storage/copydir.h"
#include "storage/fd.h"
+#include "storage/proc.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
@@ -255,12 +259,15 @@ static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
bool txn_prepared);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
+static void ReorderBufferCleanup(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+static void StageSlotDirForRemoval(const char *slotname, const char *slotpath);
+static void RemoveStagedSlotDirectory(const char *path);
/*
* ---------------------------------------
@@ -4430,6 +4437,202 @@ ReorderBufferCleanupSerializedTXNs(const char *slotname)
FreeDir(spill_dir);
}
+/*
+ * Cleanup everything in the logical slot directory except for the "state" file.
+ * This is specially written for StartupReorderBuffer(), which has special logic
+ * to handle crashes at inconvenient times.
+ *
+ * NB: If anything except for the "state" file cannot be removed after startup,
+ * this will need to be updated.
+ */
+static void
+ReorderBufferCleanup(const char *slotname)
+{
+ char path[MAXPGPATH];
+ char newpath[MAXPGPATH];
+ char statepath[MAXPGPATH];
+ char newstatepath[MAXPGPATH];
+ struct stat statbuf;
+
+ sprintf(path, "pg_replslot/%s", slotname);
+ sprintf(newpath, "pg_replslot/%s.new", slotname);
+ sprintf(statepath, "pg_replslot/%s/state", slotname);
+ sprintf(newstatepath, "pg_replslot/%s.new/state", slotname);
+
+ /* we're only handling directories here, skip if it's not ours */
+ if (lstat(path, &statbuf) == 0 && !S_ISDIR(statbuf.st_mode))
+ return;
+
+ /*
+ * Build our new slot directory, suffixed with ".new". The caller (likely
+ * StartupReorderBuffer()) should have already ensured that any pre-existing
+ * ".new" directories leftover after a crash have been cleaned up.
+ */
+ if (MakePGDirectory(newpath) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create directory \"%s\": %m", newpath)));
+
+ copy_file(statepath, newstatepath);
+
+ fsync_fname(newstatepath, false);
+ fsync_fname(newpath, true);
+ fsync_fname("pg_replslot", true);
+
+ /*
+ * Move the slot directory aside for cleanup by the custodian. After this
+ * step, there will be no slot directory. StartupReorderBuffer() has
+ * special logic to make sure we don't lose the slot if we crash at this
+ * point.
+ */
+ StageSlotDirForRemoval(slotname, path);
+
+ /*
+ * Move our ".new" directory to become our new slot directory.
+ */
+ if (rename(newpath, path) != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not rename file \"%s\": %m", newpath)));
+
+ fsync_fname(path, true);
+ fsync_fname("pg_replslot", true);
+}
+
+/*
+ * This function renames the given directory with a special suffix that the
+ * custodian will know to look for. An integer is appended to the end of the
+ * new directory name in case previously staged slot directories have not yet
+ * been removed.
+ */
+static void
+StageSlotDirForRemoval(const char *slotname, const char *slotpath)
+{
+ char stage_path[MAXPGPATH];
+
+ /*
+ * Find a name for the stage directory. We just increment an integer at the
+ * end of the name until we find one that doesn't exist.
+ */
+ for (int n = 0; n <= INT_MAX; n++)
+ {
+ DIR *dir;
+
+ sprintf(stage_path, "pg_replslot/%s.to_remove_%d", slotname, n);
+
+ dir = AllocateDir(stage_path);
+ if (dir == NULL)
+ {
+ if (errno == ENOENT)
+ break;
+
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open directory \"%s\": %m",
+ stage_path)));
+ }
+ FreeDir(dir);
+
+ stage_path[0] = '\0';
+ }
+
+ /*
+ * In the unlikely event that we couldn't find a name for the stage
+ * directory, bail out.
+ */
+ if (stage_path[0] == '\0')
+ ereport(ERROR,
+ (errmsg("could not stage \"%s\" for deletion",
+ slotpath)));
+
+ /*
+ * Rename the slot directory.
+ */
+ if (rename(slotpath, stage_path) != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not rename file \"%s\": %m", slotpath)));
+
+ fsync_fname(stage_path, true);
+ fsync_fname("pg_replslot", true);
+}
+
+/*
+ * Remove slot directories that have been staged for deletion by
+ * ReorderBufferCleanup().
+ */
+void
+RemoveStagedSlotDirectories(void)
+{
+ DIR *dir;
+ struct dirent *de;
+
+ dir = AllocateDir("pg_replslot");
+ while (!ShutdownRequestPending &&
+ (de = ReadDir(dir, "pg_replslot")) != NULL)
+ {
+ struct stat st;
+ char path[MAXPGPATH];
+
+ if (strstr(de->d_name, ".to_remove") == NULL)
+ continue;
+
+ sprintf(path, "pg_replslot/%s", de->d_name);
+ if (lstat(path, &st) != 0)
+ ereport(ERROR,
+ (errmsg("could not stat file \"%s\": %m", path)));
+
+ if (!S_ISDIR(st.st_mode))
+ continue;
+
+ RemoveStagedSlotDirectory(path);
+ }
+ FreeDir(dir);
+}
+
+/*
+ * Removes one slot directory that has been staged for deletion by
+ * ReorderBufferCleanup(). If a shutdown request is pending, exit as soon as
+ * possible.
+ */
+static void
+RemoveStagedSlotDirectory(const char *path)
+{
+ DIR *dir;
+ struct dirent *de;
+
+ dir = AllocateDir(path);
+ while (!ShutdownRequestPending &&
+ (de = ReadDir(dir, path)) != NULL)
+ {
+ struct stat st;
+ char filepath[MAXPGPATH];
+
+ if (strcmp(de->d_name, ".") == 0 ||
+ strcmp(de->d_name, "..") == 0)
+ continue;
+
+ sprintf(filepath, "%s/%s", path, de->d_name);
+
+ if (lstat(filepath, &st) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not stat file \"%s\": %m", filepath)));
+ else if (S_ISDIR(st.st_mode))
+ RemoveStagedSlotDirectory(filepath);
+ else if (unlink(filepath) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", filepath)));
+ }
+ FreeDir(dir);
+
+ if (rmdir(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove directory \"%s\": %m", path)));
+}
+
/*
* Given a replication slot, transaction ID and segment number, fill in the
* corresponding spill file into 'path', which is a caller-owned buffer of size
@@ -4458,6 +4661,83 @@ StartupReorderBuffer(void)
DIR *logical_dir;
struct dirent *logical_de;
+ /*
+ * First, handle any ".new" directories that were leftover after a crash.
+ * These are created and swapped with the actual replication slot
+ * directories so that cleanup of spilled data can be done asynchronously by
+ * the custodian.
+ */
+ logical_dir = AllocateDir("pg_replslot");
+ while ((logical_de = ReadDir(logical_dir, "pg_replslot")) != NULL)
+ {
+ char name[NAMEDATALEN];
+ char path[NAMEDATALEN + 12];
+ struct stat statbuf;
+
+ if (strcmp(logical_de->d_name, ".") == 0 ||
+ strcmp(logical_de->d_name, "..") == 0)
+ continue;
+
+ /*
+ * Make sure it's a valid ".new" directory.
+ */
+ if (!pg_str_endswith(logical_de->d_name, ".new") ||
+ strlen(logical_de->d_name) >= NAMEDATALEN + 4)
+ continue;
+
+ strncpy(name, logical_de->d_name, sizeof(name));
+ name[strlen(logical_de->d_name) - 4] = '\0';
+ if (!ReplicationSlotValidateName(name, DEBUG2))
+ continue;
+
+ sprintf(path, "pg_replslot/%s", name);
+ if (lstat(path, &statbuf) == 0)
+ {
+ if (!S_ISDIR(statbuf.st_mode))
+ continue;
+
+ /*
+ * If the original directory still exists, just delete the ".new"
+ * directory. We'll try again when we call ReorderBufferCleanup()
+ * later on.
+ */
+ if (!rmtree(path, true))
+ ereport(ERROR,
+ (errmsg("could not remove directory \"%s\"", path)));
+ }
+ else if (errno == ENOENT)
+ {
+ char newpath[NAMEDATALEN + 16];
+
+ /*
+ * If the original directory is gone, we need to rename the ".new"
+ * directory to take its place. We know that the ".new" directory
+ * is ready to be the real deal if we previously made it far enough
+ * to delete the original directory.
+ */
+ sprintf(newpath, "pg_replslot/%s", logical_de->d_name);
+ if (rename(newpath, path) != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not rename file \"%s\" to \"%s\": %m",
+ newpath, path)));
+
+ fsync_fname(path, true);
+ }
+ else
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not stat file \"%s\": %m", path)));
+
+ fsync_fname("pg_replslot", true);
+ }
+ FreeDir(logical_dir);
+
+ /*
+ * Now we can proceed with deleting all spilled data. (This actually just
+ * moves the directories aside so that the custodian can clean it up
+ * asynchronously.)
+ */
logical_dir = AllocateDir("pg_replslot");
while ((logical_de = ReadDir(logical_dir, "pg_replslot")) != NULL)
{
@@ -4470,12 +4750,18 @@ StartupReorderBuffer(void)
continue;
/*
- * ok, has to be a surviving logical slot, iterate and delete
- * everything starting with xid-*
+ * ok, has to be a surviving logical slot, delete everything except for
+ * state
*/
- ReorderBufferCleanupSerializedTXNs(logical_de->d_name);
+ ReorderBufferCleanup(logical_de->d_name);
}
FreeDir(logical_dir);
+
+ /*
+ * Wake up the custodian so it cleans up our old slot data.
+ */
+ if (ProcGlobal->custodianLatch)
+ SetLatch(ProcGlobal->custodianLatch);
}
/* ---------------------------------------
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 90ba9b417d..e9dbc18e22 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -1430,6 +1430,10 @@ StartupReplicationSlots(void)
continue;
}
+ /* if it's an old slot directory that's staged for removal, ignore it */
+ if (strstr(replication_de->d_name, ".to_remove") != NULL)
+ continue;
+
/* looks like a slot in a normal state, restore */
RestoreSlotFromDisk(replication_de->d_name);
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 5b40ff75f7..807b7038d9 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -681,5 +681,6 @@ TransactionId ReorderBufferGetOldestXmin(ReorderBuffer *rb);
void ReorderBufferSetRestartPoint(ReorderBuffer *, XLogRecPtr ptr);
void StartupReorderBuffer(void);
+void RemoveStagedSlotDirectories(void);
#endif
--
2.16.6
[application/octet-stream] v2-0007-Use-syncfs-in-CheckPointLogicalRewriteHeap-for-sh.patch (3.2K, ../../[email protected]/4-v2-0007-Use-syncfs-in-CheckPointLogicalRewriteHeap-for-sh.patch)
download | inline diff:
From 2d7978947c9891585692a84e5ae236664f60cee0 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Mon, 13 Dec 2021 20:20:12 -0800
Subject: [PATCH v2 7/8] Use syncfs() in CheckPointLogicalRewriteHeap() for
shutdown and end-of-recovery checkpoints.
This may save quite a bit of time when there are many mapping files
to flush to disk.
---
src/backend/access/heap/rewriteheap.c | 35 ++++++++++++++++++++++++++++++++++-
src/backend/access/transam/xlog.c | 2 +-
src/include/access/rewriteheap.h | 2 +-
3 files changed, 36 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/heap/rewriteheap.c b/src/backend/access/heap/rewriteheap.c
index e11f5bfb80..d697d46fbb 100644
--- a/src/backend/access/heap/rewriteheap.c
+++ b/src/backend/access/heap/rewriteheap.c
@@ -1193,7 +1193,7 @@ heap_xlog_logical_rewrite(XLogReaderState *r)
* ---
*/
void
-CheckPointLogicalRewriteHeap(void)
+CheckPointLogicalRewriteHeap(bool shutdown)
{
XLogRecPtr cutoff;
XLogRecPtr redo;
@@ -1219,6 +1219,39 @@ CheckPointLogicalRewriteHeap(void)
if (ProcGlobal->custodianLatch)
SetLatch(ProcGlobal->custodianLatch);
+#ifdef HAVE_SYNCFS
+
+ /*
+ * If we are doing a shutdown or end-of-recovery checkpoint, let's use
+ * syncfs() to flush the mappings to disk instead of flushing each one
+ * individually. This may save us quite a bit of time when there are many
+ * such files to flush.
+ */
+ if (shutdown)
+ {
+ int fd;
+
+ fd = OpenTransientFile("pg_logical/mappings", O_RDONLY);
+ if (fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"pg_logical/mappings\": %m")));
+
+ if (syncfs(fd) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not synchronize file system for file \"pg_logical/mappings\": %m")));
+
+ if (CloseTransientFile(fd) != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not close file \"pg_logical/mappings\": %m")));
+
+ return;
+ }
+
+#endif /* HAVE_SYNCFS */
+
mappings_dir = AllocateDir("pg_logical/mappings");
while ((mapping_de = ReadDir(mappings_dir, "pg_logical/mappings")) != NULL)
{
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 2a5a3fe765..5fb8ac9483 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -9572,7 +9572,7 @@ CheckPointGuts(XLogRecPtr checkPointRedo, int flags)
{
CheckPointRelationMap();
CheckPointReplicationSlots();
- CheckPointLogicalRewriteHeap();
+ CheckPointLogicalRewriteHeap(flags & (CHECKPOINT_IS_SHUTDOWN | CHECKPOINT_END_OF_RECOVERY));
CheckPointReplicationOrigin();
/* Write out all dirty data in SLRUs and the main buffer pool */
diff --git a/src/include/access/rewriteheap.h b/src/include/access/rewriteheap.h
index 2df5f4f5cd..dda0629db2 100644
--- a/src/include/access/rewriteheap.h
+++ b/src/include/access/rewriteheap.h
@@ -52,7 +52,7 @@ typedef struct LogicalRewriteMappingData
* ---
*/
#define LOGICAL_REWRITE_FORMAT "map-%x-%x-%X_%X-%x-%x"
-void CheckPointLogicalRewriteHeap(void);
+void CheckPointLogicalRewriteHeap(bool shutdown);
void RemoveOldLogicalRewriteMappings(void);
#endif /* REWRITE_HEAP_H */
--
2.16.6
[application/octet-stream] v2-0006-Move-removal-of-old-logical-rewrite-mapping-files.patch (8.0K, ../../[email protected]/5-v2-0006-Move-removal-of-old-logical-rewrite-mapping-files.patch)
download | inline diff:
From ac5bec20bd74a61c69103ec35196c3920ea8d050 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Sun, 12 Dec 2021 22:07:11 -0800
Subject: [PATCH v2 6/8] Move removal of old logical rewrite mapping files to
custodian.
If there are many such files to remove, checkpoints can take much
longer. To avoid this, move this work to the newly-introduced
custodian process.
---
src/backend/access/heap/rewriteheap.c | 83 ++++++++++++++++++++++++++++++-----
src/backend/postmaster/checkpointer.c | 33 ++++++++++++++
src/backend/postmaster/custodian.c | 10 +++++
src/include/access/rewriteheap.h | 1 +
src/include/postmaster/bgwriter.h | 3 ++
5 files changed, 120 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/heap/rewriteheap.c b/src/backend/access/heap/rewriteheap.c
index 986a776bbd..e11f5bfb80 100644
--- a/src/backend/access/heap/rewriteheap.c
+++ b/src/backend/access/heap/rewriteheap.c
@@ -116,10 +116,13 @@
#include "lib/ilist.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/bgwriter.h"
+#include "postmaster/interrupt.h"
#include "replication/logical.h"
#include "replication/slot.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
+#include "storage/proc.h"
#include "storage/procarray.h"
#include "storage/smgr.h"
#include "utils/memutils.h"
@@ -1182,7 +1185,8 @@ heap_xlog_logical_rewrite(XLogReaderState *r)
* Perform a checkpoint for logical rewrite mappings
*
* This serves two tasks:
- * 1) Remove all mappings not needed anymore based on the logical restart LSN
+ * 1) Alert the custodian to remove all mappings not needed anymore based on the
+ * logical restart LSN
* 2) Flush all remaining mappings to disk, so that replay after a checkpoint
* only has to deal with the parts of a mapping that have been written out
* after the checkpoint started.
@@ -1210,6 +1214,11 @@ CheckPointLogicalRewriteHeap(void)
if (cutoff != InvalidXLogRecPtr && redo < cutoff)
cutoff = redo;
+ /* let the custodian know what it can remove */
+ CheckPointSetLogicalRewriteCutoff(cutoff);
+ if (ProcGlobal->custodianLatch)
+ SetLatch(ProcGlobal->custodianLatch);
+
mappings_dir = AllocateDir("pg_logical/mappings");
while ((mapping_de = ReadDir(mappings_dir, "pg_logical/mappings")) != NULL)
{
@@ -1240,15 +1249,7 @@ CheckPointLogicalRewriteHeap(void)
lsn = ((uint64) hi) << 32 | lo;
- if (lsn < cutoff || cutoff == InvalidXLogRecPtr)
- {
- elog(DEBUG1, "removing logical rewrite file \"%s\"", path);
- if (unlink(path) < 0)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not remove file \"%s\": %m", path)));
- }
- else
+ if (lsn >= cutoff && cutoff != InvalidXLogRecPtr)
{
/* on some operating systems fsyncing a file requires O_RDWR */
int fd = OpenTransientFile(path, O_RDWR | PG_BINARY);
@@ -1283,3 +1284,65 @@ CheckPointLogicalRewriteHeap(void)
}
FreeDir(mappings_dir);
}
+
+/*
+ * Remove all mappings not needed anymore based on the logical restart LSN saved
+ * by the checkpointer. We use this saved value instead of calling
+ * ReplicationSlotsComputeLogicalRestartLSN() so that we don't interfere with an
+ * ongoing call to CheckPointLogicalRewriteHeap() that is flushing mappings to
+ * disk.
+ */
+void
+RemoveOldLogicalRewriteMappings(void)
+{
+ XLogRecPtr cutoff;
+ DIR *mappings_dir;
+ struct dirent *mapping_de;
+ char path[MAXPGPATH + 20];
+ bool value_set = false;
+
+ cutoff = CheckPointGetLogicalRewriteCutoff(&value_set);
+ if (!value_set)
+ return;
+
+ mappings_dir = AllocateDir("pg_logical/mappings");
+ while (!ShutdownRequestPending &&
+ (mapping_de = ReadDir(mappings_dir, "pg_logical/mappings")) != NULL)
+ {
+ struct stat statbuf;
+ Oid dboid;
+ Oid relid;
+ XLogRecPtr lsn;
+ TransactionId rewrite_xid;
+ TransactionId create_xid;
+ uint32 hi,
+ lo;
+
+ if (strcmp(mapping_de->d_name, ".") == 0 ||
+ strcmp(mapping_de->d_name, "..") == 0)
+ continue;
+
+ snprintf(path, sizeof(path), "pg_logical/mappings/%s", mapping_de->d_name);
+ if (lstat(path, &statbuf) == 0 && !S_ISREG(statbuf.st_mode))
+ continue;
+
+ /* Skip over files that cannot be ours. */
+ if (strncmp(mapping_de->d_name, "map-", 4) != 0)
+ continue;
+
+ if (sscanf(mapping_de->d_name, LOGICAL_REWRITE_FORMAT,
+ &dboid, &relid, &hi, &lo, &rewrite_xid, &create_xid) != 6)
+ elog(ERROR, "could not parse filename \"%s\"", mapping_de->d_name);
+
+ lsn = ((uint64) hi) << 32 | lo;
+ if (lsn >= cutoff && cutoff != InvalidXLogRecPtr)
+ continue;
+
+ elog(DEBUG1, "removing logical rewrite file \"%s\"", path);
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+ }
+ FreeDir(mappings_dir);
+}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 25a18b7a14..0c5563cb4b 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -127,6 +127,9 @@ typedef struct
uint32 num_backend_writes; /* counts user backend buffer writes */
uint32 num_backend_fsync; /* counts user backend fsync calls */
+ XLogRecPtr logical_rewrite_mappings_cutoff; /* can remove older mappings */
+ XLogRecPtr logical_rewrite_mappings_cutoff_set;
+
int num_requests; /* current # of requests */
int max_requests; /* allocated array size */
CheckpointerRequest requests[FLEXIBLE_ARRAY_MEMBER];
@@ -1337,3 +1340,33 @@ FirstCallSinceLastCheckpoint(void)
return FirstCall;
}
+
+/*
+ * Used by CheckPointLogicalRewriteHeap() to tell the custodian which logical
+ * rewrite mapping files it can remove.
+ */
+void
+CheckPointSetLogicalRewriteCutoff(XLogRecPtr cutoff)
+{
+ SpinLockAcquire(&CheckpointerShmem->ckpt_lck);
+ CheckpointerShmem->logical_rewrite_mappings_cutoff = cutoff;
+ CheckpointerShmem->logical_rewrite_mappings_cutoff_set = true;
+ SpinLockRelease(&CheckpointerShmem->ckpt_lck);
+}
+
+/*
+ * Used by the custodian to determine which logical rewrite mapping files it can
+ * remove.
+ */
+XLogRecPtr
+CheckPointGetLogicalRewriteCutoff(bool *value_set)
+{
+ XLogRecPtr cutoff;
+
+ SpinLockAcquire(&CheckpointerShmem->ckpt_lck);
+ cutoff = CheckpointerShmem->logical_rewrite_mappings_cutoff;
+ *value_set = CheckpointerShmem->logical_rewrite_mappings_cutoff_set;
+ SpinLockRelease(&CheckpointerShmem->ckpt_lck);
+
+ return cutoff;
+}
diff --git a/src/backend/postmaster/custodian.c b/src/backend/postmaster/custodian.c
index fbac2c6add..e3318878d0 100644
--- a/src/backend/postmaster/custodian.c
+++ b/src/backend/postmaster/custodian.c
@@ -34,6 +34,7 @@
*/
#include "postgres.h"
+#include "access/rewriteheap.h"
#include "libpq/pqsignal.h"
#include "pgstat.h"
#include "postmaster/custodian.h"
@@ -215,6 +216,15 @@ CustodianMain(void)
*/
RemoveOldSerializedSnapshots();
+ /*
+ * Remove logical rewrite mapping files that are no longer needed.
+ *
+ * It is not important for these to be removed in single-user mode, so
+ * we don't need any extra handling outside of the custodian process for
+ * this.
+ */
+ RemoveOldLogicalRewriteMappings();
+
/* Calculate how long to sleep */
end_time = (pg_time_t) time(NULL);
elapsed_secs = end_time - start_time;
diff --git a/src/include/access/rewriteheap.h b/src/include/access/rewriteheap.h
index 121f552405..2df5f4f5cd 100644
--- a/src/include/access/rewriteheap.h
+++ b/src/include/access/rewriteheap.h
@@ -53,5 +53,6 @@ typedef struct LogicalRewriteMappingData
*/
#define LOGICAL_REWRITE_FORMAT "map-%x-%x-%X_%X-%x-%x"
void CheckPointLogicalRewriteHeap(void);
+void RemoveOldLogicalRewriteMappings(void);
#endif /* REWRITE_HEAP_H */
diff --git a/src/include/postmaster/bgwriter.h b/src/include/postmaster/bgwriter.h
index c430b1b236..bc9f57c93c 100644
--- a/src/include/postmaster/bgwriter.h
+++ b/src/include/postmaster/bgwriter.h
@@ -42,4 +42,7 @@ extern void CheckpointerShmemInit(void);
extern bool FirstCallSinceLastCheckpoint(void);
+extern void CheckPointSetLogicalRewriteCutoff(XLogRecPtr cutoff);
+extern XLogRecPtr CheckPointGetLogicalRewriteCutoff(bool *value_set);
+
#endif /* _BGWRITER_H */
--
2.16.6
[application/octet-stream] v2-0005-Move-removal-of-old-serialized-snapshots-to-custo.patch (4.3K, ../../[email protected]/6-v2-0005-Move-removal-of-old-serialized-snapshots-to-custo.patch)
download | inline diff:
From 4e41d80a0ab0f820602b1df998605a2920363323 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Sun, 5 Dec 2021 22:02:40 -0800
Subject: [PATCH v2 5/8] Move removal of old serialized snapshots to custodian.
This was only done during checkpoints because it was a convenient
place to put it. However, if there are many snapshots to remove,
it can significantly extend checkpoint time. To avoid this, move
this work to the newly-introduced custodian process.
---
src/backend/access/transam/xlog.c | 2 --
src/backend/postmaster/custodian.c | 11 +++++++++++
src/backend/replication/logical/snapbuild.c | 13 +++++++------
src/include/replication/snapbuild.h | 2 +-
4 files changed, 19 insertions(+), 9 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 72aeb42961..2a5a3fe765 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -56,7 +56,6 @@
#include "replication/logical.h"
#include "replication/origin.h"
#include "replication/slot.h"
-#include "replication/snapbuild.h"
#include "replication/walreceiver.h"
#include "replication/walsender.h"
#include "storage/bufmgr.h"
@@ -9573,7 +9572,6 @@ CheckPointGuts(XLogRecPtr checkPointRedo, int flags)
{
CheckPointRelationMap();
CheckPointReplicationSlots();
- CheckPointSnapBuild();
CheckPointLogicalRewriteHeap();
CheckPointReplicationOrigin();
diff --git a/src/backend/postmaster/custodian.c b/src/backend/postmaster/custodian.c
index a5443f9a21..fbac2c6add 100644
--- a/src/backend/postmaster/custodian.c
+++ b/src/backend/postmaster/custodian.c
@@ -38,6 +38,7 @@
#include "pgstat.h"
#include "postmaster/custodian.h"
#include "postmaster/interrupt.h"
+#include "replication/snapbuild.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/proc.h"
@@ -204,6 +205,16 @@ CustodianMain(void)
*/
RemovePgTempFiles(false, false);
+ /*
+ * Remove serialized snapshots that are no longer required by any
+ * logical replication slot.
+ *
+ * It is not important for these to be removed in single-user mode, so
+ * we don't need any extra handling outside of the custodian process for
+ * this.
+ */
+ RemoveOldSerializedSnapshots();
+
/* Calculate how long to sleep */
end_time = (pg_time_t) time(NULL);
elapsed_secs = end_time - start_time;
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index dbdc172a2b..cf873b519b 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -125,6 +125,7 @@
#include "access/xact.h"
#include "miscadmin.h"
#include "pgstat.h"
+#include "postmaster/interrupt.h"
#include "replication/logical.h"
#include "replication/reorderbuffer.h"
#include "replication/snapbuild.h"
@@ -1912,14 +1913,13 @@ snapshot_not_interesting:
/*
* Remove all serialized snapshots that are not required anymore because no
- * slot can need them. This doesn't actually have to run during a checkpoint,
- * but it's a convenient point to schedule this.
+ * slot can need them.
*
- * NB: We run this during checkpoints even if logical decoding is disabled so
- * we cleanup old slots at some point after it got disabled.
+ * NB: We run this even if logical decoding is disabled so we cleanup old slots
+ * at some point after it got disabled.
*/
void
-CheckPointSnapBuild(void)
+RemoveOldSerializedSnapshots(void)
{
XLogRecPtr cutoff;
XLogRecPtr redo;
@@ -1942,7 +1942,8 @@ CheckPointSnapBuild(void)
cutoff = redo;
snap_dir = AllocateDir("pg_logical/snapshots");
- while ((snap_de = ReadDir(snap_dir, "pg_logical/snapshots")) != NULL)
+ while (!ShutdownRequestPending &&
+ (snap_de = ReadDir(snap_dir, "pg_logical/snapshots")) != NULL)
{
uint32 hi;
uint32 lo;
diff --git a/src/include/replication/snapbuild.h b/src/include/replication/snapbuild.h
index 82aa86125b..ba7276058d 100644
--- a/src/include/replication/snapbuild.h
+++ b/src/include/replication/snapbuild.h
@@ -57,7 +57,7 @@ struct ReorderBuffer;
struct xl_heap_new_cid;
struct xl_running_xacts;
-extern void CheckPointSnapBuild(void);
+extern void RemoveOldSerializedSnapshots(void);
extern SnapBuild *AllocateSnapshotBuilder(struct ReorderBuffer *cache,
TransactionId xmin_horizon, XLogRecPtr start_lsn,
--
2.16.6
[application/octet-stream] v2-0004-Move-pgsql_tmp-file-removal-to-custodian-process.patch (6.4K, ../../[email protected]/7-v2-0004-Move-pgsql_tmp-file-removal-to-custodian-process.patch)
download | inline diff:
From 5b133a676a64fea4cc7463daa755ad60f7c8789e Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Sun, 5 Dec 2021 21:42:52 -0800
Subject: [PATCH v2 4/8] Move pgsql_tmp file removal to custodian process.
With this change, startup (and restart after a crash) simply
renames the pgsql_tmp directories, and the custodian process
actually removes all the files in the staged directories as well as
the staged directories themselves. This should help avoid long
startup delays due to many leftover temporary files.
---
src/backend/postmaster/custodian.c | 13 ++++++++++++-
src/backend/postmaster/postmaster.c | 14 +++++++++-----
src/backend/storage/file/fd.c | 32 +++++++++++++++++++++++---------
3 files changed, 44 insertions(+), 15 deletions(-)
diff --git a/src/backend/postmaster/custodian.c b/src/backend/postmaster/custodian.c
index 0ba59949bb..a5443f9a21 100644
--- a/src/backend/postmaster/custodian.c
+++ b/src/backend/postmaster/custodian.c
@@ -191,7 +191,18 @@ CustodianMain(void)
start_time = (pg_time_t) time(NULL);
- /* TODO: offloaded tasks go here */
+ /*
+ * Remove any pgsql_tmp directories that have been staged for deletion.
+ * Since pgsql_tmp directories can accumulate many files, removing all
+ * of the files during startup (which we used to do) can take a very
+ * long time. To avoid delaying startup, we simply have startup rename
+ * the temporary directories, and we clean them up here.
+ *
+ * pgsql_tmp directories are not staged or cleaned in single-user mode,
+ * so we don't need any extra handling outside of the custodian process
+ * for this.
+ */
+ RemovePgTempFiles(false, false);
/* Calculate how long to sleep */
end_time = (pg_time_t) time(NULL);
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 1ae2dc179e..b098482496 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -1391,9 +1391,11 @@ PostmasterMain(int argc, char *argv[])
/*
* Remove old temporary files. At this point there can be no other
* Postgres processes running in this directory, so this should be safe.
+ *
+ * Note that this just stages the pgsql_tmp directories for deletion. The
+ * custodian process is responsible for actually removing the files.
*/
RemovePgTempFiles(true, true);
- RemovePgTempFiles(false, false);
/*
* Initialize stats collection subsystem (this does NOT start the
@@ -4139,12 +4141,14 @@ PostmasterStateMachine(void)
ereport(LOG,
(errmsg("all server processes terminated; reinitializing")));
- /* remove leftover temporary files after a crash */
+ /*
+ * Remove leftover temporary files after a crash.
+ *
+ * Note that this just stages the pgsql_tmp directories for deletion.
+ * The custodian process is responsible for actually removing the files.
+ */
if (remove_temp_files_after_crash)
- {
RemovePgTempFiles(true, true);
- RemovePgTempFiles(false, false);
- }
/* allow background workers to immediately restart */
ResetBackgroundWorkerCrashTimes();
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 633c6eee18..ac56c41562 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -97,9 +97,12 @@
#include "pgstat.h"
#include "port/pg_iovec.h"
#include "portability/mem.h"
+#include "postmaster/interrupt.h"
#include "postmaster/startup.h"
#include "storage/fd.h"
#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
#include "utils/guc.h"
#include "utils/resowner_private.h"
@@ -1640,9 +1643,9 @@ PathNameOpenFilePerm(const char *fileName, int fileFlags, mode_t fileMode)
*
* Directories created within the top-level temporary directory should begin
* with PG_TEMP_FILE_PREFIX, so that they can be identified as temporary and
- * deleted at startup by RemovePgTempFiles(). Further subdirectories below
- * that do not need any particular prefix.
-*/
+ * deleted by RemovePgTempFiles(). Further subdirectories below that do not
+ * need any particular prefix.
+ */
void
PathNameCreateTemporaryDir(const char *basedir, const char *directory)
{
@@ -1840,9 +1843,9 @@ OpenTemporaryFileInTablespace(Oid tblspcOid, bool rejectError)
*
* If the file is inside the top-level temporary directory, its name should
* begin with PG_TEMP_FILE_PREFIX so that it can be identified as temporary
- * and deleted at startup by RemovePgTempFiles(). Alternatively, it can be
- * inside a directory created with PathNameCreateTemporaryDir(), in which case
- * the prefix isn't needed.
+ * and deleted by RemovePgTempFiles(). Alternatively, it can be inside a
+ * directory created with PathNameCreateTemporaryDir(), in which case the prefix
+ * isn't needed.
*/
File
PathNameCreateTemporaryFile(const char *path, bool error_on_failure)
@@ -3175,7 +3178,8 @@ RemovePgTempFiles(bool stage, bool remove_relation_files)
*/
spc_dir = AllocateDir("pg_tblspc");
- while ((spc_de = ReadDirExtended(spc_dir, "pg_tblspc", LOG)) != NULL)
+ while (!ShutdownRequestPending &&
+ (spc_de = ReadDirExtended(spc_dir, "pg_tblspc", LOG)) != NULL)
{
if (strcmp(spc_de->d_name, ".") == 0 ||
strcmp(spc_de->d_name, "..") == 0)
@@ -3211,6 +3215,14 @@ RemovePgTempFiles(bool stage, bool remove_relation_files)
* would create a race condition. It's done separately, earlier in
* postmaster startup.
*/
+
+ /*
+ * If we just staged some pgsql_tmp directories for removal, wake up the
+ * custodian process so that it deletes all the files in the staged
+ * directories as well as the directories themselves.
+ */
+ if (stage && ProcGlobal->custodianLatch)
+ SetLatch(ProcGlobal->custodianLatch);
}
/*
@@ -3315,7 +3327,8 @@ RemoveStagedPgTempDirs(const char *spc_dir)
struct dirent *de;
dir = AllocateDir(spc_dir);
- while ((de = ReadDirExtended(dir, spc_dir, LOG)) != NULL)
+ while (!ShutdownRequestPending &&
+ (de = ReadDirExtended(dir, spc_dir, LOG)) != NULL)
{
if (strncmp(de->d_name, PG_TEMP_DIR_TO_REMOVE_PREFIX,
strlen(PG_TEMP_DIR_TO_REMOVE_PREFIX)) != 0)
@@ -3354,7 +3367,8 @@ RemovePgTempDir(const char *tmpdirname, bool missing_ok, bool unlink_all)
if (temp_dir == NULL && errno == ENOENT && missing_ok)
return;
- while ((temp_de = ReadDirExtended(temp_dir, tmpdirname, LOG)) != NULL)
+ while (!ShutdownRequestPending &&
+ (temp_de = ReadDirExtended(temp_dir, tmpdirname, LOG)) != NULL)
{
if (strcmp(temp_de->d_name, ".") == 0 ||
strcmp(temp_de->d_name, "..") == 0)
--
2.16.6
[application/octet-stream] v2-0003-Split-pgsql_tmp-cleanup-into-two-stages.patch (9.8K, ../../[email protected]/8-v2-0003-Split-pgsql_tmp-cleanup-into-two-stages.patch)
download | inline diff:
From 1f19b7887874b484a3b2afe0f3555599e58d426d Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Sun, 5 Dec 2021 21:16:44 -0800
Subject: [PATCH v2 3/8] Split pgsql_tmp cleanup into two stages.
First, pgsql_tmp directories will be renamed to stage them for
removal. Then, all files in pgsql_tmp are removed before removing
the staged directories themselves. This change is being made in
preparation for a follow-up change to offload most temporary file
cleanup to the new custodian process.
Note that temporary relation files cannot be cleaned up via the
aforementioned strategy and will not be offloaded to the custodian.
---
src/backend/postmaster/postmaster.c | 8 +-
src/backend/storage/file/fd.c | 176 +++++++++++++++++++++++++++++++-----
src/include/storage/fd.h | 2 +-
3 files changed, 162 insertions(+), 24 deletions(-)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 51613aaa2a..1ae2dc179e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -1392,7 +1392,8 @@ PostmasterMain(int argc, char *argv[])
* Remove old temporary files. At this point there can be no other
* Postgres processes running in this directory, so this should be safe.
*/
- RemovePgTempFiles();
+ RemovePgTempFiles(true, true);
+ RemovePgTempFiles(false, false);
/*
* Initialize stats collection subsystem (this does NOT start the
@@ -4140,7 +4141,10 @@ PostmasterStateMachine(void)
/* remove leftover temporary files after a crash */
if (remove_temp_files_after_crash)
- RemovePgTempFiles();
+ {
+ RemovePgTempFiles(true, true);
+ RemovePgTempFiles(false, false);
+ }
/* allow background workers to immediately restart */
ResetBackgroundWorkerCrashTimes();
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 545e91978c..633c6eee18 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -112,6 +112,8 @@
#define PG_FLUSH_DATA_WORKS 1
#endif
+#define PG_TEMP_DIR_TO_REMOVE_PREFIX (PG_TEMP_FILES_DIR "_to_remove_")
+
/*
* We must leave some file descriptors free for system(), the dynamic loader,
* and other code that tries to open files without consulting fd.c. This
@@ -338,6 +340,8 @@ static void BeforeShmemExit_Files(int code, Datum arg);
static void CleanupTempFiles(bool isCommit, bool isProcExit);
static void RemovePgTempRelationFiles(const char *tsdirname);
static void RemovePgTempRelationFilesInDbspace(const char *dbspacedirname);
+static void StagePgTempDirForRemoval(const char *tmp_dir);
+static void RemoveStagedPgTempDirs(const char *spc_dir);
static void walkdir(const char *path,
void (*action) (const char *fname, bool isdir, int elevel),
@@ -3133,24 +3137,20 @@ CleanupTempFiles(bool isCommit, bool isProcExit)
* Remove temporary and temporary relation files left over from a prior
* postmaster session
*
- * This should be called during postmaster startup. It will forcibly
- * remove any leftover files created by OpenTemporaryFile and any leftover
- * temporary relation files created by mdcreate.
+ * If stage is true, this function will simply rename all pgsql_tmp directories
+ * to stage them for removal at a later time. If stage is false, this function
+ * will delete all files in the staged directories as well as the directories
+ * themselves.
*
- * During post-backend-crash restart cycle, this routine is called when
- * remove_temp_files_after_crash GUC is enabled. Multiple crashes while
- * queries are using temp files could result in useless storage usage that can
- * only be reclaimed by a service restart. The argument against enabling it is
- * that someone might want to examine the temporary files for debugging
- * purposes. This does however mean that OpenTemporaryFile had better allow for
- * collision with an existing temp file name.
+ * If remove_relation_files is true, this function will remove the temporary
+ * relation files. Otherwise, this step is skipped.
*
* NOTE: this function and its subroutines generally report syscall failures
* with ereport(LOG) and keep going. Removing temp files is not so critical
* that we should fail to start the database when we can't do it.
*/
void
-RemovePgTempFiles(void)
+RemovePgTempFiles(bool stage, bool remove_relation_files)
{
char temp_path[MAXPGPATH + 10 + sizeof(TABLESPACE_VERSION_DIRECTORY) + sizeof(PG_TEMP_FILES_DIR)];
DIR *spc_dir;
@@ -3159,9 +3159,16 @@ RemovePgTempFiles(void)
/*
* First process temp files in pg_default ($PGDATA/base)
*/
- snprintf(temp_path, sizeof(temp_path), "base/%s", PG_TEMP_FILES_DIR);
- RemovePgTempDir(temp_path, true, false);
- RemovePgTempRelationFiles("base");
+ if (stage)
+ {
+ snprintf(temp_path, sizeof(temp_path), "base/%s", PG_TEMP_FILES_DIR);
+ StagePgTempDirForRemoval(temp_path);
+ }
+ else
+ RemoveStagedPgTempDirs("base");
+
+ if (remove_relation_files)
+ RemovePgTempRelationFiles("base");
/*
* Cycle through temp directories for all non-default tablespaces.
@@ -3174,13 +3181,26 @@ RemovePgTempFiles(void)
strcmp(spc_de->d_name, "..") == 0)
continue;
- snprintf(temp_path, sizeof(temp_path), "pg_tblspc/%s/%s/%s",
- spc_de->d_name, TABLESPACE_VERSION_DIRECTORY, PG_TEMP_FILES_DIR);
- RemovePgTempDir(temp_path, true, false);
+ if (stage)
+ {
+ snprintf(temp_path, sizeof(temp_path), "pg_tblspc/%s/%s/%s",
+ spc_de->d_name, TABLESPACE_VERSION_DIRECTORY,
+ PG_TEMP_FILES_DIR);
+ StagePgTempDirForRemoval(temp_path);
+ }
+ else
+ {
+ snprintf(temp_path, sizeof(temp_path), "pg_tblspc/%s/%s",
+ spc_de->d_name, TABLESPACE_VERSION_DIRECTORY);
+ RemoveStagedPgTempDirs(temp_path);
+ }
- snprintf(temp_path, sizeof(temp_path), "pg_tblspc/%s/%s",
- spc_de->d_name, TABLESPACE_VERSION_DIRECTORY);
- RemovePgTempRelationFiles(temp_path);
+ if (remove_relation_files)
+ {
+ snprintf(temp_path, sizeof(temp_path), "pg_tblspc/%s/%s",
+ spc_de->d_name, TABLESPACE_VERSION_DIRECTORY);
+ RemovePgTempRelationFiles(temp_path);
+ }
}
FreeDir(spc_dir);
@@ -3194,7 +3214,121 @@ RemovePgTempFiles(void)
}
/*
- * Process one pgsql_tmp directory for RemovePgTempFiles.
+ * StagePgTempDirForRemoval
+ *
+ * This function renames the given directory with a special prefix that
+ * RemoveStagedPgTempDirs() will know to look for. An integer is appended to
+ * the end of the new directory name in case previously staged pgsql_tmp
+ * directories have not yet been removed.
+ */
+static void
+StagePgTempDirForRemoval(const char *tmp_dir)
+{
+ DIR *dir;
+ char stage_path[MAXPGPATH * 2];
+ char parent_path[MAXPGPATH * 2];
+
+ /*
+ * If tmp_dir doesn't exist, there is nothing to stage.
+ */
+ dir = AllocateDir(tmp_dir);
+ if (dir == NULL)
+ {
+ if (errno != ENOENT)
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not open directory \"%s\": %m", tmp_dir)));
+ return;
+ }
+ FreeDir(dir);
+
+ strlcpy(parent_path, tmp_dir, MAXPGPATH * 2);
+ get_parent_directory(parent_path);
+
+ /*
+ * get_parent_directory() returns an empty string if the input argument is
+ * just a file name (see comments in path.c), so handle that as being the
+ * current directory.
+ */
+ if (strlen(parent_path) == 0)
+ strlcpy(parent_path, ".", MAXPGPATH * 2);
+
+ /*
+ * Find a name for the stage directory. We just increment an integer at the
+ * end of the name until we find one that doesn't exist.
+ */
+ for (int n = 0; n <= INT_MAX; n++)
+ {
+ snprintf(stage_path, sizeof(stage_path), "%s/%s%d", parent_path,
+ PG_TEMP_DIR_TO_REMOVE_PREFIX, n);
+
+ dir = AllocateDir(stage_path);
+ if (dir == NULL)
+ {
+ if (errno == ENOENT)
+ break;
+
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not open directory \"%s\": %m",
+ stage_path)));
+ return;
+ }
+ FreeDir(dir);
+
+ stage_path[0] = '\0';
+ }
+
+ /*
+ * In the unlikely event that we couldn't find a name for the stage
+ * directory, bail out.
+ */
+ if (stage_path[0] == '\0')
+ {
+ ereport(LOG,
+ (errmsg("could not stage \"%s\" for deletion",
+ tmp_dir)));
+ return;
+ }
+
+ /*
+ * Rename the temporary directory.
+ */
+ if (rename(tmp_dir, stage_path) != 0)
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not rename directory \"%s\" to \"%s\": %m",
+ tmp_dir, stage_path)));
+}
+
+/*
+ * RemoveStagedPgTempDirs
+ *
+ * This function removes all pgsql_tmp directories that have been staged for
+ * removal by StagePgTempDirForRemoval() in the given tablespace directory.
+ */
+static void
+RemoveStagedPgTempDirs(const char *spc_dir)
+{
+ char temp_path[MAXPGPATH * 2];
+ DIR *dir;
+ struct dirent *de;
+
+ dir = AllocateDir(spc_dir);
+ while ((de = ReadDirExtended(dir, spc_dir, LOG)) != NULL)
+ {
+ if (strncmp(de->d_name, PG_TEMP_DIR_TO_REMOVE_PREFIX,
+ strlen(PG_TEMP_DIR_TO_REMOVE_PREFIX)) != 0)
+ continue;
+
+ snprintf(temp_path, sizeof(temp_path), "%s/%s", spc_dir, de->d_name);
+ RemovePgTempDir(temp_path, true, false);
+ }
+ FreeDir(dir);
+}
+
+/*
+ * Process one pgsql_tmp directory for RemoveStagedPgTempDirs.
*
* If missing_ok is true, it's all right for the named directory to not exist.
* Any other problem results in a LOG message. (missing_ok should be true at
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index 762f6b46c1..85fa987aca 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -168,7 +168,7 @@ extern Oid GetNextTempTableSpace(void);
extern void AtEOXact_Files(bool isCommit);
extern void AtEOSubXact_Files(bool isCommit, SubTransactionId mySubid,
SubTransactionId parentSubid);
-extern void RemovePgTempFiles(void);
+extern void RemovePgTempFiles(bool stage, bool remove_relation_files);
extern void RemovePgTempDir(const char *tmpdirname, bool missing_ok,
bool unlink_all);
extern bool looks_like_temp_rel_name(const char *name);
--
2.16.6
[application/octet-stream] v2-0002-Also-remove-pgsql_tmp-directories-during-startup.patch (4.8K, ../../[email protected]/9-v2-0002-Also-remove-pgsql_tmp-directories-during-startup.patch)
download | inline diff:
From 2546e9ed46c218903773aa50eaa1056b475a1a7b Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Sun, 5 Dec 2021 19:38:20 -0800
Subject: [PATCH v2 2/8] Also remove pgsql_tmp directories during startup.
Presently, the server only removes the contents of the temporary
directories during startup, not the directory itself. This changes
that to prepare for future commits that will move temporary file
cleanup to a separate auxiliary process.
---
src/backend/postmaster/postmaster.c | 2 +-
src/backend/storage/file/fd.c | 20 ++++++++++----------
src/include/storage/fd.h | 4 ++--
src/test/recovery/t/022_crash_temp_files.pl | 6 ++++--
4 files changed, 17 insertions(+), 15 deletions(-)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 635313cdb7..51613aaa2a 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -1117,7 +1117,7 @@ PostmasterMain(int argc, char *argv[])
* safe to do so now, because we verified earlier that there are no
* conflicting Postgres processes in this data directory.
*/
- RemovePgTempFilesInDir(PG_TEMP_FILES_DIR, true, false);
+ RemovePgTempDir(PG_TEMP_FILES_DIR, true, false);
#endif
/*
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 263057841d..545e91978c 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -3160,7 +3160,7 @@ RemovePgTempFiles(void)
* First process temp files in pg_default ($PGDATA/base)
*/
snprintf(temp_path, sizeof(temp_path), "base/%s", PG_TEMP_FILES_DIR);
- RemovePgTempFilesInDir(temp_path, true, false);
+ RemovePgTempDir(temp_path, true, false);
RemovePgTempRelationFiles("base");
/*
@@ -3176,7 +3176,7 @@ RemovePgTempFiles(void)
snprintf(temp_path, sizeof(temp_path), "pg_tblspc/%s/%s/%s",
spc_de->d_name, TABLESPACE_VERSION_DIRECTORY, PG_TEMP_FILES_DIR);
- RemovePgTempFilesInDir(temp_path, true, false);
+ RemovePgTempDir(temp_path, true, false);
snprintf(temp_path, sizeof(temp_path), "pg_tblspc/%s/%s",
spc_de->d_name, TABLESPACE_VERSION_DIRECTORY);
@@ -3209,7 +3209,7 @@ RemovePgTempFiles(void)
* them separate.)
*/
void
-RemovePgTempFilesInDir(const char *tmpdirname, bool missing_ok, bool unlink_all)
+RemovePgTempDir(const char *tmpdirname, bool missing_ok, bool unlink_all)
{
DIR *temp_dir;
struct dirent *temp_de;
@@ -3247,13 +3247,7 @@ RemovePgTempFilesInDir(const char *tmpdirname, bool missing_ok, bool unlink_all)
if (S_ISDIR(statbuf.st_mode))
{
/* recursively remove contents, then directory itself */
- RemovePgTempFilesInDir(rm_path, false, true);
-
- if (rmdir(rm_path) < 0)
- ereport(LOG,
- (errcode_for_file_access(),
- errmsg("could not remove directory \"%s\": %m",
- rm_path)));
+ RemovePgTempDir(rm_path, false, true);
}
else
{
@@ -3271,6 +3265,12 @@ RemovePgTempFilesInDir(const char *tmpdirname, bool missing_ok, bool unlink_all)
}
FreeDir(temp_dir);
+
+ if (rmdir(tmpdirname) < 0)
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not remove directory \"%s\": %m",
+ tmpdirname)));
}
/* Process one tablespace directory, look for per-DB subdirectories */
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index 34602ae006..762f6b46c1 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -169,8 +169,8 @@ extern void AtEOXact_Files(bool isCommit);
extern void AtEOSubXact_Files(bool isCommit, SubTransactionId mySubid,
SubTransactionId parentSubid);
extern void RemovePgTempFiles(void);
-extern void RemovePgTempFilesInDir(const char *tmpdirname, bool missing_ok,
- bool unlink_all);
+extern void RemovePgTempDir(const char *tmpdirname, bool missing_ok,
+ bool unlink_all);
extern bool looks_like_temp_rel_name(const char *name);
extern int pg_fsync(int fd);
diff --git a/src/test/recovery/t/022_crash_temp_files.pl b/src/test/recovery/t/022_crash_temp_files.pl
index bf95a30761..481f1f23a2 100644
--- a/src/test/recovery/t/022_crash_temp_files.pl
+++ b/src/test/recovery/t/022_crash_temp_files.pl
@@ -143,7 +143,8 @@ $node->poll_query_until('postgres', undef, '');
# Check for temporary files
is( $node->safe_psql(
- 'postgres', 'SELECT COUNT(1) FROM pg_ls_dir($$base/pgsql_tmp$$)'),
+ 'postgres',
+ 'SELECT COUNT(1) FROM pg_ls_dir($$base$$) WHERE pg_ls_dir = \'pgsql_tmp\''),
qq(0),
'no temporary files');
@@ -241,7 +242,8 @@ $node->restart();
# Check the temporary files -- should be gone
is( $node->safe_psql(
- 'postgres', 'SELECT COUNT(1) FROM pg_ls_dir($$base/pgsql_tmp$$)'),
+ 'postgres',
+ 'SELECT COUNT(1) FROM pg_ls_dir($$base$$) WHERE pg_ls_dir = \'pgsql_tmp\''),
qq(0),
'temporary file was removed');
--
2.16.6
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
@ 2022-06-23 11:58 Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Simon Riggs @ 2022-06-23 11:58 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; Robert Haas <[email protected]>; pgsql-hackers
On Fri, 18 Feb 2022 at 20:51, Nathan Bossart <[email protected]> wrote:
>
> > On Thu, Feb 17, 2022 at 03:12:47PM -0800, Andres Freund wrote:
> >>> > The improvements around deleting temporary files and serialized snapshots
> >>> > afaict don't require a dedicated process - they're only relevant during
> >>> > startup. We could use the approach of renaming the directory out of the way as
> >>> > done in this patchset but perform the cleanup in the startup process after
> >>> > we're up.
>
> BTW I know you don't like the dedicated process approach, but one
> improvement to that approach could be to shut down the custodian process
> when it has nothing to do.
Having a central cleanup process makes a lot of sense. There is a long
list of potential tasks for such a process. My understanding is that
autovacuum already has an interface for handling additional workload
types, which is how BRIN indexes are handled. Do we really need a new
process? If so, lets do this now.
Nathan's point that certain tasks are blocking fast startup is a good
one and higher availability is a critical end goal. The thought that
we should complete these tasks during checkpoint is a good one, but
checkpoints should NOT be delayed by long running tasks that are
secondary to availability.
Andres' point that it would be better to avoid long running tasks is
good, if that is possible. That can be done better over time. This
point does not block the higher level goal of better availability
asap, so I support Nathan's overall proposals.
--
Simon Riggs http://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
@ 2022-06-23 13:46 ` Robert Haas <[email protected]>
2022-06-23 17:13 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
0 siblings, 2 replies; 59+ messages in thread
From: Robert Haas @ 2022-06-23 13:46 UTC (permalink / raw)
To: Simon Riggs <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Thu, Jun 23, 2022 at 7:58 AM Simon Riggs
<[email protected]> wrote:
> Having a central cleanup process makes a lot of sense. There is a long
> list of potential tasks for such a process. My understanding is that
> autovacuum already has an interface for handling additional workload
> types, which is how BRIN indexes are handled. Do we really need a new
> process?
It seems to me that if there's a long list of possible tasks for such
a process, that's actually a trickier situation than if there were
only one or two, because it may happen that when task X is really
urgent, the process is already busy with task Y.
I don't think that piggybacking more stuff onto autovacuum is a very
good idea for this exact reason. We already know that autovacuum
workers can get so busy that they can't keep up with the need to
vacuum and analyze tables. If we give them more things to do, that
figures to make it worse, at least on busy systems.
I do agree that a general mechanism for getting cleanup tasks done in
the background could be a useful thing to have, but I feel like it's
hard to see exactly how to make it work well. We can't just allow it
to spin up a million new processes, but at the same time, if it can't
guarantee that time-critical tasks get performed relatively quickly,
it's pretty worthless.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
@ 2022-06-23 17:13 ` Simon Riggs <[email protected]>
1 sibling, 0 replies; 59+ messages in thread
From: Simon Riggs @ 2022-06-23 17:13 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Thu, 23 Jun 2022 at 14:46, Robert Haas <[email protected]> wrote:
>
> On Thu, Jun 23, 2022 at 7:58 AM Simon Riggs
> <[email protected]> wrote:
> > Having a central cleanup process makes a lot of sense. There is a long
> > list of potential tasks for such a process. My understanding is that
> > autovacuum already has an interface for handling additional workload
> > types, which is how BRIN indexes are handled. Do we really need a new
> > process?
>
> It seems to me that if there's a long list of possible tasks for such
> a process, that's actually a trickier situation than if there were
> only one or two, because it may happen that when task X is really
> urgent, the process is already busy with task Y.
>
> I don't think that piggybacking more stuff onto autovacuum is a very
> good idea for this exact reason. We already know that autovacuum
> workers can get so busy that they can't keep up with the need to
> vacuum and analyze tables. If we give them more things to do, that
> figures to make it worse, at least on busy systems.
>
> I do agree that a general mechanism for getting cleanup tasks done in
> the background could be a useful thing to have, but I feel like it's
> hard to see exactly how to make it work well. We can't just allow it
> to spin up a million new processes, but at the same time, if it can't
> guarantee that time-critical tasks get performed relatively quickly,
> it's pretty worthless.
Most of the tasks mentioned aren't time critical.
I have no objection to a new auxiliary process to execute those tasks,
which can be spawned when needed.
--
Simon Riggs http://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
@ 2022-06-23 17:15 ` Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
1 sibling, 1 reply; 59+ messages in thread
From: Nathan Bossart @ 2022-06-23 17:15 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Simon Riggs <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Thu, Jun 23, 2022 at 09:46:28AM -0400, Robert Haas wrote:
> I do agree that a general mechanism for getting cleanup tasks done in
> the background could be a useful thing to have, but I feel like it's
> hard to see exactly how to make it work well. We can't just allow it
> to spin up a million new processes, but at the same time, if it can't
> guarantee that time-critical tasks get performed relatively quickly,
> it's pretty worthless.
My intent with this new auxiliary process is to offload tasks that aren't
particularly time-critical. They are only time-critical in the sense that
1) you might eventually run out of space and 2) you might encounter
wraparound with the logical replication files. But AFAICT these same risks
exist today in the checkpointer approach, although maybe not to the same
extent. In any case, 2 seems solvable to me outside of this patch set.
I'm grateful for the discussion in this thread so far, but I'm not seeing a
clear path forward. I'm glad to see threads like the one to stop doing
end-of-recovery checkpoints [0], but I don't know if it will be possible to
solve all of these availability concerns in a piecemeal fashion. I remain
open to exploring other suggested approaches beyond creating a new
auxiliary process.
[0] https://postgr.es/m/CA%2BTgmobrM2jvkiccCS9NgFcdjNSgAvk1qcAPx5S6F%2BoJT3D2mQ%40mail.gmail.com
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
@ 2022-06-24 10:45 ` Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Simon Riggs @ 2022-06-24 10:45 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Thu, 23 Jun 2022 at 18:15, Nathan Bossart <[email protected]> wrote:
> I'm grateful for the discussion in this thread so far, but I'm not seeing a
> clear path forward.
+1 to add the new auxiliary process.
--
Simon Riggs http://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
@ 2022-07-02 22:05 ` Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Nathan Bossart @ 2022-07-02 22:05 UTC (permalink / raw)
To: Simon Riggs <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Fri, Jun 24, 2022 at 11:45:22AM +0100, Simon Riggs wrote:
> On Thu, 23 Jun 2022 at 18:15, Nathan Bossart <[email protected]> wrote:
>> I'm grateful for the discussion in this thread so far, but I'm not seeing a
>> clear path forward.
>
> +1 to add the new auxiliary process.
I went ahead and put together a new patch set for this in which I've
attempted to address most of the feedback from upthread. Notably, I've
abandoned 0007 and 0008, added a way for processes to request specific
tasks for the custodian, and removed all the checks for
ShutdownRequestPending.
I haven't addressed the existing transaction ID wraparound risk with the
logical replication files. My instinct is that this deserveѕ its own
thread, and it might need to be considered a prerequisite to this change
based on the prior discussion here.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
@ 2022-07-02 22:54 ` Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Andres Freund @ 2022-07-02 22:54 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Simon Riggs <[email protected]>; Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
Hi,
On 2022-07-02 15:05:54 -0700, Nathan Bossart wrote:
> + /* Obtain requested tasks */
> + SpinLockAcquire(&CustodianShmem->cust_lck);
> + flags = CustodianShmem->cust_flags;
> + CustodianShmem->cust_flags = 0;
> + SpinLockRelease(&CustodianShmem->cust_lck);
Just resetting the flags to 0 is problematic. Consider what happens if there's
two tasks and and the one processed first errors out. You'll loose information
about needing to run the second task.
> + /* TODO: offloaded tasks go here */
Seems we're going to need some sorting of which tasks are most "urgent" / need
to be processed next if we plan to make this into some generic facility.
> +/*
> + * RequestCustodian
> + * Called to request a custodian task.
> + */
> +void
> +RequestCustodian(int flags)
> +{
> + SpinLockAcquire(&CustodianShmem->cust_lck);
> + CustodianShmem->cust_flags |= flags;
> + SpinLockRelease(&CustodianShmem->cust_lck);
> +
> + if (ProcGlobal->custodianLatch)
> + SetLatch(ProcGlobal->custodianLatch);
> +}
With this representation we can't really implement waiting for a task or
such. And it doesn't seem like a great API for the caller to just specify a
mix of flags.
> + /* Calculate how long to sleep */
> + end_time = (pg_time_t) time(NULL);
> + elapsed_secs = end_time - start_time;
> + if (elapsed_secs >= CUSTODIAN_TIMEOUT_S)
> + continue; /* no sleep for us */
> + cur_timeout = CUSTODIAN_TIMEOUT_S - elapsed_secs;
> +
> + (void) WaitLatch(MyLatch,
> + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
> + cur_timeout * 1000L /* convert to ms */ ,
> + WAIT_EVENT_CUSTODIAN_MAIN);
> + }
I don't think we should have this thing wake up on a regular basis. We're
doing way too much of that already, and I don't think we should add
more. Either we need a list of times when tasks need to be processed and wake
up at that time, or just wake up if somebody requests a task.
> From 5e95666efa31d6c8aa351e430c37ead6e27acb72 Mon Sep 17 00:00:00 2001
> From: Nathan Bossart <[email protected]>
> Date: Sun, 5 Dec 2021 21:16:44 -0800
> Subject: [PATCH v6 3/6] Split pgsql_tmp cleanup into two stages.
>
> First, pgsql_tmp directories will be renamed to stage them for
> removal. Then, all files in pgsql_tmp are removed before removing
> the staged directories themselves. This change is being made in
> preparation for a follow-up change to offload most temporary file
> cleanup to the new custodian process.
> Note that temporary relation files cannot be cleaned up via the
> aforementioned strategy and will not be offloaded to the custodian.
> ---
> src/backend/postmaster/postmaster.c | 8 +-
> src/backend/storage/file/fd.c | 174 ++++++++++++++++++++++++----
> src/include/storage/fd.h | 2 +-
> 3 files changed, 160 insertions(+), 24 deletions(-)
>
> diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
> index e67370012f..82aa0c6307 100644
> --- a/src/backend/postmaster/postmaster.c
> +++ b/src/backend/postmaster/postmaster.c
> @@ -1402,7 +1402,8 @@ PostmasterMain(int argc, char *argv[])
> * Remove old temporary files. At this point there can be no other
> * Postgres processes running in this directory, so this should be safe.
> */
> - RemovePgTempFiles();
> + RemovePgTempFiles(true, true);
> + RemovePgTempFiles(false, false);
This is imo hard to read and easy to get wrong. Make it multiple functions or
pass named flags in.
> + * StagePgTempDirForRemoval
> + *
> + * This function renames the given directory with a special prefix that
> + * RemoveStagedPgTempDirs() will know to look for. An integer is appended to
> + * the end of the new directory name in case previously staged pgsql_tmp
> + * directories have not yet been removed.
> + */
It doesn't seem great to need to iterate through a directory that contains
other files, potentially a significant number. How about having a
staged_for_removal/ directory, and then only scanning that?
> +static void
> +StagePgTempDirForRemoval(const char *tmp_dir)
> +{
> + DIR *dir;
> + char stage_path[MAXPGPATH * 2];
> + char parent_path[MAXPGPATH * 2];
> + struct stat statbuf;
> +
> + /*
> + * If tmp_dir doesn't exist, there is nothing to stage.
> + */
> + dir = AllocateDir(tmp_dir);
> + if (dir == NULL)
> + {
> + if (errno != ENOENT)
> + ereport(LOG,
> + (errcode_for_file_access(),
> + errmsg("could not open directory \"%s\": %m", tmp_dir)));
> + return;
> + }
> + FreeDir(dir);
> +
> + strlcpy(parent_path, tmp_dir, MAXPGPATH * 2);
> + get_parent_directory(parent_path);
> +
> + /*
> + * get_parent_directory() returns an empty string if the input argument is
> + * just a file name (see comments in path.c), so handle that as being the
> + * current directory.
> + */
> + if (strlen(parent_path) == 0)
> + strlcpy(parent_path, ".", MAXPGPATH * 2);
> +
> + /*
> + * Find a name for the stage directory. We just increment an integer at the
> + * end of the name until we find one that doesn't exist.
> + */
> + for (int n = 0; n <= INT_MAX; n++)
> + {
> + snprintf(stage_path, sizeof(stage_path), "%s/%s%d", parent_path,
> + PG_TEMP_DIR_TO_REMOVE_PREFIX, n);
> +
> + if (stat(stage_path, &statbuf) != 0)
> + {
> + if (errno == ENOENT)
> + break;
> +
> + ereport(LOG,
> + (errcode_for_file_access(),
> + errmsg("could not stat file \"%s\": %m", stage_path)));
> + return;
> + }
> +
> + stage_path[0] = '\0';
I still dislike this approach. Loops until INT_MAX, not interruptible... Can't
we prevent conflicts by adding a timestamp or such?
> + }
> +
> + /*
> + * In the unlikely event that we couldn't find a name for the stage
> + * directory, bail out.
> + */
> + if (stage_path[0] == '\0')
> + {
> + ereport(LOG,
> + (errmsg("could not stage \"%s\" for deletion",
> + tmp_dir)));
> + return;
> + }
That's imo very much not ok. Just continuing in unexpected situations is a
recipe for introducing bugs / being hard to debug.
> From 43042799b96b588a446c509637b5acf570e2a325 Mon Sep 17 00:00:00 2001
> From a58a6bb70785a557a150680b64cd8ce78ce1b73a Mon Sep 17 00:00:00 2001
> From: Nathan Bossart <[email protected]>
> Date: Sun, 5 Dec 2021 22:02:40 -0800
> Subject: [PATCH v6 5/6] Move removal of old serialized snapshots to custodian.
>
> This was only done during checkpoints because it was a convenient
> place to put it.
As mentioned before, having it done as part of checkpoints provides pretty
decent wraparound protection - yes, it's not theoretically perfect, but in
reality it's very unlikely you can have an xid wraparound within one
checkpoint. I've mentioned this before, so at the very least I'd like to see
this acknowledged in the commit message.
> However, if there are many snapshots to remove, it can significantly extend
> checkpoint time.
I'd really like to see a reproducer or profile for this...
> + /*
> + * Remove serialized snapshots that are no longer required by any
> + * logical replication slot.
> + *
> + * It is not important for these to be removed in single-user mode, so
> + * we don't need any extra handling outside of the custodian process for
> + * this.
> + */
I don't think this claim is correct.
> From 0add8bb19a4ee83c6a6ec1f313329d737bf304a5 Mon Sep 17 00:00:00 2001
> From: Nathan Bossart <[email protected]>
> Date: Sun, 12 Dec 2021 22:07:11 -0800
> Subject: [PATCH v6 6/6] Move removal of old logical rewrite mapping files to
> custodian.
>
> If there are many such files to remove, checkpoints can take much
> longer. To avoid this, move this work to the newly-introduced
> custodian process.
As above I'd like to know why this could take that long. What are you doing
that there's so many mapping files (which only exist for catalog tables!) that
this is a significant fraction of a checkpoint?
> ---
> src/backend/access/heap/rewriteheap.c | 79 +++++++++++++++++++++++----
> src/backend/postmaster/custodian.c | 44 +++++++++++++++
> src/include/access/rewriteheap.h | 1 +
> src/include/postmaster/custodian.h | 5 ++
> 4 files changed, 119 insertions(+), 10 deletions(-)
>
> diff --git a/src/backend/access/heap/rewriteheap.c b/src/backend/access/heap/rewriteheap.c
> index 2a53826736..edeab65e60 100644
> --- a/src/backend/access/heap/rewriteheap.c
> +++ b/src/backend/access/heap/rewriteheap.c
> @@ -116,6 +116,7 @@
> #include "lib/ilist.h"
> #include "miscadmin.h"
> #include "pgstat.h"
> +#include "postmaster/custodian.h"
> #include "replication/logical.h"
> #include "replication/slot.h"
> #include "storage/bufmgr.h"
> @@ -1182,7 +1183,8 @@ heap_xlog_logical_rewrite(XLogReaderState *r)
> * Perform a checkpoint for logical rewrite mappings
> *
> * This serves two tasks:
> - * 1) Remove all mappings not needed anymore based on the logical restart LSN
> + * 1) Alert the custodian to remove all mappings not needed anymore based on the
> + * logical restart LSN
> * 2) Flush all remaining mappings to disk, so that replay after a checkpoint
> * only has to deal with the parts of a mapping that have been written out
> * after the checkpoint started.
> @@ -1210,6 +1212,10 @@ CheckPointLogicalRewriteHeap(void)
> if (cutoff != InvalidXLogRecPtr && redo < cutoff)
> cutoff = redo;
>
> + /* let the custodian know what it can remove */
> + CustodianSetLogicalRewriteCutoff(cutoff);
Setting this variable in a custodian datastructure and then fetching it from
there seems architecturally wrong to me.
> + RequestCustodian(CUSTODIAN_REMOVE_REWRITE_MAPPINGS);
What about single user mode?
ISTM that RequestCustodian() needs to either assert out if called in single
user mode, or execute tasks immediately in that context.
> +
> +/*
> + * Remove all mappings not needed anymore based on the logical restart LSN saved
> + * by the checkpointer. We use this saved value instead of calling
> + * ReplicationSlotsComputeLogicalRestartLSN() so that we don't interfere with an
> + * ongoing call to CheckPointLogicalRewriteHeap() that is flushing mappings to
> + * disk.
> + */
What interference could there be?
> +void
> +RemoveOldLogicalRewriteMappings(void)
> +{
> + XLogRecPtr cutoff;
> + DIR *mappings_dir;
> + struct dirent *mapping_de;
> + char path[MAXPGPATH + 20];
> + bool value_set = false;
> +
> + cutoff = CustodianGetLogicalRewriteCutoff(&value_set);
> + if (!value_set)
> + return;
Afaics nothing clears values_set - is that a good idea?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
@ 2022-07-03 17:07 ` Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Nathan Bossart @ 2022-07-03 17:07 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Simon Riggs <[email protected]>; Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
Hi Andres,
Thanks for the prompt review.
On Sat, Jul 02, 2022 at 03:54:56PM -0700, Andres Freund wrote:
> On 2022-07-02 15:05:54 -0700, Nathan Bossart wrote:
>> + /* Obtain requested tasks */
>> + SpinLockAcquire(&CustodianShmem->cust_lck);
>> + flags = CustodianShmem->cust_flags;
>> + CustodianShmem->cust_flags = 0;
>> + SpinLockRelease(&CustodianShmem->cust_lck);
>
> Just resetting the flags to 0 is problematic. Consider what happens if there's
> two tasks and and the one processed first errors out. You'll loose information
> about needing to run the second task.
I think we also want to retry any failed tasks. The way v6 handles this is
by requesting all tasks after an exception. Another way to handle this
could be to reset each individual flag before the task is executed, and
then we could surround each one with a PG_CATCH block that resets the flag.
I'll do it this way in the next revision.
>> +/*
>> + * RequestCustodian
>> + * Called to request a custodian task.
>> + */
>> +void
>> +RequestCustodian(int flags)
>> +{
>> + SpinLockAcquire(&CustodianShmem->cust_lck);
>> + CustodianShmem->cust_flags |= flags;
>> + SpinLockRelease(&CustodianShmem->cust_lck);
>> +
>> + if (ProcGlobal->custodianLatch)
>> + SetLatch(ProcGlobal->custodianLatch);
>> +}
>
> With this representation we can't really implement waiting for a task or
> such. And it doesn't seem like a great API for the caller to just specify a
> mix of flags.
At the moment, the idea is that nothing should need to wait for a task
because the custodian only handles things that are relatively non-critical.
If that changes, this could probably be expanded to look more like
RequestCheckpoint().
What would you suggest using instead of a mix of flags?
>> + /* Calculate how long to sleep */
>> + end_time = (pg_time_t) time(NULL);
>> + elapsed_secs = end_time - start_time;
>> + if (elapsed_secs >= CUSTODIAN_TIMEOUT_S)
>> + continue; /* no sleep for us */
>> + cur_timeout = CUSTODIAN_TIMEOUT_S - elapsed_secs;
>> +
>> + (void) WaitLatch(MyLatch,
>> + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
>> + cur_timeout * 1000L /* convert to ms */ ,
>> + WAIT_EVENT_CUSTODIAN_MAIN);
>> + }
>
> I don't think we should have this thing wake up on a regular basis. We're
> doing way too much of that already, and I don't think we should add
> more. Either we need a list of times when tasks need to be processed and wake
> up at that time, or just wake up if somebody requests a task.
I agree. I will remove the timeout in the next revision.
>> diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
>> index e67370012f..82aa0c6307 100644
>> --- a/src/backend/postmaster/postmaster.c
>> +++ b/src/backend/postmaster/postmaster.c
>> @@ -1402,7 +1402,8 @@ PostmasterMain(int argc, char *argv[])
>> * Remove old temporary files. At this point there can be no other
>> * Postgres processes running in this directory, so this should be safe.
>> */
>> - RemovePgTempFiles();
>> + RemovePgTempFiles(true, true);
>> + RemovePgTempFiles(false, false);
>
> This is imo hard to read and easy to get wrong. Make it multiple functions or
> pass named flags in.
Will do.
>> + * StagePgTempDirForRemoval
>> + *
>> + * This function renames the given directory with a special prefix that
>> + * RemoveStagedPgTempDirs() will know to look for. An integer is appended to
>> + * the end of the new directory name in case previously staged pgsql_tmp
>> + * directories have not yet been removed.
>> + */
>
> It doesn't seem great to need to iterate through a directory that contains
> other files, potentially a significant number. How about having a
> staged_for_removal/ directory, and then only scanning that?
Yeah, that seems like a good idea. Will do.
>> + /*
>> + * Find a name for the stage directory. We just increment an integer at the
>> + * end of the name until we find one that doesn't exist.
>> + */
>> + for (int n = 0; n <= INT_MAX; n++)
>> + {
>> + snprintf(stage_path, sizeof(stage_path), "%s/%s%d", parent_path,
>> + PG_TEMP_DIR_TO_REMOVE_PREFIX, n);
>> +
>> + if (stat(stage_path, &statbuf) != 0)
>> + {
>> + if (errno == ENOENT)
>> + break;
>> +
>> + ereport(LOG,
>> + (errcode_for_file_access(),
>> + errmsg("could not stat file \"%s\": %m", stage_path)));
>> + return;
>> + }
>> +
>> + stage_path[0] = '\0';
>
> I still dislike this approach. Loops until INT_MAX, not interruptible... Can't
> we prevent conflicts by adding a timestamp or such?
I suppose it's highly unlikely that we'd see a conflict if we used the
timestamp instead. I'll do it this way in the next revision if that seems
good enough.
>> From a58a6bb70785a557a150680b64cd8ce78ce1b73a Mon Sep 17 00:00:00 2001
>> From: Nathan Bossart <[email protected]>
>> Date: Sun, 5 Dec 2021 22:02:40 -0800
>> Subject: [PATCH v6 5/6] Move removal of old serialized snapshots to custodian.
>>
>> This was only done during checkpoints because it was a convenient
>> place to put it.
>
> As mentioned before, having it done as part of checkpoints provides pretty
> decent wraparound protection - yes, it's not theoretically perfect, but in
> reality it's very unlikely you can have an xid wraparound within one
> checkpoint. I've mentioned this before, so at the very least I'd like to see
> this acknowledged in the commit message.
Will do.
>> + /* let the custodian know what it can remove */
>> + CustodianSetLogicalRewriteCutoff(cutoff);
>
> Setting this variable in a custodian datastructure and then fetching it from
> there seems architecturally wrong to me.
Where do you think it should go? I previously had it in the checkpointer's
shared memory, but you didn't like that the functions were declared in
bgwriter.h (along with the other checkpoint stuff). If the checkpointer
shared memory is the right place, should we create checkpointer.h and use
that instead?
>> + RequestCustodian(CUSTODIAN_REMOVE_REWRITE_MAPPINGS);
>
> What about single user mode?
>
>
> ISTM that RequestCustodian() needs to either assert out if called in single
> user mode, or execute tasks immediately in that context.
I like the idea of executing the tasks immediately since that's what
happens today in single-user mode. I will try doing it that way.
>> +/*
>> + * Remove all mappings not needed anymore based on the logical restart LSN saved
>> + * by the checkpointer. We use this saved value instead of calling
>> + * ReplicationSlotsComputeLogicalRestartLSN() so that we don't interfere with an
>> + * ongoing call to CheckPointLogicalRewriteHeap() that is flushing mappings to
>> + * disk.
>> + */
>
> What interference could there be?
My concern is that the custodian could obtain a later cutoff than what the
checkpointer does, which might cause files to be concurrently unlinked and
fsync'd. If we always use the checkpointer's cutoff, that shouldn't be a
problem. This could probably be better explained in this comment.
>> +void
>> +RemoveOldLogicalRewriteMappings(void)
>> +{
>> + XLogRecPtr cutoff;
>> + DIR *mappings_dir;
>> + struct dirent *mapping_de;
>> + char path[MAXPGPATH + 20];
>> + bool value_set = false;
>> +
>> + cutoff = CustodianGetLogicalRewriteCutoff(&value_set);
>> + if (!value_set)
>> + return;
>
> Afaics nothing clears values_set - is that a good idea?
I'm using value_set to differentiate the case where InvalidXLogRecPtr means
the checkpointer hasn't determined a value yet versus the case where it
has. In the former, we don't want to take any action. In the latter, we
want to unlink all the files. Since we're moving to a request model for
the custodian, I might be able to remove this value_set stuff completely.
If that's not possible, it probably deserves a better comment.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
@ 2022-07-03 17:27 ` Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Andres Freund @ 2022-07-03 17:27 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Simon Riggs <[email protected]>; Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
Hi,
On 2022-07-03 10:07:54 -0700, Nathan Bossart wrote:
> Thanks for the prompt review.
>
> On Sat, Jul 02, 2022 at 03:54:56PM -0700, Andres Freund wrote:
> > On 2022-07-02 15:05:54 -0700, Nathan Bossart wrote:
> >> + /* Obtain requested tasks */
> >> + SpinLockAcquire(&CustodianShmem->cust_lck);
> >> + flags = CustodianShmem->cust_flags;
> >> + CustodianShmem->cust_flags = 0;
> >> + SpinLockRelease(&CustodianShmem->cust_lck);
> >
> > Just resetting the flags to 0 is problematic. Consider what happens if there's
> > two tasks and and the one processed first errors out. You'll loose information
> > about needing to run the second task.
>
> I think we also want to retry any failed tasks.
I don't think so, at least not if it's just going to retry that task straight
away - then we'll get stuck on that one task forever. If we had the ability to
"queue" it the end, to be processed after other already dequeued tasks, it'd
be a different story.
> The way v6 handles this is by requesting all tasks after an exception.
Ick. That strikes me as a bad idea.
> >> +/*
> >> + * RequestCustodian
> >> + * Called to request a custodian task.
> >> + */
> >> +void
> >> +RequestCustodian(int flags)
> >> +{
> >> + SpinLockAcquire(&CustodianShmem->cust_lck);
> >> + CustodianShmem->cust_flags |= flags;
> >> + SpinLockRelease(&CustodianShmem->cust_lck);
> >> +
> >> + if (ProcGlobal->custodianLatch)
> >> + SetLatch(ProcGlobal->custodianLatch);
> >> +}
> >
> > With this representation we can't really implement waiting for a task or
> > such. And it doesn't seem like a great API for the caller to just specify a
> > mix of flags.
>
> At the moment, the idea is that nothing should need to wait for a task
> because the custodian only handles things that are relatively non-critical.
Which is just plainly not true as the patchset stands...
I think we're going to have to block if some cleanup as part of a checkpoint
hasn't been completed by the next checkpoint - otherwise it'll just end up
being way too confusing and there's absolutely no backpressure anymore.
> If that changes, this could probably be expanded to look more like
> RequestCheckpoint().
>
> What would you suggest using instead of a mix of flags?
I suspect an array of tasks with requested and completed counters or such?
With a condition variable to wait on?
> >> + /* let the custodian know what it can remove */
> >> + CustodianSetLogicalRewriteCutoff(cutoff);
> >
> > Setting this variable in a custodian datastructure and then fetching it from
> > there seems architecturally wrong to me.
>
> Where do you think it should go? I previously had it in the checkpointer's
> shared memory, but you didn't like that the functions were declared in
> bgwriter.h (along with the other checkpoint stuff). If the checkpointer
> shared memory is the right place, should we create checkpointer.h and use
> that instead?
Well, so far I have not understood what the whole point of the shared state
is, so i have a bit of a hard time answering this ;)
> >> +/*
> >> + * Remove all mappings not needed anymore based on the logical restart LSN saved
> >> + * by the checkpointer. We use this saved value instead of calling
> >> + * ReplicationSlotsComputeLogicalRestartLSN() so that we don't interfere with an
> >> + * ongoing call to CheckPointLogicalRewriteHeap() that is flushing mappings to
> >> + * disk.
> >> + */
> >
> > What interference could there be?
>
> My concern is that the custodian could obtain a later cutoff than what the
> checkpointer does, which might cause files to be concurrently unlinked and
> fsync'd. If we always use the checkpointer's cutoff, that shouldn't be a
> problem. This could probably be better explained in this comment.
How about having a Datum argument to RequestCustodian() that is forwarded to
the task?
> >> +void
> >> +RemoveOldLogicalRewriteMappings(void)
> >> +{
> >> + XLogRecPtr cutoff;
> >> + DIR *mappings_dir;
> >> + struct dirent *mapping_de;
> >> + char path[MAXPGPATH + 20];
> >> + bool value_set = false;
> >> +
> >> + cutoff = CustodianGetLogicalRewriteCutoff(&value_set);
> >> + if (!value_set)
> >> + return;
> >
> > Afaics nothing clears values_set - is that a good idea?
>
> I'm using value_set to differentiate the case where InvalidXLogRecPtr means
> the checkpointer hasn't determined a value yet versus the case where it
> has. In the former, we don't want to take any action. In the latter, we
> want to unlink all the files. Since we're moving to a request model for
> the custodian, I might be able to remove this value_set stuff completely.
> If that's not possible, it probably deserves a better comment.
It would.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
@ 2022-07-06 16:51 ` Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Nathan Bossart @ 2022-07-06 16:51 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Simon Riggs <[email protected]>; Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
Here's a new revision where I've attempted to address all the feedback I've
received thus far. Notably, the custodian now uses a queue for registering
tasks and determining which tasks to execute. Other changes include
splitting the temporary file functions apart to avoid consecutive boolean
flags, using a timestamp instead of an integer for the staging name for
temporary directories, moving temporary directories to a dedicated
directory so that the custodian doesn't need to scan relation files,
ERROR-ing when something goes wrong when cleaning up temporary files,
executing requested tasks immediately in single-user mode, and more.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
@ 2022-08-11 23:09 ` Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Nathan Bossart @ 2022-08-11 23:09 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Simon Riggs <[email protected]>; Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Wed, Jul 06, 2022 at 09:51:10AM -0700, Nathan Bossart wrote:
> Here's a new revision where I've attempted to address all the feedback I've
> received thus far. Notably, the custodian now uses a queue for registering
> tasks and determining which tasks to execute. Other changes include
> splitting the temporary file functions apart to avoid consecutive boolean
> flags, using a timestamp instead of an integer for the staging name for
> temporary directories, moving temporary directories to a dedicated
> directory so that the custodian doesn't need to scan relation files,
> ERROR-ing when something goes wrong when cleaning up temporary files,
> executing requested tasks immediately in single-user mode, and more.
Here is a rebased patch set for cfbot. There are no other differences
between v7 and v8.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
@ 2022-08-24 16:46 ` Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Nathan Bossart @ 2022-08-24 16:46 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Simon Riggs <[email protected]>; Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Thu, Aug 11, 2022 at 04:09:21PM -0700, Nathan Bossart wrote:
> Here is a rebased patch set for cfbot. There are no other differences
> between v7 and v8.
Another rebase for cfbot.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
@ 2022-09-02 22:07 ` Nathan Bossart <[email protected]>
2022-09-23 17:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Nathan Bossart @ 2022-09-02 22:07 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Simon Riggs <[email protected]>; Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Wed, Aug 24, 2022 at 09:46:24AM -0700, Nathan Bossart wrote:
> Another rebase for cfbot.
And another.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
@ 2022-09-23 17:41 ` Nathan Bossart <[email protected]>
2022-11-06 22:38 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Nathan Bossart @ 2022-09-23 17:41 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Simon Riggs <[email protected]>; Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Fri, Sep 02, 2022 at 03:07:44PM -0700, Nathan Bossart wrote:
> And another.
v11 adds support for building with meson.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-23 17:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
@ 2022-11-06 22:38 ` Nathan Bossart <[email protected]>
2022-11-24 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Nathan Bossart @ 2022-11-06 22:38 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Simon Riggs <[email protected]>; Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Fri, Sep 23, 2022 at 10:41:54AM -0700, Nathan Bossart wrote:
> v11 adds support for building with meson.
rebased
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-23 17:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-06 22:38 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
@ 2022-11-24 00:19 ` Nathan Bossart <[email protected]>
2022-11-24 17:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Nathan Bossart @ 2022-11-24 00:19 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Simon Riggs <[email protected]>; Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Sun, Nov 06, 2022 at 02:38:42PM -0800, Nathan Bossart wrote:
> rebased
another rebase for cfbot
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-23 17:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-06 22:38 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
@ 2022-11-24 17:31 ` Simon Riggs <[email protected]>
2022-11-27 23:34 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Simon Riggs @ 2022-11-24 17:31 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Andres Freund <[email protected]>; Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Thu, 24 Nov 2022 at 00:19, Nathan Bossart <[email protected]> wrote:
>
> On Sun, Nov 06, 2022 at 02:38:42PM -0800, Nathan Bossart wrote:
> > rebased
>
> another rebase for cfbot
0001 seems good to me
* I like that it sleeps forever until requested
* not sure I believe that everything it does can always be aborted out
of and shutdown - to achieve that you will need a
CHECK_FOR_INTERRUPTS() calls in the loops in patches 5 and 6 at least
* not sure why you want immediate execution of custodian tasks - I
feel supporting two modes will be a lot harder. For me, I would run
locally when !IsUnderPostmaster and also in an Assert build, so we can
test it works right - i.e. running in its own process is just a
production optimization for performance (which is the stated reason
for having this)
0005 seems good from what I know
* There is no check to see if it worked in any sane time
* It seems possible that "Old" might change meaning - will that make
it break/fail?
0006 seems good also
* same comments for 5
Rather than explicitly use DEBUG1 everywhere I would have an
#define CUSTODIAN_LOG_LEVEL LOG
so we can run with it in LOG mode and then set it to DEBUG1 with a one
line change in a later phase of Beta
I can't really comment with knowledge on sub-patches 0002 to 0004.
Perhaps you should aim to get 1, 5, 6 committed first and then return
to the others in a later CF/separate thread?
--
Simon Riggs http://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-23 17:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-06 22:38 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 17:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
@ 2022-11-27 23:34 ` Nathan Bossart <[email protected]>
2022-11-28 13:08 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Nathan Bossart @ 2022-11-27 23:34 UTC (permalink / raw)
To: Simon Riggs <[email protected]>; +Cc: Andres Freund <[email protected]>; Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
Thanks for taking a look!
On Thu, Nov 24, 2022 at 05:31:02PM +0000, Simon Riggs wrote:
> * not sure I believe that everything it does can always be aborted out
> of and shutdown - to achieve that you will need a
> CHECK_FOR_INTERRUPTS() calls in the loops in patches 5 and 6 at least
I did something like this earlier, but was advised to simply let the
functions finish as usual during shutdown [0]. I think this is what the
checkpointer process does today, anyway.
> * not sure why you want immediate execution of custodian tasks - I
> feel supporting two modes will be a lot harder. For me, I would run
> locally when !IsUnderPostmaster and also in an Assert build, so we can
> test it works right - i.e. running in its own process is just a
> production optimization for performance (which is the stated reason
> for having this)
I added this because 0004 involves requesting a task from the postmaster,
so checking for IsUnderPostmaster doesn't work. Those tasks would always
run immediately. However, we could use IsPostmasterEnvironment instead,
which would allow us to remove the "immediate" argument. I did it this way
in v14.
I'm not sure about running locally in Assert builds. It's true that would
help ensure there's test coverage for the task logic, but it would also
reduce coverage for the custodian logic. And in general, I'm worried about
having Assert builds use a different code path than production builds.
> 0005 seems good from what I know
> * There is no check to see if it worked in any sane time
What did you have in mind? Should the custodian begin emitting WARNINGs
after a while?
> * It seems possible that "Old" might change meaning - will that make
> it break/fail?
I don't believe so.
> Rather than explicitly use DEBUG1 everywhere I would have an
> #define CUSTODIAN_LOG_LEVEL LOG
> so we can run with it in LOG mode and then set it to DEBUG1 with a one
> line change in a later phase of Beta
I can create a separate patch for this, but I don't think I've ever seen
this sort of thing before. Is the idea just to help with debugging during
the development phase?
> I can't really comment with knowledge on sub-patches 0002 to 0004.
>
> Perhaps you should aim to get 1, 5, 6 committed first and then return
> to the others in a later CF/separate thread?
That seems like a good idea since those are all relatively self-contained.
I removed 0002-0004 in v14.
[0] https://postgr.es/m/20220217065938.x2esfdppzypegn5j%40alap3.anarazel.de
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-23 17:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-06 22:38 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 17:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-27 23:34 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
@ 2022-11-28 13:08 ` Simon Riggs <[email protected]>
2022-11-28 18:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Simon Riggs @ 2022-11-28 13:08 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Andres Freund <[email protected]>; Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Sun, 27 Nov 2022 at 23:34, Nathan Bossart <[email protected]> wrote:
>
> Thanks for taking a look!
>
> On Thu, Nov 24, 2022 at 05:31:02PM +0000, Simon Riggs wrote:
> > * not sure I believe that everything it does can always be aborted out
> > of and shutdown - to achieve that you will need a
> > CHECK_FOR_INTERRUPTS() calls in the loops in patches 5 and 6 at least
>
> I did something like this earlier, but was advised to simply let the
> functions finish as usual during shutdown [0]. I think this is what the
> checkpointer process does today, anyway.
If we say "The custodian is not an essential process and can shutdown
quickly when requested.", and yet we know its not true in all cases,
then that will lead to misunderstandings and bugs.
If we perform a restart and the custodian is performing extra work
that delays shutdown, then it also delays restart. Given the title of
the thread, we should be looking to improve that, or at least know it
occurred.
> > * not sure why you want immediate execution of custodian tasks - I
> > feel supporting two modes will be a lot harder. For me, I would run
> > locally when !IsUnderPostmaster and also in an Assert build, so we can
> > test it works right - i.e. running in its own process is just a
> > production optimization for performance (which is the stated reason
> > for having this)
>
> I added this because 0004 involves requesting a task from the postmaster,
> so checking for IsUnderPostmaster doesn't work. Those tasks would always
> run immediately. However, we could use IsPostmasterEnvironment instead,
> which would allow us to remove the "immediate" argument. I did it this way
> in v14.
Thanks
> > 0005 seems good from what I know
> > * There is no check to see if it worked in any sane time
>
> What did you have in mind? Should the custodian begin emitting WARNINGs
> after a while?
I think it might be useful if it logged anything that took an
"extended period", TBD.
Maybe that is already covered by startup process logging. Please tell
me that still works?
> > Rather than explicitly use DEBUG1 everywhere I would have an
> > #define CUSTODIAN_LOG_LEVEL LOG
> > so we can run with it in LOG mode and then set it to DEBUG1 with a one
> > line change in a later phase of Beta
>
> I can create a separate patch for this, but I don't think I've ever seen
> this sort of thing before.
Much of recovery is coded that way, for the same reason.
> Is the idea just to help with debugging during
> the development phase?
"Just", yes. Tests would be desirable also, under src/test/modules.
--
Simon Riggs http://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-23 17:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-06 22:38 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 17:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-27 23:34 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-28 13:08 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
@ 2022-11-28 18:31 ` Andres Freund <[email protected]>
2022-11-28 18:37 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Andres Freund @ 2022-11-28 18:31 UTC (permalink / raw)
To: Simon Riggs <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On 2022-11-28 13:08:57 +0000, Simon Riggs wrote:
> On Sun, 27 Nov 2022 at 23:34, Nathan Bossart <[email protected]> wrote:
> > > Rather than explicitly use DEBUG1 everywhere I would have an
> > > #define CUSTODIAN_LOG_LEVEL LOG
> > > so we can run with it in LOG mode and then set it to DEBUG1 with a one
> > > line change in a later phase of Beta
> >
> > I can create a separate patch for this, but I don't think I've ever seen
> > this sort of thing before.
>
> Much of recovery is coded that way, for the same reason.
I think that's not a good thing to copy without a lot more justification than
"some old code also does it that way". It's sometimes justified, but also
makes code harder to read (one doesn't know what it does without looking up
the #define, line length).
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-23 17:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-06 22:38 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 17:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-27 23:34 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-28 13:08 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-28 18:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
@ 2022-11-28 18:37 ` Robert Haas <[email protected]>
2022-11-28 23:40 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Robert Haas @ 2022-11-28 18:37 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Simon Riggs <[email protected]>; Nathan Bossart <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Mon, Nov 28, 2022 at 1:31 PM Andres Freund <[email protected]> wrote:
> On 2022-11-28 13:08:57 +0000, Simon Riggs wrote:
> > On Sun, 27 Nov 2022 at 23:34, Nathan Bossart <[email protected]> wrote:
> > > > Rather than explicitly use DEBUG1 everywhere I would have an
> > > > #define CUSTODIAN_LOG_LEVEL LOG
> > > > so we can run with it in LOG mode and then set it to DEBUG1 with a one
> > > > line change in a later phase of Beta
> > >
> > > I can create a separate patch for this, but I don't think I've ever seen
> > > this sort of thing before.
> >
> > Much of recovery is coded that way, for the same reason.
>
> I think that's not a good thing to copy without a lot more justification than
> "some old code also does it that way". It's sometimes justified, but also
> makes code harder to read (one doesn't know what it does without looking up
> the #define, line length).
Yeah. If people need some of the log messages at a higher level during
development, they can patch their own copies.
I think there might be some argument for having a facility that lets
you pick subsystems or even individual messages that you want to trace
and pump up the log level for just those call sites. But I don't know
exactly what that would look like, and I don't think inventing one-off
mechanisms for particular cases is a good idea.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-23 17:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-06 22:38 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 17:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-27 23:34 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-28 13:08 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-28 18:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-11-28 18:37 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
@ 2022-11-28 23:40 ` Nathan Bossart <[email protected]>
2022-11-29 12:02 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Nathan Bossart @ 2022-11-28 23:40 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Andres Freund <[email protected]>; Simon Riggs <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
Okay, here is a new patch set. 0004 adds logic to prevent custodian tasks
from delaying shutdown.
I haven't added any logging for long-running tasks yet. Tasks might
ordinarily take a while, so such logs wouldn't necessarily indicate
something is wrong. Perhaps we could add a GUC for the amount of time to
wait before logging. This feature would be off by default. Another option
could be to create a log_custodian GUC that causes tasks to be logged when
completed, similar to log_checkpoints. Thoughts?
On Mon, Nov 28, 2022 at 01:37:01PM -0500, Robert Haas wrote:
> On Mon, Nov 28, 2022 at 1:31 PM Andres Freund <[email protected]> wrote:
>> On 2022-11-28 13:08:57 +0000, Simon Riggs wrote:
>> > On Sun, 27 Nov 2022 at 23:34, Nathan Bossart <[email protected]> wrote:
>> > > > Rather than explicitly use DEBUG1 everywhere I would have an
>> > > > #define CUSTODIAN_LOG_LEVEL LOG
>> > > > so we can run with it in LOG mode and then set it to DEBUG1 with a one
>> > > > line change in a later phase of Beta
>> > >
>> > > I can create a separate patch for this, but I don't think I've ever seen
>> > > this sort of thing before.
>> >
>> > Much of recovery is coded that way, for the same reason.
>>
>> I think that's not a good thing to copy without a lot more justification than
>> "some old code also does it that way". It's sometimes justified, but also
>> makes code harder to read (one doesn't know what it does without looking up
>> the #define, line length).
>
> Yeah. If people need some of the log messages at a higher level during
> development, they can patch their own copies.
>
> I think there might be some argument for having a facility that lets
> you pick subsystems or even individual messages that you want to trace
> and pump up the log level for just those call sites. But I don't know
> exactly what that would look like, and I don't think inventing one-off
> mechanisms for particular cases is a good idea.
Given this discussion, I haven't made any changes to the logging in the new
patch set.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-23 17:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-06 22:38 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 17:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-27 23:34 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-28 13:08 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-28 18:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-11-28 18:37 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-11-28 23:40 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
@ 2022-11-29 12:02 ` Simon Riggs <[email protected]>
2022-11-30 03:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Simon Riggs @ 2022-11-29 12:02 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Mon, 28 Nov 2022 at 23:40, Nathan Bossart <[email protected]> wrote:
>
> Okay, here is a new patch set. 0004 adds logic to prevent custodian tasks
> from delaying shutdown.
That all seems good, thanks.
The last important point for me is tests, in src/test/modules
probably. It might be possible to reuse the final state of other
modules' tests to test cleanup, or at least integrate a custodian test
into each module.
--
Simon Riggs http://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-23 17:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-06 22:38 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 17:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-27 23:34 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-28 13:08 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-28 18:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-11-28 18:37 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-11-28 23:40 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-29 12:02 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
@ 2022-11-30 03:56 ` Nathan Bossart <[email protected]>
2022-11-30 05:18 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-30 06:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
0 siblings, 2 replies; 59+ messages in thread
From: Nathan Bossart @ 2022-11-30 03:56 UTC (permalink / raw)
To: Simon Riggs <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Tue, Nov 29, 2022 at 12:02:44PM +0000, Simon Riggs wrote:
> The last important point for me is tests, in src/test/modules
> probably. It might be possible to reuse the final state of other
> modules' tests to test cleanup, or at least integrate a custodian test
> into each module.
Of course. I found some existing tests for the test_decoding plugin that
appear to reliably generate the files we want the custodian to clean up, so
I added them there.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-23 17:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-06 22:38 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 17:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-27 23:34 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-28 13:08 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-28 18:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-11-28 18:37 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-11-28 23:40 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-29 12:02 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-30 03:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
@ 2022-11-30 05:18 ` Nathan Bossart <[email protected]>
2022-11-30 11:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
1 sibling, 1 reply; 59+ messages in thread
From: Nathan Bossart @ 2022-11-30 05:18 UTC (permalink / raw)
To: Simon Riggs <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Tue, Nov 29, 2022 at 07:56:53PM -0800, Nathan Bossart wrote:
> On Tue, Nov 29, 2022 at 12:02:44PM +0000, Simon Riggs wrote:
>> The last important point for me is tests, in src/test/modules
>> probably. It might be possible to reuse the final state of other
>> modules' tests to test cleanup, or at least integrate a custodian test
>> into each module.
>
> Of course. I found some existing tests for the test_decoding plugin that
> appear to reliably generate the files we want the custodian to clean up, so
> I added them there.
cfbot is not happy with v16. AFAICT this is just due to poor placement, so
here is another attempt with the tests moved to a new location. Apologies
for the noise.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-23 17:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-06 22:38 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 17:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-27 23:34 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-28 13:08 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-28 18:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-11-28 18:37 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-11-28 23:40 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-29 12:02 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-30 03:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-30 05:18 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
@ 2022-11-30 11:22 ` Bharath Rupireddy <[email protected]>
2022-11-30 11:57 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Bharath Rupireddy @ 2022-11-30 11:22 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Simon Riggs <[email protected]>; Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Wed, Nov 30, 2022 at 10:48 AM Nathan Bossart
<[email protected]> wrote:
>
>
> cfbot is not happy with v16. AFAICT this is just due to poor placement, so
> here is another attempt with the tests moved to a new location. Apologies
> for the noise.
Thanks for the patches. I spent some time on reviewing v17 patch set
and here are my comments:
0001:
1. I think the custodian process needs documentation - it needs a
definition in glossary.sgml and perhaps a dedicated page describing
what tasks it takes care of.
2.
+ LWLockReleaseAll();
+ ConditionVariableCancelSleep();
+ AbortBufferIO();
+ UnlockBuffers();
+ ReleaseAuxProcessResources(false);
+ AtEOXact_Buffers(false);
+ AtEOXact_SMgr();
+ AtEOXact_Files(false);
+ AtEOXact_HashTables(false);
Do we need all of these in the exit path? Isn't the stuff that
ShutdownAuxiliaryProcess() does enough for the custodian process?
AFAICS, the custodian process uses LWLocks (which the
ShutdownAuxiliaryProcess() takes care of) and it doesn't access shared
buffers and so on.
Having said that, I'm fine to keep them for future use and all of
those cleanup functions exit if nothing related occurs.
3.
+ * Advertise out latch that backends can use to wake us up while we're
Typo - %s/out/our
4. Is it a good idea to add log messages in the DoCustodianTasks()
loop? Maybe at a debug level? The log message can say the current task
the custodian is processing. And/Or setting the custodian's status on
the ps display is also a good idea IMO.
0002 and 0003:
1.
+CHECKPOINT;
+DO $$
I think we need to ensure that there are some snapshot files before
the checkpoint. Otherwise, it may happen that the above test case
exits without the custodian process doing anything.
2. I think the best way to test the custodian process code is by
adding a TAP test module to see actually the custodian process kicks
in. Perhaps, add elog(DEBUGX,...) messages to various custodian
process functions and see if we see the logs in server logs.
0004:
I think the 0004 patch can be merged into 0001, 0002 and 0003 patches.
Otherwise the patch LGTM.
Few thoughts:
1. I think we can trivially extend the custodian process to remove any
future WAL files on the old timeline, something like the attached
0001-Move-removal-of-future-WAL-files-on-the-old-timeline.text file).
While this offloads the recovery a bit, the server may archive such
WAL files before the custodian removes them. We can do a bit more to
stop the server from archiving such WAL files, but that needs more
coding. I don't think we need to do all that now, perhaps, we can give
it a try once the basic custodian stuff gets in.
2. Moving RemovePgTempFiles() to the custodian can bring up the server
soon. The idea is that the postmaster just renames the temp
directories and informs the custodian so that it can go delete such
temp files and directories. I have personally seen cases where the
server spent a good amount of time cleaning up temp files. We can park
it for later.
3. Moving RemoveOldXlogFiles() to the custodian can make checkpoints faster.
4. PreallocXlogFiles() - if we ever have plans to make pre-allocation
more aggressive (pre-allocate more than 1 WAL file), perhaps letting
custodian do that is a good idea. Again, too many tasks for a single
process.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-23 17:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-06 22:38 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 17:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-27 23:34 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-28 13:08 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-28 18:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-11-28 18:37 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-11-28 23:40 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-29 12:02 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-30 03:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-30 05:18 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-30 11:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
@ 2022-11-30 11:57 ` Bharath Rupireddy <[email protected]>
2022-12-01 21:40 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Bharath Rupireddy @ 2022-11-30 11:57 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Simon Riggs <[email protected]>; Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Wed, Nov 30, 2022 at 4:52 PM Bharath Rupireddy
<[email protected]> wrote:
>
> On Wed, Nov 30, 2022 at 10:48 AM Nathan Bossart
> <[email protected]> wrote:
> >
> >
> > cfbot is not happy with v16. AFAICT this is just due to poor placement, so
> > here is another attempt with the tests moved to a new location. Apologies
> > for the noise.
>
> Thanks for the patches. I spent some time on reviewing v17 patch set
> and here are my comments:
>
> 0001:
> 1. I think the custodian process needs documentation - it needs a
> definition in glossary.sgml and perhaps a dedicated page describing
> what tasks it takes care of.
>
> 2.
> + LWLockReleaseAll();
> + ConditionVariableCancelSleep();
> + AbortBufferIO();
> + UnlockBuffers();
> + ReleaseAuxProcessResources(false);
> + AtEOXact_Buffers(false);
> + AtEOXact_SMgr();
> + AtEOXact_Files(false);
> + AtEOXact_HashTables(false);
> Do we need all of these in the exit path? Isn't the stuff that
> ShutdownAuxiliaryProcess() does enough for the custodian process?
> AFAICS, the custodian process uses LWLocks (which the
> ShutdownAuxiliaryProcess() takes care of) and it doesn't access shared
> buffers and so on.
> Having said that, I'm fine to keep them for future use and all of
> those cleanup functions exit if nothing related occurs.
>
> 3.
> + * Advertise out latch that backends can use to wake us up while we're
> Typo - %s/out/our
>
> 4. Is it a good idea to add log messages in the DoCustodianTasks()
> loop? Maybe at a debug level? The log message can say the current task
> the custodian is processing. And/Or setting the custodian's status on
> the ps display is also a good idea IMO.
>
> 0002 and 0003:
> 1.
> +CHECKPOINT;
> +DO $$
> I think we need to ensure that there are some snapshot files before
> the checkpoint. Otherwise, it may happen that the above test case
> exits without the custodian process doing anything.
>
> 2. I think the best way to test the custodian process code is by
> adding a TAP test module to see actually the custodian process kicks
> in. Perhaps, add elog(DEBUGX,...) messages to various custodian
> process functions and see if we see the logs in server logs.
>
> 0004:
> I think the 0004 patch can be merged into 0001, 0002 and 0003 patches.
> Otherwise the patch LGTM.
>
> Few thoughts:
> 1. I think we can trivially extend the custodian process to remove any
> future WAL files on the old timeline, something like the attached
> 0001-Move-removal-of-future-WAL-files-on-the-old-timeline.text file).
> While this offloads the recovery a bit, the server may archive such
> WAL files before the custodian removes them. We can do a bit more to
> stop the server from archiving such WAL files, but that needs more
> coding. I don't think we need to do all that now, perhaps, we can give
> it a try once the basic custodian stuff gets in.
> 2. Moving RemovePgTempFiles() to the custodian can bring up the server
> soon. The idea is that the postmaster just renames the temp
> directories and informs the custodian so that it can go delete such
> temp files and directories. I have personally seen cases where the
> server spent a good amount of time cleaning up temp files. We can park
> it for later.
> 3. Moving RemoveOldXlogFiles() to the custodian can make checkpoints faster.
> 4. PreallocXlogFiles() - if we ever have plans to make pre-allocation
> more aggressive (pre-allocate more than 1 WAL file), perhaps letting
> custodian do that is a good idea. Again, too many tasks for a single
> process.
Another comment:
IIUC, there's no custodian_delay GUC as we want to avoid unnecessary
wakeups for power savings (being discussed in the other thread).
However, can it happen that the custodian missed to capture SetLatch
wakeups by other backends? In other words, can the custodian process
be sleeping when there's work to do?
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-23 17:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-06 22:38 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 17:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-27 23:34 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-28 13:08 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-28 18:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-11-28 18:37 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-11-28 23:40 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-29 12:02 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-30 03:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-30 05:18 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-30 11:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2022-11-30 11:57 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
@ 2022-12-01 21:40 ` Nathan Bossart <[email protected]>
2022-12-02 06:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Nathan Bossart @ 2022-12-01 21:40 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Simon Riggs <[email protected]>; Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Wed, Nov 30, 2022 at 05:27:10PM +0530, Bharath Rupireddy wrote:
> On Wed, Nov 30, 2022 at 4:52 PM Bharath Rupireddy
> <[email protected]> wrote:
>> Thanks for the patches. I spent some time on reviewing v17 patch set
>> and here are my comments:
Thanks for reviewing!
>> 0001:
>> 1. I think the custodian process needs documentation - it needs a
>> definition in glossary.sgml and perhaps a dedicated page describing
>> what tasks it takes care of.
Good catch. I added this in v18. I stopped short of adding a dedicated
page to describe the tasks because 1) there are no parameters for the
custodian and 2) AFAICT none of its tasks are described in the docs today.
>> 2.
>> + LWLockReleaseAll();
>> + ConditionVariableCancelSleep();
>> + AbortBufferIO();
>> + UnlockBuffers();
>> + ReleaseAuxProcessResources(false);
>> + AtEOXact_Buffers(false);
>> + AtEOXact_SMgr();
>> + AtEOXact_Files(false);
>> + AtEOXact_HashTables(false);
>> Do we need all of these in the exit path? Isn't the stuff that
>> ShutdownAuxiliaryProcess() does enough for the custodian process?
>> AFAICS, the custodian process uses LWLocks (which the
>> ShutdownAuxiliaryProcess() takes care of) and it doesn't access shared
>> buffers and so on.
>> Having said that, I'm fine to keep them for future use and all of
>> those cleanup functions exit if nothing related occurs.
Yeah, I don't think we need a few of these. In v18, I've kept the
following:
* LWLockReleaseAll()
* ConditionVariableCancelSleep()
* ReleaseAuxProcessResources(false)
* AtEOXact_Files(false)
>> 3.
>> + * Advertise out latch that backends can use to wake us up while we're
>> Typo - %s/out/our
fixed
>> 4. Is it a good idea to add log messages in the DoCustodianTasks()
>> loop? Maybe at a debug level? The log message can say the current task
>> the custodian is processing. And/Or setting the custodian's status on
>> the ps display is also a good idea IMO.
I'd like to pick these up in a new thread if/when this initial patch set is
committed. The tasks already do some logging, and the checkpointer process
doesn't update the ps display for these tasks today.
>> 0002 and 0003:
>> 1.
>> +CHECKPOINT;
>> +DO $$
>> I think we need to ensure that there are some snapshot files before
>> the checkpoint. Otherwise, it may happen that the above test case
>> exits without the custodian process doing anything.
>>
>> 2. I think the best way to test the custodian process code is by
>> adding a TAP test module to see actually the custodian process kicks
>> in. Perhaps, add elog(DEBUGX,...) messages to various custodian
>> process functions and see if we see the logs in server logs.
The test appears to reliably create snapshot and mapping files, so if the
directories are empty at some point after the checkpoint at the end, we can
be reasonably certain the custodian took action. I didn't add explicit
checks that there are files in the directories before the checkpoint
because a concurrent checkpoint could make such checks unreliable.
>> 0004:
>> I think the 0004 patch can be merged into 0001, 0002 and 0003 patches.
>> Otherwise the patch LGTM.
I'm keeping this one separate because I've received conflicting feedback
about the idea.
>> 1. I think we can trivially extend the custodian process to remove any
>> future WAL files on the old timeline, something like the attached
>> 0001-Move-removal-of-future-WAL-files-on-the-old-timeline.text file).
>> While this offloads the recovery a bit, the server may archive such
>> WAL files before the custodian removes them. We can do a bit more to
>> stop the server from archiving such WAL files, but that needs more
>> coding. I don't think we need to do all that now, perhaps, we can give
>> it a try once the basic custodian stuff gets in.
>> 2. Moving RemovePgTempFiles() to the custodian can bring up the server
>> soon. The idea is that the postmaster just renames the temp
>> directories and informs the custodian so that it can go delete such
>> temp files and directories. I have personally seen cases where the
>> server spent a good amount of time cleaning up temp files. We can park
>> it for later.
>> 3. Moving RemoveOldXlogFiles() to the custodian can make checkpoints faster.
>> 4. PreallocXlogFiles() - if we ever have plans to make pre-allocation
>> more aggressive (pre-allocate more than 1 WAL file), perhaps letting
>> custodian do that is a good idea. Again, too many tasks for a single
>> process.
I definitely want to do #2. І have some patches for that upthread, but I
removed them for now based on Simon's feedback. I intend to pick that up
in a new thread. I haven't thought too much about the others yet.
> Another comment:
> IIUC, there's no custodian_delay GUC as we want to avoid unnecessary
> wakeups for power savings (being discussed in the other thread).
> However, can it happen that the custodian missed to capture SetLatch
> wakeups by other backends? In other words, can the custodian process
> be sleeping when there's work to do?
I'm not aware of any way this could happen, but if there is one, I think we
should treat it as a bug instead of relying on the custodian process to
periodically wake up and check for work to do.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-23 17:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-06 22:38 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 17:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-27 23:34 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-28 13:08 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-28 18:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-11-28 18:37 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-11-28 23:40 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-29 12:02 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-30 03:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-30 05:18 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-30 11:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2022-11-30 11:57 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2022-12-01 21:40 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
@ 2022-12-02 06:41 ` Bharath Rupireddy <[email protected]>
2022-12-02 19:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Bharath Rupireddy @ 2022-12-02 06:41 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Simon Riggs <[email protected]>; Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Fri, Dec 2, 2022 at 3:10 AM Nathan Bossart <[email protected]> wrote:
>
> >> 4. Is it a good idea to add log messages in the DoCustodianTasks()
> >> loop? Maybe at a debug level? The log message can say the current task
> >> the custodian is processing. And/Or setting the custodian's status on
> >> the ps display is also a good idea IMO.
>
> I'd like to pick these up in a new thread if/when this initial patch set is
> committed. The tasks already do some logging, and the checkpointer process
> doesn't update the ps display for these tasks today.
It'll be good to have some kind of dedicated monitoring for the
custodian process as it can do a "good" amount of work at times and
users will have a way to know what it currently is doing - it can be
logs at debug level, progress reporting via
ereport_startup_progress()-sort of mechanism, ps display,
pg_stat_custodian or a special function that tells some details, or
some other. In any case, I agree to park this for later.
> >> 0002 and 0003:
> >> 1.
> >> +CHECKPOINT;
> >> +DO $$
> >> I think we need to ensure that there are some snapshot files before
> >> the checkpoint. Otherwise, it may happen that the above test case
> >> exits without the custodian process doing anything.
> >>
> >> 2. I think the best way to test the custodian process code is by
> >> adding a TAP test module to see actually the custodian process kicks
> >> in. Perhaps, add elog(DEBUGX,...) messages to various custodian
> >> process functions and see if we see the logs in server logs.
>
> The test appears to reliably create snapshot and mapping files, so if the
> directories are empty at some point after the checkpoint at the end, we can
> be reasonably certain the custodian took action. I didn't add explicit
> checks that there are files in the directories before the checkpoint
> because a concurrent checkpoint could make such checks unreliable.
I think you're right. I added sqls to see if the snapshot and mapping
files count > 0, see [1] and the cirrus-ci members are happy too -
https://github.com/BRupireddy/postgres/tree/custodian_review_2. I
think we can consider adding these count > 0 checks to tests.
> >> 0004:
> >> I think the 0004 patch can be merged into 0001, 0002 and 0003 patches.
> >> Otherwise the patch LGTM.
>
> I'm keeping this one separate because I've received conflicting feedback
> about the idea.
If we classify custodian as a process doing non-critical tasks that
have nothing to do with regular server functioning, then processing
ShutdownRequestPending looks okay. However, delaying these
non-critical tasks such as file removals which reclaims disk space
might impact the server overall especially when it's reaching 100%
disk usage and we want the custodian to do its job fully before we
shutdown the server.
If we delay processing shutdown requests, that can impact the server
overall (might delay restarts, failovers etc.), because at times there
can be a lot of tasks with a good amount of work pending in the
custodian's task queue.
Having said above, I'm okay to process ShutdownRequestPending as early
as possible, however, should we also add CHECK_FOR_INTERRUPTS()
alongside ShutdownRequestPending?
Also, I think it's enough to just have ShutdownRequestPending check in
DoCustodianTasks(void)'s main loop and we can let
RemoveOldSerializedSnapshots() and RemoveOldLogicalRewriteMappings()
do their jobs to the fullest as they do today.
While thinking about this, one thing that really struck me is what
happens if we let the custodian exit, say after processing
ShutdownRequestPending immediately or after a restart, leaving other
queued tasks? The custodian will never get to work on those tasks
unless the requestors (say checkpoint or some other process) requests
it to do so after restart. Maybe, we don't need to worry about it.
Maybe we need to worry about it. Maybe it's an overkill to save the
custodian's task state to disk so that it can come up and do the
leftover tasks upon restart.
> > Another comment:
> > IIUC, there's no custodian_delay GUC as we want to avoid unnecessary
> > wakeups for power savings (being discussed in the other thread).
> > However, can it happen that the custodian missed to capture SetLatch
> > wakeups by other backends? In other words, can the custodian process
> > be sleeping when there's work to do?
>
> I'm not aware of any way this could happen, but if there is one, I think we
> should treat it as a bug instead of relying on the custodian process to
> periodically wake up and check for work to do.
One possible scenario is that the requestor adds its task details to
the queue and sets the latch, the custodian can miss this SetLatch()
when it's in the midst of processing a task. However, it guarantees
the requester that it'll process the added task after it completes the
current task. And, I don't know the other reasons when the custodian
can miss SetLatch().
[1]
diff --git a/contrib/test_decoding/expected/rewrite.out
b/contrib/test_decoding/expected/rewrite.out
index 214a514a0a..0029e48852 100644
--- a/contrib/test_decoding/expected/rewrite.out
+++ b/contrib/test_decoding/expected/rewrite.out
@@ -163,6 +163,20 @@ DROP FUNCTION iamalongfunction();
DROP FUNCTION exec(text);
DROP ROLE regress_justforcomments;
-- make sure custodian cleans up files
+-- make sure snapshot files exist for custodian to clean up
+SELECT count(*) > 0 FROM pg_ls_logicalsnapdir();
+ ?column?
+----------
+ t
+(1 row)
+
+-- make sure rewrite mapping files exist for custodian to clean up
+SELECT count(*) > 0 FROM pg_ls_logicalmapdir();
+ ?column?
+----------
+ t
+(1 row)
+
CHECKPOINT;
DO $$
DECLARE
diff --git a/contrib/test_decoding/sql/rewrite.sql
b/contrib/test_decoding/sql/rewrite.sql
index d66f70f837..c076809f37 100644
--- a/contrib/test_decoding/sql/rewrite.sql
+++ b/contrib/test_decoding/sql/rewrite.sql
@@ -107,6 +107,13 @@ DROP FUNCTION exec(text);
DROP ROLE regress_justforcomments;
-- make sure custodian cleans up files
+
+-- make sure snapshot files exist for custodian to clean up
+SELECT count(*) > 0 FROM pg_ls_logicalsnapdir();
+
+-- make sure rewrite mapping files exist for custodian to clean up
+SELECT count(*) > 0 FROM pg_ls_logicalmapdir();
+
CHECKPOINT;
DO $$
DECLARE
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-23 17:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-06 22:38 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 17:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-27 23:34 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-28 13:08 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-28 18:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-11-28 18:37 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-11-28 23:40 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-29 12:02 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-30 03:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-30 05:18 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-30 11:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2022-11-30 11:57 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2022-12-01 21:40 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-12-02 06:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
@ 2022-12-02 19:15 ` Nathan Bossart <[email protected]>
2022-12-06 07:28 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
0 siblings, 1 reply; 59+ messages in thread
From: Nathan Bossart @ 2022-12-02 19:15 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Simon Riggs <[email protected]>; Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Fri, Dec 02, 2022 at 12:11:35PM +0530, Bharath Rupireddy wrote:
> On Fri, Dec 2, 2022 at 3:10 AM Nathan Bossart <[email protected]> wrote:
>> The test appears to reliably create snapshot and mapping files, so if the
>> directories are empty at some point after the checkpoint at the end, we can
>> be reasonably certain the custodian took action. I didn't add explicit
>> checks that there are files in the directories before the checkpoint
>> because a concurrent checkpoint could make such checks unreliable.
>
> I think you're right. I added sqls to see if the snapshot and mapping
> files count > 0, see [1] and the cirrus-ci members are happy too -
> https://github.com/BRupireddy/postgres/tree/custodian_review_2. I
> think we can consider adding these count > 0 checks to tests.
My worry about adding "count > 0" checks is that a concurrent checkpoint
could make them unreliable. In other words, those checks might ordinarily
work, but if an automatic checkpoint causes the files be cleaned up just
beforehand, they will fail.
> Having said above, I'm okay to process ShutdownRequestPending as early
> as possible, however, should we also add CHECK_FOR_INTERRUPTS()
> alongside ShutdownRequestPending?
I'm not seeing a need for CHECK_FOR_INTERRUPTS. Do you see one?
> While thinking about this, one thing that really struck me is what
> happens if we let the custodian exit, say after processing
> ShutdownRequestPending immediately or after a restart, leaving other
> queued tasks? The custodian will never get to work on those tasks
> unless the requestors (say checkpoint or some other process) requests
> it to do so after restart. Maybe, we don't need to worry about it.
> Maybe we need to worry about it. Maybe it's an overkill to save the
> custodian's task state to disk so that it can come up and do the
> leftover tasks upon restart.
Yes, tasks will need to be retried when the server starts again. The ones
in this patch set should be requested again during the next checkpoint.
Temporary file cleanup would always be requested during server start, so
that should be handled as well. Even today, the server might abruptly shut
down while executing these tasks, and we don't have any special handling
for that.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-23 17:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-06 22:38 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 17:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-27 23:34 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-28 13:08 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-28 18:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-11-28 18:37 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-11-28 23:40 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-29 12:02 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-30 03:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-30 05:18 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-30 11:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2022-11-30 11:57 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2022-12-01 21:40 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-12-02 06:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2022-12-02 19:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
@ 2022-12-06 07:28 ` Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 59+ messages in thread
From: Bharath Rupireddy @ 2022-12-06 07:28 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Simon Riggs <[email protected]>; Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Sat, Dec 3, 2022 at 12:45 AM Nathan Bossart <[email protected]> wrote:
>
> On Fri, Dec 02, 2022 at 12:11:35PM +0530, Bharath Rupireddy wrote:
> > On Fri, Dec 2, 2022 at 3:10 AM Nathan Bossart <[email protected]> wrote:
> >> The test appears to reliably create snapshot and mapping files, so if the
> >> directories are empty at some point after the checkpoint at the end, we can
> >> be reasonably certain the custodian took action. I didn't add explicit
> >> checks that there are files in the directories before the checkpoint
> >> because a concurrent checkpoint could make such checks unreliable.
> >
> > I think you're right. I added sqls to see if the snapshot and mapping
> > files count > 0, see [1] and the cirrus-ci members are happy too -
> > https://github.com/BRupireddy/postgres/tree/custodian_review_2. I
> > think we can consider adding these count > 0 checks to tests.
>
> My worry about adding "count > 0" checks is that a concurrent checkpoint
> could make them unreliable. In other words, those checks might ordinarily
> work, but if an automatic checkpoint causes the files be cleaned up just
> beforehand, they will fail.
Hm. It would have been better with a TAP test module for testing the
custodian code reliably. Anyway, that mustn't stop the patch getting
in. If required, we can park the TAP test module for later - IMO.
Others may have different thoughts here.
> > Having said above, I'm okay to process ShutdownRequestPending as early
> > as possible, however, should we also add CHECK_FOR_INTERRUPTS()
> > alongside ShutdownRequestPending?
>
> I'm not seeing a need for CHECK_FOR_INTERRUPTS. Do you see one?
Since the custodian has SignalHandlerForShutdownRequest as SIGINT and
SIGTERM handlers, unlike StatementCancelHandler and die respectively,
no need of CFI I guess. And also none of the CFI signal handler flags
applies to the custodian.
> > While thinking about this, one thing that really struck me is what
> > happens if we let the custodian exit, say after processing
> > ShutdownRequestPending immediately or after a restart, leaving other
> > queued tasks? The custodian will never get to work on those tasks
> > unless the requestors (say checkpoint or some other process) requests
> > it to do so after restart. Maybe, we don't need to worry about it.
> > Maybe we need to worry about it. Maybe it's an overkill to save the
> > custodian's task state to disk so that it can come up and do the
> > leftover tasks upon restart.
>
> Yes, tasks will need to be retried when the server starts again. The ones
> in this patch set should be requested again during the next checkpoint.
> Temporary file cleanup would always be requested during server start, so
> that should be handled as well. Even today, the server might abruptly shut
> down while executing these tasks, and we don't have any special handling
> for that.
Right.
The v18 patch set posted upthread
https://www.postgresql.org/message-id/20221201214026.GA1799688%40nathanxps13
looks good to me. I see the CF entry is marked RfC -
https://commitfest.postgresql.org/41/3448/.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 59+ messages in thread
* Re: O(n) tasks cause lengthy startups and checkpoints
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-23 17:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-06 22:38 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 17:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-27 23:34 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-28 13:08 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-28 18:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-11-28 18:37 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-11-28 23:40 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-29 12:02 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-30 03:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
@ 2022-11-30 06:22 ` Simon Riggs <[email protected]>
1 sibling, 0 replies; 59+ messages in thread
From: Simon Riggs @ 2022-11-30 06:22 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; Maxim Orlov <[email protected]>; Amul Sul <[email protected]>; Bruce Momjian <[email protected]>; pgsql-hackers
On Wed, 30 Nov 2022 at 03:56, Nathan Bossart <[email protected]> wrote:
>
> On Tue, Nov 29, 2022 at 12:02:44PM +0000, Simon Riggs wrote:
> > The last important point for me is tests, in src/test/modules
> > probably. It might be possible to reuse the final state of other
> > modules' tests to test cleanup, or at least integrate a custodian test
> > into each module.
>
> Of course. I found some existing tests for the test_decoding plugin that
> appear to reliably generate the files we want the custodian to clean up, so
> I added them there.
Thanks for adding the tests; I can see they run clean.
The only minor thing I would personally add is a note in each piece of
code to explain where the tests are for each one and/or something in
the main custodian file that says tests exist within src/test/module.
Otherwise, ready for committer.
--
Simon Riggs http://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 59+ messages in thread
end of thread, other threads:[~2022-12-06 07:28 UTC | newest]
Thread overview: 59+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-07-05 16:16 [PATCH 02/33] Add some missing newlines after function definitions Justin Pryzby <[email protected]>
2021-07-05 16:16 [PATCH 08/10] Add some missing newlines after function definitions Justin Pryzby <[email protected]>
2021-07-05 16:16 [PATCH v2 05/12] Add some missing newlines after function definitions Justin Pryzby <[email protected]>
2021-07-05 16:16 [PATCH 16/16] Add some missing newlines after function definitions Justin Pryzby <[email protected]>
2021-12-01 20:24 O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-01 21:35 ` Re: O(n) tasks cause lengthy startups and checkpoints SATYANARAYANA NARLAPURAM <[email protected]>
2021-12-01 22:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2021-12-02 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-02 02:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Euler Taveira <[email protected]>
2021-12-02 21:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-02 02:47 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-02 21:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-03 13:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-03 18:20 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-06 11:43 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2021-12-06 19:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-10 19:03 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-13 13:53 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2021-12-13 17:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Justin Pryzby <[email protected]>
2021-12-13 18:30 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-13 18:21 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-13 20:36 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2021-12-13 23:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-14 16:59 ` Re: O(n) tasks cause lengthy startups and checkpoints Bruce Momjian <[email protected]>
2021-12-14 20:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-14 20:13 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2021-12-14 20:23 ` Re: O(n) tasks cause lengthy startups and checkpoints Bossart, Nathan <[email protected]>
2022-06-23 11:58 Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 13:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-06-23 17:13 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-06-23 17:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-06-24 10:45 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-07-02 22:05 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-02 22:54 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-03 17:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-07-03 17:27 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-07-06 16:51 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-11 23:09 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-08-24 16:46 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-02 22:07 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-09-23 17:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-06 22:38 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 00:19 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-24 17:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-27 23:34 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-28 13:08 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-28 18:31 ` Re: O(n) tasks cause lengthy startups and checkpoints Andres Freund <[email protected]>
2022-11-28 18:37 ` Re: O(n) tasks cause lengthy startups and checkpoints Robert Haas <[email protected]>
2022-11-28 23:40 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-29 12:02 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
2022-11-30 03:56 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-30 05:18 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-11-30 11:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2022-11-30 11:57 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2022-12-01 21:40 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-12-02 06:41 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2022-12-02 19:15 ` Re: O(n) tasks cause lengthy startups and checkpoints Nathan Bossart <[email protected]>
2022-12-06 07:28 ` Re: O(n) tasks cause lengthy startups and checkpoints Bharath Rupireddy <[email protected]>
2022-11-30 06:22 ` Re: O(n) tasks cause lengthy startups and checkpoints Simon Riggs <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox