agora inbox for [email protected]
help / color / mirror / Atom feed[PATCH v45 7/7] Exclude pg_stat directory from base backup
11+ messages / 7 participants
[nested] [flat]
* [PATCH v45 7/7] Exclude pg_stat directory from base backup
@ 2020-09-29 14:19 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 11+ messages in thread
From: Kyotaro Horiguchi @ 2020-09-29 14:19 UTC (permalink / raw)
basebackup sends the content of pg_stat directory, which is doomed to
be removed at startup from the backup. Now that pg_stat_statements
saves a temporary file into the directory, let exclude pg_stat
directory from a base backup.
---
src/backend/replication/basebackup.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index d2c3064678..25677c5c6e 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -149,6 +149,13 @@ struct exclude_list_item
*/
static const char *const excludeDirContents[] =
{
+ /*
+ * Skip statistics files. PGSTAT_STAT_PERMANENT_DIRECTORY must be skipped
+ * because the files in the directory will be removed at startup from the
+ * backup.
+ */
+ PGSTAT_STAT_PERMANENT_DIRECTORY,
+
/*
* It is generally not useful to backup the contents of this directory
* even if the intention is to restore to another primary. See backup.sgml
--
2.27.0
----Next_Part(Fri_Jan__8_10_24_34_2021_185)----
^ permalink raw reply [nested|flat] 11+ messages in thread
* [PATCH v2 1/1] add --no-sync to pg_upgrade's calls to pg_dump[all]
@ 2024-05-03 15:35 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 11+ messages in thread
From: Nathan Bossart @ 2024-05-03 15:35 UTC (permalink / raw)
---
src/bin/pg_upgrade/dump.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/bin/pg_upgrade/dump.c b/src/bin/pg_upgrade/dump.c
index 29fb45b928..8345f55be8 100644
--- a/src/bin/pg_upgrade/dump.c
+++ b/src/bin/pg_upgrade/dump.c
@@ -22,7 +22,7 @@ generate_old_dump(void)
/* run new pg_dumpall binary for globals */
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
"\"%s/pg_dumpall\" %s --globals-only --quote-all-identifiers "
- "--binary-upgrade %s -f \"%s/%s\"",
+ "--binary-upgrade %s --no-sync -f \"%s/%s\"",
new_cluster.bindir, cluster_conn_opts(&old_cluster),
log_opts.verbose ? "--verbose" : "",
log_opts.dumpdir,
@@ -53,7 +53,7 @@ generate_old_dump(void)
parallel_exec_prog(log_file_name, NULL,
"\"%s/pg_dump\" %s --schema-only --quote-all-identifiers "
- "--binary-upgrade --format=custom %s --file=\"%s/%s\" %s",
+ "--binary-upgrade --format=custom %s --no-sync --file=\"%s/%s\" %s",
new_cluster.bindir, cluster_conn_opts(&old_cluster),
log_opts.verbose ? "--verbose" : "",
log_opts.dumpdir,
--
2.25.1
--TB36FDmn/VVEgNH/--
^ permalink raw reply [nested|flat] 11+ messages in thread
* [PATCH v1 1/1] add --no-sync to pg_upgrade's calls to pg_dump[all]
@ 2024-05-03 15:35 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 11+ messages in thread
From: Nathan Bossart @ 2024-05-03 15:35 UTC (permalink / raw)
---
src/bin/pg_upgrade/dump.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/src/bin/pg_upgrade/dump.c b/src/bin/pg_upgrade/dump.c
index 29fb45b928..f052a4985c 100644
--- a/src/bin/pg_upgrade/dump.c
+++ b/src/bin/pg_upgrade/dump.c
@@ -22,9 +22,10 @@ generate_old_dump(void)
/* run new pg_dumpall binary for globals */
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
"\"%s/pg_dumpall\" %s --globals-only --quote-all-identifiers "
- "--binary-upgrade %s -f \"%s/%s\"",
+ "--binary-upgrade %s %s -f \"%s/%s\"",
new_cluster.bindir, cluster_conn_opts(&old_cluster),
log_opts.verbose ? "--verbose" : "",
+ GET_MAJOR_VERSION(old_cluster.major_version) >= 1000 ? "--no-sync" : "",
log_opts.dumpdir,
GLOBALS_DUMP_FILE);
check_ok();
@@ -53,9 +54,10 @@ generate_old_dump(void)
parallel_exec_prog(log_file_name, NULL,
"\"%s/pg_dump\" %s --schema-only --quote-all-identifiers "
- "--binary-upgrade --format=custom %s --file=\"%s/%s\" %s",
+ "--binary-upgrade --format=custom %s %s --file=\"%s/%s\" %s",
new_cluster.bindir, cluster_conn_opts(&old_cluster),
log_opts.verbose ? "--verbose" : "",
+ GET_MAJOR_VERSION(old_cluster.major_version) >= 1000 ? "--no-sync" : "",
log_opts.dumpdir,
sql_file_name, escaped_connstr.data);
--
2.25.1
--DocE+STaALJfprDB--
^ permalink raw reply [nested|flat] 11+ messages in thread
* add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall
@ 2024-05-03 17:13 Nathan Bossart <[email protected]>
2024-05-08 08:09 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Peter Eisentraut <[email protected]>
0 siblings, 1 reply; 11+ messages in thread
From: Nathan Bossart @ 2024-05-03 17:13 UTC (permalink / raw)
To: pgsql-hackers
This is likely small potatoes compared to some of the other
pg_upgrade-related improvements I've proposed [0] [1] or plan to propose,
but this is easy enough, and I already wrote the patch, so here it is.
AFAICT there's no reason to bother syncing these dump files to disk. If
someone pulls the plug during pg_upgrade, it's not like you can resume
pg_upgrade from where it left off. Also, I think we skipped syncing before
v10, anyway, as the --no-sync flag was only added in commit 96a7128, which
added the code to sync dump files, too.
[0] https://postgr.es/m/20240418041712.GA3441570%40nathanxps13
[1] https://postgr.es/m/20240503025140.GA1227404%40nathanxps13
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall
2024-05-03 17:13 add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
@ 2024-05-08 08:09 ` Peter Eisentraut <[email protected]>
2024-05-08 18:38 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
0 siblings, 1 reply; 11+ messages in thread
From: Peter Eisentraut @ 2024-05-08 08:09 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; pgsql-hackers
On 03.05.24 19:13, Nathan Bossart wrote:
> This is likely small potatoes compared to some of the other
> pg_upgrade-related improvements I've proposed [0] [1] or plan to propose,
> but this is easy enough, and I already wrote the patch, so here it is.
> AFAICT there's no reason to bother syncing these dump files to disk. If
> someone pulls the plug during pg_upgrade, it's not like you can resume
> pg_upgrade from where it left off. Also, I think we skipped syncing before
> v10, anyway, as the --no-sync flag was only added in commit 96a7128, which
> added the code to sync dump files, too.
Looks good to me.
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall
2024-05-03 17:13 add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
2024-05-08 08:09 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Peter Eisentraut <[email protected]>
@ 2024-05-08 18:38 ` Nathan Bossart <[email protected]>
2024-05-08 18:49 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Tom Lane <[email protected]>
0 siblings, 1 reply; 11+ messages in thread
From: Nathan Bossart @ 2024-05-08 18:38 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: pgsql-hackers
On Wed, May 08, 2024 at 10:09:46AM +0200, Peter Eisentraut wrote:
> On 03.05.24 19:13, Nathan Bossart wrote:
>> This is likely small potatoes compared to some of the other
>> pg_upgrade-related improvements I've proposed [0] [1] or plan to propose,
>> but this is easy enough, and I already wrote the patch, so here it is.
>> AFAICT there's no reason to bother syncing these dump files to disk. If
>> someone pulls the plug during pg_upgrade, it's not like you can resume
>> pg_upgrade from where it left off. Also, I think we skipped syncing before
>> v10, anyway, as the --no-sync flag was only added in commit 96a7128, which
>> added the code to sync dump files, too.
>
> Looks good to me.
Thanks for looking. I noticed that the version check is unnecessary since
we always use the new binary's pg_dump[all], so I removed that in v2.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall
2024-05-03 17:13 add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
2024-05-08 08:09 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Peter Eisentraut <[email protected]>
2024-05-08 18:38 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
@ 2024-05-08 18:49 ` Tom Lane <[email protected]>
2024-05-09 00:03 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Michael Paquier <[email protected]>
0 siblings, 1 reply; 11+ messages in thread
From: Tom Lane @ 2024-05-08 18:49 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; pgsql-hackers
Nathan Bossart <[email protected]> writes:
> Thanks for looking. I noticed that the version check is unnecessary since
> we always use the new binary's pg_dump[all], so I removed that in v2.
+1
regards, tom lane
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall
2024-05-03 17:13 add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
2024-05-08 08:09 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Peter Eisentraut <[email protected]>
2024-05-08 18:38 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
2024-05-08 18:49 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Tom Lane <[email protected]>
@ 2024-05-09 00:03 ` Michael Paquier <[email protected]>
2024-05-09 19:34 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
0 siblings, 1 reply; 11+ messages in thread
From: Michael Paquier @ 2024-05-09 00:03 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Peter Eisentraut <[email protected]>; pgsql-hackers
On Wed, May 08, 2024 at 02:49:58PM -0400, Tom Lane wrote:
> Nathan Bossart <[email protected]> writes:
>> Thanks for looking. I noticed that the version check is unnecessary since
>> we always use the new binary's pg_dump[all], so I removed that in v2.
>
> +1
+1. Could there be an argument in favor of a backpatch? This is a
performance improvement, but one could also side that the addition of
sync support in pg_dump[all] has made that a regression that we'd
better fix because the flushes don't matter in this context. They
also bring costs for no gain.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall
2024-05-03 17:13 add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
2024-05-08 08:09 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Peter Eisentraut <[email protected]>
2024-05-08 18:38 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
2024-05-08 18:49 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Tom Lane <[email protected]>
2024-05-09 00:03 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Michael Paquier <[email protected]>
@ 2024-05-09 19:34 ` Nathan Bossart <[email protected]>
2024-05-09 19:58 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Daniel Gustafsson <[email protected]>
0 siblings, 1 reply; 11+ messages in thread
From: Nathan Bossart @ 2024-05-09 19:34 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; pgsql-hackers
On Thu, May 09, 2024 at 09:03:56AM +0900, Michael Paquier wrote:
> +1. Could there be an argument in favor of a backpatch? This is a
> performance improvement, but one could also side that the addition of
> sync support in pg_dump[all] has made that a regression that we'd
> better fix because the flushes don't matter in this context. They
> also bring costs for no gain.
I don't see a strong need to back-patch this, if for no other reason than
it seems to have gone unnoticed for 7 major versions. Plus, based on my
admittedly limited testing, this is unlikely to provide significant
improvements.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall
2024-05-03 17:13 add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
2024-05-08 08:09 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Peter Eisentraut <[email protected]>
2024-05-08 18:38 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
2024-05-08 18:49 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Tom Lane <[email protected]>
2024-05-09 00:03 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Michael Paquier <[email protected]>
2024-05-09 19:34 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
@ 2024-05-09 19:58 ` Daniel Gustafsson <[email protected]>
2024-07-01 15:22 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
0 siblings, 1 reply; 11+ messages in thread
From: Daniel Gustafsson @ 2024-05-09 19:58 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Michael Paquier <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; pgsql-hackers
> On 9 May 2024, at 21:34, Nathan Bossart <[email protected]> wrote:
>
> On Thu, May 09, 2024 at 09:03:56AM +0900, Michael Paquier wrote:
>> +1. Could there be an argument in favor of a backpatch? This is a
>> performance improvement, but one could also side that the addition of
>> sync support in pg_dump[all] has made that a regression that we'd
>> better fix because the flushes don't matter in this context. They
>> also bring costs for no gain.
>
> I don't see a strong need to back-patch this, if for no other reason than
> it seems to have gone unnoticed for 7 major versions. Plus, based on my
> admittedly limited testing, this is unlikely to provide significant
> improvements.
Agreed, this is a nice little improvement but it's unlikely to be enough of a
speedup to warrant changing the backbranches.
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall
2024-05-03 17:13 add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
2024-05-08 08:09 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Peter Eisentraut <[email protected]>
2024-05-08 18:38 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
2024-05-08 18:49 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Tom Lane <[email protected]>
2024-05-09 00:03 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Michael Paquier <[email protected]>
2024-05-09 19:34 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
2024-05-09 19:58 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Daniel Gustafsson <[email protected]>
@ 2024-07-01 15:22 ` Nathan Bossart <[email protected]>
0 siblings, 0 replies; 11+ messages in thread
From: Nathan Bossart @ 2024-07-01 15:22 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Michael Paquier <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; pgsql-hackers
Committed.
--
nathan
^ permalink raw reply [nested|flat] 11+ messages in thread
end of thread, other threads:[~2024-07-01 15:22 UTC | newest]
Thread overview: 11+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-09-29 14:19 [PATCH v45 7/7] Exclude pg_stat directory from base backup Kyotaro Horiguchi <[email protected]>
2024-05-03 15:35 [PATCH v2 1/1] add --no-sync to pg_upgrade's calls to pg_dump[all] Nathan Bossart <[email protected]>
2024-05-03 15:35 [PATCH v1 1/1] add --no-sync to pg_upgrade's calls to pg_dump[all] Nathan Bossart <[email protected]>
2024-05-03 17:13 add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
2024-05-08 08:09 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Peter Eisentraut <[email protected]>
2024-05-08 18:38 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
2024-05-08 18:49 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Tom Lane <[email protected]>
2024-05-09 00:03 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Michael Paquier <[email protected]>
2024-05-09 19:34 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
2024-05-09 19:58 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Daniel Gustafsson <[email protected]>
2024-07-01 15:22 ` Re: add --no-sync to pg_upgrade's calls to pg_dump and pg_dumpall Nathan Bossart <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox