public inbox for [email protected]
help / color / mirror / Atom feedstandby recovery fails (tablespace related) (tentative patch and discussion)
109+ messages / 18 participants
[nested] [flat]
* standby recovery fails (tablespace related) (tentative patch and discussion)
@ 2019-04-17 07:56 Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Paul Guo @ 2019-04-17 07:56 UTC (permalink / raw)
To: pgsql-hackers
Hello postgres hackers,
Recently my colleagues and I encountered an issue: a standby can
not recover after an unclean shutdown and it's related to tablespace.
The issue is that the standby re-replay some xlog that needs tablespace
directories (e.g. create a database with tablespace),
but the tablespace directories has already been removed in the
previous replay.
In details, the standby normally finishes replaying for the below
operations, but due to unclean shutdown, the redo lsn
is not updated in pg_control and is still kept a value before the 'create
db with tabspace' xlog, however since the tablespace
directories were removed so it reports error when repay the database create
wal.
create db with tablespace
drop database
drop tablespace.
Here is the log on the standby.
2019-04-17 14:52:14.926 CST [23029] LOG: starting PostgreSQL 12devel on
x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat
4.8.5-4), 64-bit
2019-04-17 14:52:14.927 CST [23029] LOG: listening on IPv4 address
"192.168.35.130", port 5432
2019-04-17 14:52:14.929 CST [23029] LOG: listening on Unix socket
"/tmp/.s.PGSQL.5432"
2019-04-17 14:52:14.943 CST [23030] LOG: database system was interrupted
while in recovery at log time 2019-04-17 14:48:27 CST
2019-04-17 14:52:14.943 CST [23030] HINT: If this has occurred more than
once some data might be corrupted and you might need to choose an earlier
recovery target.
2019-04-17 14:52:14.949 CST [23030] LOG: entering standby mode
2019-04-17 14:52:14.950 CST [23030] LOG: redo starts at 0/30105B8
2019-04-17 14:52:14.951 CST [23030] FATAL: could not create directory
"pg_tblspc/65546/PG_12_201904072/65547": No such file or directory
2019-04-17 14:52:14.951 CST [23030] CONTEXT: WAL redo at 0/3011650 for
Database/CREATE: copy dir 1663/1 to 65546/65547
2019-04-17 14:52:14.951 CST [23029] LOG: startup process (PID 23030)
exited with exit code 1
2019-04-17 14:52:14.951 CST [23029] LOG: terminating any other active
server processes
2019-04-17 14:52:14.953 CST [23029] LOG: database system is shut down
Steps to reprodce:
1. setup a master and standby.
2. On both side, run: mkdir /tmp/some_isolation2_pg_basebackup_tablespace
3. Run SQLs:
drop tablespace if exists some_isolation2_pg_basebackup_tablespace;
create tablespace some_isolation2_pg_basebackup_tablespace location
'/tmp/some_isolation2_pg_basebackup_tablespace';
3. Clean shutdown and restart both postgres instances.
4. Run the following SQLs:
drop database if exists some_database_with_tablespace;
create database some_database_with_tablespace tablespace
some_isolation2_pg_basebackup_tablespace;
drop database some_database_with_tablespace;
drop tablespace some_isolation2_pg_basebackup_tablespace;
\! pkill -9 postgres; ssh host70 pkill -9 postgres
Note immediate shutdown via pg_ctl should also be able to reproduce and the
above steps probably does not 100% reproduce.
I created an initial patch for this issue (see the attachment). The idea is
re-creating those directories recursively. The above issue exists
in dbase_redo(),
but TablespaceCreateDbspace (for relation file create redo) is probably
buggy also so I modified that function also. Even there is no bug
in that function, it seems that using simple pg_mkdir_p() is cleaner. Note
reading TablespaceCreateDbspace(), I found it seems that this issue
has already be thought though insufficient but frankly this solution
(directory recreation) seems to be not perfect given actually this should
have been the responsibility of tablespace creation (also tablespace
creation does more like symlink creation, etc). Also, I'm not sure whether
we need to use invalid page mechanism (see xlogutils.c).
Another solution is that, actually, we create a checkpoint when
createdb/movedb/dropdb/droptablespace, maybe we should enforce to create
restartpoint on standby for such special kind of checkpoint wal - that
means we need to set a flag in checkpoing wal and let checkpoint redo
code to create restartpoint if that flag is set. This solution seems to be
safer.
Thanks,
Paul
Attachments:
[application/octet-stream] 0001-Recursively-create-tablespace-directories-if-those-a.patch (3.7K, ../../CAEET0ZGx9AvioViLf7nbR_8tH9-=27DN5xWJ2P9-ROH16e4JUA@mail.gmail.com/3-0001-Recursively-create-tablespace-directories-if-those-a.patch)
download | inline diff:
From 7beccf4b41ebf8acf83ea706e2869e48867a40d3 Mon Sep 17 00:00:00 2001
From: Paul Guo <[email protected]>
Date: Wed, 17 Apr 2019 00:12:31 -0700
Subject: [PATCH] Recursively create tablespace directories if those are gone
but we need that when re-redoing some tablespace related xlogs (e.g. database
create).
---
src/backend/commands/dbcommands.c | 18 ++++++++++++++++++
src/backend/commands/tablespace.c | 28 +---------------------------
2 files changed, 19 insertions(+), 27 deletions(-)
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index 9707afa..9999b9b 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -45,6 +45,7 @@
#include "commands/defrem.h"
#include "commands/seclabel.h"
#include "commands/tablespace.h"
+#include "common/file_perm.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "pgstat.h"
@@ -2089,6 +2090,7 @@ dbase_redo(XLogReaderState *record)
xl_dbase_create_rec *xlrec = (xl_dbase_create_rec *) XLogRecGetData(record);
char *src_path;
char *dst_path;
+ char *parent_path;
struct stat st;
src_path = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
@@ -2107,6 +2109,22 @@ dbase_redo(XLogReaderState *record)
(errmsg("some useless files may be left behind in old database directory \"%s\"",
dst_path)));
}
+ else
+ {
+ /*
+ * It is possible that the tablespace was later dropped, but we are
+ * re-redoing database create before that. In that case,
+ * those directories are gone, and we do not create symlink.
+ */
+ parent_path = pstrdup(dst_path);
+ get_parent_directory(parent_path);
+ if (stat(parent_path, &st) != 0 && pg_mkdir_p(parent_path, pg_dir_create_mode) != 0)
+ {
+ ereport(WARNING,
+ (errmsg("can not recursively create directory \"%s\"",
+ parent_path)));
+ }
+ }
/*
* Force dirty buffers out to disk, to ensure source database is
diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c
index 3784ea4..f0fac11 100644
--- a/src/backend/commands/tablespace.c
+++ b/src/backend/commands/tablespace.c
@@ -154,8 +154,6 @@ TablespaceCreateDbspace(Oid spcNode, Oid dbNode, bool isRedo)
/* Directory creation failed? */
if (MakePGDirectory(dir) < 0)
{
- char *parentdir;
-
/* Failure other than not exists or not in WAL replay? */
if (errno != ENOENT || !isRedo)
ereport(ERROR,
@@ -168,32 +166,8 @@ TablespaceCreateDbspace(Oid spcNode, Oid dbNode, bool isRedo)
* continue by creating simple parent directories rather
* than a symlink.
*/
-
- /* create two parents up if not exist */
- parentdir = pstrdup(dir);
- get_parent_directory(parentdir);
- get_parent_directory(parentdir);
- /* Can't create parent and it doesn't already exist? */
- if (MakePGDirectory(parentdir) < 0 && errno != EEXIST)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not create directory \"%s\": %m",
- parentdir)));
- pfree(parentdir);
-
- /* create one parent up if not exist */
- parentdir = pstrdup(dir);
- get_parent_directory(parentdir);
- /* Can't create parent and it doesn't already exist? */
- if (MakePGDirectory(parentdir) < 0 && errno != EEXIST)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not create directory \"%s\": %m",
- parentdir)));
- pfree(parentdir);
-
/* Create database directory */
- if (MakePGDirectory(dir) < 0)
+ if (pg_mkdir_p(dir, pg_dir_create_mode) < 0)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not create directory \"%s\": %m",
--
1.8.3.1
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
@ 2019-04-19 04:38 ` Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Asim R P @ 2019-04-19 04:38 UTC (permalink / raw)
To: Paul Guo <[email protected]>; +Cc: pgsql-hackers
On Wed, Apr 17, 2019 at 1:27 PM Paul Guo <[email protected]> wrote:
>
> create db with tablespace
> drop database
> drop tablespace.
Essentially, that sequence of operations causes crash recovery to fail
if the "drop tablespace" transaction was committed before crashing.
This is a bug in crash recovery in general and should be reproducible
without configuring a standby. Is that right?
Your patch creates missing directories in the destination. Don't we
need to create the tablespace symlink under pg_tblspc/? I would
prefer extending the invalid page mechanism to deal with this, as
suggested by Ashwin off-list. It will allow us to avoid creating
directories and files only to remove them shortly afterwards when the
drop database and drop tablespace records are replayed.
Asim
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
@ 2019-04-22 04:36 ` Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Paul Guo @ 2019-04-22 04:36 UTC (permalink / raw)
To: Asim R P <[email protected]>; +Cc: pgsql-hackers
Please see my replies inline. Thanks.
On Fri, Apr 19, 2019 at 12:38 PM Asim R P <[email protected]> wrote:
> On Wed, Apr 17, 2019 at 1:27 PM Paul Guo <[email protected]> wrote:
> >
> > create db with tablespace
> > drop database
> > drop tablespace.
>
> Essentially, that sequence of operations causes crash recovery to fail
> if the "drop tablespace" transaction was committed before crashing.
> This is a bug in crash recovery in general and should be reproducible
> without configuring a standby. Is that right?
>
No. In general, checkpoint is done for drop_db/create_db/drop_tablespace on
master.
That makes the file/directory update-to-date if I understand the related
code correctly.
For standby, checkpoint redo does not ensure that.
>
> Your patch creates missing directories in the destination. Don't we
> need to create the tablespace symlink under pg_tblspc/? I would
>
'create db with tablespace' redo log does not include the tablespace real
directory information.
Yes, we could add in it into the xlog, but that seems to be an overdesign.
> prefer extending the invalid page mechanism to deal with this, as
> suggested by Ashwin off-list. It will allow us to avoid creating
directories and files only to remove them shortly afterwards when the
> drop database and drop tablespace records are replayed.
>
>
'invalid page' mechanism seems to be more proper for missing pages of a
file. For
missing directories, we could, of course, hack to use that (e.g. reading
any page of
a relfile in that database) to make sure the tablespace create code
(without symlink)
safer (It assumes those directories will be deleted soon).
More feedback about all of the previous discussed solutions is welcome.
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
@ 2019-04-22 07:15 ` Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Kyotaro HORIGUCHI @ 2019-04-22 07:15 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; pgsql-hackers
Hello.
At Mon, 22 Apr 2019 12:36:43 +0800, Paul Guo <[email protected]> wrote in <CAEET0ZGpUrMGUzfyzVF9FuSq+zb=QovYa2cvyRnDOTvZ5vXxTw@mail.gmail.com>
> Please see my replies inline. Thanks.
>
> On Fri, Apr 19, 2019 at 12:38 PM Asim R P <[email protected]> wrote:
>
> > On Wed, Apr 17, 2019 at 1:27 PM Paul Guo <[email protected]> wrote:
> > >
> > > create db with tablespace
> > > drop database
> > > drop tablespace.
> >
> > Essentially, that sequence of operations causes crash recovery to fail
> > if the "drop tablespace" transaction was committed before crashing.
> > This is a bug in crash recovery in general and should be reproducible
> > without configuring a standby. Is that right?
> >
>
> No. In general, checkpoint is done for drop_db/create_db/drop_tablespace on
> master.
> That makes the file/directory update-to-date if I understand the related
> code correctly.
> For standby, checkpoint redo does not ensure that.
That's right partly. As you must have seen, fast shutdown forces
restartpoint for the last checkpoint and it prevents the problem
from happening. Anyway it seems to be a problem.
> > Your patch creates missing directories in the destination. Don't we
> > need to create the tablespace symlink under pg_tblspc/? I would
> >
>
> 'create db with tablespace' redo log does not include the tablespace real
> directory information.
> Yes, we could add in it into the xlog, but that seems to be an overdesign.
But I don't think creating directory that is to be removed just
after is a wanted solution. The directory most likely to be be
removed just after.
> > prefer extending the invalid page mechanism to deal with this, as
> > suggested by Ashwin off-list. It will allow us to avoid creating
>
> directories and files only to remove them shortly afterwards when the
> > drop database and drop tablespace records are replayed.
> >
> >
> 'invalid page' mechanism seems to be more proper for missing pages of a
> file. For
> missing directories, we could, of course, hack to use that (e.g. reading
> any page of
> a relfile in that database) to make sure the tablespace create code
> (without symlink)
> safer (It assumes those directories will be deleted soon).
>
> More feedback about all of the previous discussed solutions is welcome.
It doesn't seem to me that the invalid page mechanism is
applicable in straightforward way, because it doesn't consider
simple file copy.
Drop failure is ignored any time. I suppose we can ignore the
error to continue recovering as far as recovery have not reached
consistency. The attached would work *at least* your case, but I
haven't checked this covers all places where need the same
treatment.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
Attachments:
[text/x-patch] ignore_dir_create_error_before_consistency.patch (2.8K, ../../[email protected]/2-ignore_dir_create_error_before_consistency.patch)
download | inline diff:
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index 10a663bae6..0bc63f48da 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -522,6 +522,44 @@ XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
return buffer;
}
+/*
+ * XLogMakePGDirectory
+ *
+ * There is a possibility that WAL replay causes a creation of the same
+ * directory left by the previous crash. Issuing ERROR prevents the caller
+ * from continuing recovery.
+ *
+ * To prevent that case, this function issues WARNING instead of ERROR on
+ * error if consistency is not reached yet.
+ */
+int
+XLogMakePGDirectory(const char *directoryName)
+{
+ int ret;
+
+ ret = MakePGDirectory(directoryName);
+
+ if (ret != 0)
+ {
+ int elevel = ERROR;
+
+ /*
+ * We might get error trying to create existing directory that is to
+ * be removed just after. Don't issue ERROR in the case. Recovery
+ * will stop if we again failed after reaching consistency.
+ */
+ if (InRecovery && !reachedConsistency)
+ elevel = WARNING;
+
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg("could not create directory \"%s\": %m", directoryName)));
+ return ret;
+ }
+
+ return 0;
+}
+
/*
* Struct actually returned by XLogFakeRelcacheEntry, though the declared
* return type is Relation.
diff --git a/src/backend/storage/file/copydir.c b/src/backend/storage/file/copydir.c
index 30f6200a86..0216270dd3 100644
--- a/src/backend/storage/file/copydir.c
+++ b/src/backend/storage/file/copydir.c
@@ -22,11 +22,11 @@
#include <unistd.h>
#include <sys/stat.h>
+#include "access/xlogutils.h"
#include "storage/copydir.h"
#include "storage/fd.h"
#include "miscadmin.h"
#include "pgstat.h"
-
/*
* copydir: copy a directory
*
@@ -41,10 +41,12 @@ copydir(char *fromdir, char *todir, bool recurse)
char fromfile[MAXPGPATH * 2];
char tofile[MAXPGPATH * 2];
- if (MakePGDirectory(todir) != 0)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not create directory \"%s\": %m", todir)));
+ /*
+ * We might have to skip copydir to continue recovery. See the function
+ * for details.
+ */
+ if (XLogMakePGDirectory(todir) != 0)
+ return;
xldir = AllocateDir(fromdir);
diff --git a/src/include/access/xlogutils.h b/src/include/access/xlogutils.h
index 0ab5ba62f5..46a7596315 100644
--- a/src/include/access/xlogutils.h
+++ b/src/include/access/xlogutils.h
@@ -43,6 +43,7 @@ extern XLogRedoAction XLogReadBufferForRedoExtended(XLogReaderState *record,
extern Buffer XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
BlockNumber blkno, ReadBufferMode mode);
+extern int XLogMakePGDirectory(const char *directoryName);
extern Relation CreateFakeRelcacheEntry(RelFileNode rnode);
extern void FreeFakeRelcacheEntry(Relation fakerel);
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
@ 2019-04-22 07:40 ` Kyotaro HORIGUCHI <[email protected]>
2019-04-22 12:19 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
0 siblings, 2 replies; 109+ messages in thread
From: Kyotaro HORIGUCHI @ 2019-04-22 07:40 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; pgsql-hackers
Oops! The comment in the previous patch is wrong.
At Mon, 22 Apr 2019 16:15:13 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <[email protected]> wrote in <[email protected]>
> At Mon, 22 Apr 2019 12:36:43 +0800, Paul Guo <[email protected]> wrote in <CAEET0ZGpUrMGUzfyzVF9FuSq+zb=QovYa2cvyRnDOTvZ5vXxTw@mail.gmail.com>
> > Please see my replies inline. Thanks.
> >
> > On Fri, Apr 19, 2019 at 12:38 PM Asim R P <[email protected]> wrote:
> >
> > > On Wed, Apr 17, 2019 at 1:27 PM Paul Guo <[email protected]> wrote:
> > > >
> > > > create db with tablespace
> > > > drop database
> > > > drop tablespace.
> > >
> > > Essentially, that sequence of operations causes crash recovery to fail
> > > if the "drop tablespace" transaction was committed before crashing.
> > > This is a bug in crash recovery in general and should be reproducible
> > > without configuring a standby. Is that right?
> > >
> >
> > No. In general, checkpoint is done for drop_db/create_db/drop_tablespace on
> > master.
> > That makes the file/directory update-to-date if I understand the related
> > code correctly.
> > For standby, checkpoint redo does not ensure that.
>
> That's right partly. As you must have seen, fast shutdown forces
> restartpoint for the last checkpoint and it prevents the problem
> from happening. Anyway it seems to be a problem.
>
> > > Your patch creates missing directories in the destination. Don't we
> > > need to create the tablespace symlink under pg_tblspc/? I would
> > >
> >
> > 'create db with tablespace' redo log does not include the tablespace real
> > directory information.
> > Yes, we could add in it into the xlog, but that seems to be an overdesign.
>
> But I don't think creating directory that is to be removed just
> after is a wanted solution. The directory most likely to be be
> removed just after.
>
> > > prefer extending the invalid page mechanism to deal with this, as
> > > suggested by Ashwin off-list. It will allow us to avoid creating
> >
> > directories and files only to remove them shortly afterwards when the
> > > drop database and drop tablespace records are replayed.
> > >
> > >
> > 'invalid page' mechanism seems to be more proper for missing pages of a
> > file. For
> > missing directories, we could, of course, hack to use that (e.g. reading
> > any page of
> > a relfile in that database) to make sure the tablespace create code
> > (without symlink)
> > safer (It assumes those directories will be deleted soon).
> >
> > More feedback about all of the previous discussed solutions is welcome.
>
> It doesn't seem to me that the invalid page mechanism is
> applicable in straightforward way, because it doesn't consider
> simple file copy.
>
> Drop failure is ignored any time. I suppose we can ignore the
> error to continue recovering as far as recovery have not reached
> consistency. The attached would work *at least* your case, but I
> haven't checked this covers all places where need the same
> treatment.
The comment for the new function XLogMakePGDirectory is wrong:
+ * There is a possibility that WAL replay causes a creation of the same
+ * directory left by the previous crash. Issuing ERROR prevents the caller
+ * from continuing recovery.
The correct one is:
+ * There is a possibility that WAL replay causes an error by creation of a
+ * directory under a directory removed before the previous crash. Issuing
+ * ERROR prevents the caller from continuing recovery.
It is fixed in the attached.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
Attachments:
[text/x-patch] ignore_dir_create_error_before_consistency_v2.patch (2.8K, ../../[email protected]/2-ignore_dir_create_error_before_consistency_v2.patch)
download | inline diff:
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index 10a663bae6..01331f0da9 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -522,6 +522,44 @@ XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
return buffer;
}
+/*
+ * XLogMakePGDirectory
+ *
+ * There is a possibility that WAL replay causes an error by creation of a
+ * directory under a directory removed before the previous crash. Issuing
+ * ERROR prevents the caller from continuing recovery.
+ *
+ * To prevent that case, this function issues WARNING instead of ERROR on
+ * error if consistency is not reached yet.
+ */
+int
+XLogMakePGDirectory(const char *directoryName)
+{
+ int ret;
+
+ ret = MakePGDirectory(directoryName);
+
+ if (ret != 0)
+ {
+ int elevel = ERROR;
+
+ /*
+ * We might get error trying to create existing directory that is to
+ * be removed just after. Don't issue ERROR in the case. Recovery
+ * will stop if we again failed after reaching consistency.
+ */
+ if (InRecovery && !reachedConsistency)
+ elevel = WARNING;
+
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg("could not create directory \"%s\": %m", directoryName)));
+ return ret;
+ }
+
+ return 0;
+}
+
/*
* Struct actually returned by XLogFakeRelcacheEntry, though the declared
* return type is Relation.
diff --git a/src/backend/storage/file/copydir.c b/src/backend/storage/file/copydir.c
index 30f6200a86..0216270dd3 100644
--- a/src/backend/storage/file/copydir.c
+++ b/src/backend/storage/file/copydir.c
@@ -22,11 +22,11 @@
#include <unistd.h>
#include <sys/stat.h>
+#include "access/xlogutils.h"
#include "storage/copydir.h"
#include "storage/fd.h"
#include "miscadmin.h"
#include "pgstat.h"
-
/*
* copydir: copy a directory
*
@@ -41,10 +41,12 @@ copydir(char *fromdir, char *todir, bool recurse)
char fromfile[MAXPGPATH * 2];
char tofile[MAXPGPATH * 2];
- if (MakePGDirectory(todir) != 0)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not create directory \"%s\": %m", todir)));
+ /*
+ * We might have to skip copydir to continue recovery. See the function
+ * for details.
+ */
+ if (XLogMakePGDirectory(todir) != 0)
+ return;
xldir = AllocateDir(fromdir);
diff --git a/src/include/access/xlogutils.h b/src/include/access/xlogutils.h
index 0ab5ba62f5..46a7596315 100644
--- a/src/include/access/xlogutils.h
+++ b/src/include/access/xlogutils.h
@@ -43,6 +43,7 @@ extern XLogRedoAction XLogReadBufferForRedoExtended(XLogReaderState *record,
extern Buffer XLogReadBufferExtended(RelFileNode rnode, ForkNumber forknum,
BlockNumber blkno, ReadBufferMode mode);
+extern int XLogMakePGDirectory(const char *directoryName);
extern Relation CreateFakeRelcacheEntry(RelFileNode rnode);
extern void FreeFakeRelcacheEntry(Relation fakerel);
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
@ 2019-04-22 12:19 ` Kyotaro HORIGUCHI <[email protected]>
2019-04-23 02:34 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Michael Paquier <[email protected]>
1 sibling, 1 reply; 109+ messages in thread
From: Kyotaro HORIGUCHI @ 2019-04-22 12:19 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; pgsql-hackers
At Mon, 22 Apr 2019 16:40:27 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <[email protected]> wrote in <[email protected]>
> At Mon, 22 Apr 2019 16:15:13 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <[email protected]> wrote in <[email protected]>
> > At Mon, 22 Apr 2019 12:36:43 +0800, Paul Guo <[email protected]> wrote in <CAEET0ZGpUrMGUzfyzVF9FuSq+zb=QovYa2cvyRnDOTvZ5vXxTw@mail.gmail.com>
> > > Please see my replies inline. Thanks.
> > >
> > > On Fri, Apr 19, 2019 at 12:38 PM Asim R P <[email protected]> wrote:
> > >
> > > > On Wed, Apr 17, 2019 at 1:27 PM Paul Guo <[email protected]> wrote:
> > > > >
> > > > > create db with tablespace
> > > > > drop database
> > > > > drop tablespace.
> > > >
> > > > Essentially, that sequence of operations causes crash recovery to fail
> > > > if the "drop tablespace" transaction was committed before crashing.
> > > > This is a bug in crash recovery in general and should be reproducible
> > > > without configuring a standby. Is that right?
> > > >
> > >
> > > No. In general, checkpoint is done for drop_db/create_db/drop_tablespace on
> > > master.
> > > That makes the file/directory update-to-date if I understand the related
> > > code correctly.
> > > For standby, checkpoint redo does not ensure that.
The attached exercises this sequence, needing some changes in
PostgresNode.pm and RecursiveCopy.pm to allow tablespaces.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 12:19 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
@ 2019-04-23 02:34 ` Michael Paquier <[email protected]>
2019-04-23 05:02 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Michael Paquier @ 2019-04-23 02:34 UTC (permalink / raw)
To: Kyotaro HORIGUCHI <[email protected]>; +Cc: [email protected]; [email protected]; pgsql-hackers
On Mon, Apr 22, 2019 at 09:19:33PM +0900, Kyotaro HORIGUCHI wrote:
> The attached exercises this sequence, needing some changes in
> PostgresNode.pm and RecursiveCopy.pm to allow tablespaces.
+ # Check for symlink -- needed only on source dir
+ # (note: this will fall through quietly if file is already gone)
+ if (-l $srcpath)
+ {
+ croak "Cannot operate on symlink \"$srcpath\""
+ if ($srcpath !~ /\/(pg_tblspc\/[0-9]+)$/);
+
+ # We have mapped tablespaces. Copy them individually
+ my $linkname = $1;
+ my $tmpdir = TestLib::tempdir;
+ my $dstrealdir = TestLib::real_dir($tmpdir);
+ my $srcrealdir = readlink($srcpath);
+
+ opendir(my $dh, $srcrealdir);
+ while (readdir $dh)
+ {
+ next if (/^\.\.?$/);
+ my $spath = "$srcrealdir/$_";
+ my $dpath = "$dstrealdir/$_";
+
+ copypath($spath, $dpath);
+ }
+ closedir $dh;
+
+ symlink $dstrealdir, $destpath;
+ return 1;
+ }
The same stuff is proposed here:
https://www.postgresql.org/message-id/[email protected]...
So there is a lot of demand for making the recursive copy more skilled
at handling symlinks for tablespace tests, and I'd like to propose to
do something among those lines for the tests on HEAD, presumably for
v12 and not v13 as we are talking about a bug fix here? I am not sure
yet which one of the proposals is better than the other though.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 12:19 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 02:34 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Michael Paquier <[email protected]>
@ 2019-04-23 05:02 ` Kyotaro HORIGUCHI <[email protected]>
0 siblings, 0 replies; 109+ messages in thread
From: Kyotaro HORIGUCHI @ 2019-04-23 05:02 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; pgsql-hackers
At Tue, 23 Apr 2019 11:34:38 +0900, Michael Paquier <[email protected]> wrote in <[email protected]>
> On Mon, Apr 22, 2019 at 09:19:33PM +0900, Kyotaro HORIGUCHI wrote:
> > The attached exercises this sequence, needing some changes in
> > PostgresNode.pm and RecursiveCopy.pm to allow tablespaces.
>
> + # Check for symlink -- needed only on source dir
> + # (note: this will fall through quietly if file is already gone)
> + if (-l $srcpath)
> + {
> + croak "Cannot operate on symlink \"$srcpath\""
> + if ($srcpath !~ /\/(pg_tblspc\/[0-9]+)$/);
> +
> + # We have mapped tablespaces. Copy them individually
> + my $linkname = $1;
> + my $tmpdir = TestLib::tempdir;
> + my $dstrealdir = TestLib::real_dir($tmpdir);
> + my $srcrealdir = readlink($srcpath);
> +
> + opendir(my $dh, $srcrealdir);
> + while (readdir $dh)
> + {
> + next if (/^\.\.?$/);
> + my $spath = "$srcrealdir/$_";
> + my $dpath = "$dstrealdir/$_";
> +
> + copypath($spath, $dpath);
> + }
> + closedir $dh;
> +
> + symlink $dstrealdir, $destpath;
> + return 1;
> + }
>
> The same stuff is proposed here:
> https://www.postgresql.org/message-id/[email protected]...
>
> So there is a lot of demand for making the recursive copy more skilled
> at handling symlinks for tablespace tests, and I'd like to propose to
> do something among those lines for the tests on HEAD, presumably for
> v12 and not v13 as we are talking about a bug fix here? I am not sure
> yet which one of the proposals is better than the other though.
TBH I like that (my one cieted above) not so much. However, I
prefer to have v12 if this is a bug and to be fixed in
v12. Otherwise we won't add a test for this later:p
Anyway I'll visit there. Thanks.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
@ 2019-04-23 05:31 ` Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
1 sibling, 1 reply; 109+ messages in thread
From: Paul Guo @ 2019-04-23 05:31 UTC (permalink / raw)
To: Kyotaro HORIGUCHI <[email protected]>; +Cc: Asim Praveen <[email protected]>; pgsql-hackers
Hi Kyotaro, ignoring the MakePGDirectory() failure will fix this database
create redo error, but I suspect some other kind of redo, which depends on
the files under the directory (they are not copied since the directory is
not created) and also cannot be covered by the invalid page mechanism,
could fail. Thanks.
On Mon, Apr 22, 2019 at 3:40 PM Kyotaro HORIGUCHI <
[email protected]> wrote:
> Oops! The comment in the previous patch is wrong.
>
> At Mon, 22 Apr 2019 16:15:13 +0900 (Tokyo Standard Time), Kyotaro
> HORIGUCHI <[email protected]> wrote in <
> [email protected]>
> > At Mon, 22 Apr 2019 12:36:43 +0800, Paul Guo <[email protected]> wrote in
> <CAEET0ZGpUrMGUzfyzVF9FuSq+zb=QovYa2cvyRnDOTvZ5vXxTw@mail.gmail.com>
> > > Please see my replies inline. Thanks.
> > >
> > > On Fri, Apr 19, 2019 at 12:38 PM Asim R P <[email protected]> wrote:
> > >
> > > > On Wed, Apr 17, 2019 at 1:27 PM Paul Guo <[email protected]> wrote:
> > > > >
> > > > > create db with tablespace
> > > > > drop database
> > > > > drop tablespace.
> > > >
> > > > Essentially, that sequence of operations causes crash recovery to
> fail
> > > > if the "drop tablespace" transaction was committed before crashing.
> > > > This is a bug in crash recovery in general and should be reproducible
> > > > without configuring a standby. Is that right?
> > > >
> > >
> > > No. In general, checkpoint is done for
> drop_db/create_db/drop_tablespace on
> > > master.
> > > That makes the file/directory update-to-date if I understand the
> related
> > > code correctly.
> > > For standby, checkpoint redo does not ensure that.
> >
> > That's right partly. As you must have seen, fast shutdown forces
> > restartpoint for the last checkpoint and it prevents the problem
> > from happening. Anyway it seems to be a problem.
> >
> > > > Your patch creates missing directories in the destination. Don't we
> > > > need to create the tablespace symlink under pg_tblspc/? I would
> > > >
> > >
> > > 'create db with tablespace' redo log does not include the tablespace
> real
> > > directory information.
> > > Yes, we could add in it into the xlog, but that seems to be an
> overdesign.
> >
> > But I don't think creating directory that is to be removed just
> > after is a wanted solution. The directory most likely to be be
> > removed just after.
> >
> > > > prefer extending the invalid page mechanism to deal with this, as
> > > > suggested by Ashwin off-list. It will allow us to avoid creating
> > >
> > > directories and files only to remove them shortly afterwards when the
> > > > drop database and drop tablespace records are replayed.
> > > >
> > > >
> > > 'invalid page' mechanism seems to be more proper for missing pages of a
> > > file. For
> > > missing directories, we could, of course, hack to use that (e.g.
> reading
> > > any page of
> > > a relfile in that database) to make sure the tablespace create code
> > > (without symlink)
> > > safer (It assumes those directories will be deleted soon).
> > >
> > > More feedback about all of the previous discussed solutions is welcome.
> >
> > It doesn't seem to me that the invalid page mechanism is
> > applicable in straightforward way, because it doesn't consider
> > simple file copy.
> >
> > Drop failure is ignored any time. I suppose we can ignore the
> > error to continue recovering as far as recovery have not reached
> > consistency. The attached would work *at least* your case, but I
> > haven't checked this covers all places where need the same
> > treatment.
>
> The comment for the new function XLogMakePGDirectory is wrong:
>
> + * There is a possibility that WAL replay causes a creation of the same
> + * directory left by the previous crash. Issuing ERROR prevents the caller
> + * from continuing recovery.
>
> The correct one is:
>
> + * There is a possibility that WAL replay causes an error by creation of a
> + * directory under a directory removed before the previous crash. Issuing
> + * ERROR prevents the caller from continuing recovery.
>
> It is fixed in the attached.
>
> regards.
>
> --
> Kyotaro Horiguchi
> NTT Open Source Software Center
>
>
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
@ 2019-04-23 07:39 ` Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Kyotaro HORIGUCHI @ 2019-04-23 07:39 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; pgsql-hackers
Hello.
At Tue, 23 Apr 2019 13:31:58 +0800, Paul Guo <[email protected]> wrote in <CAEET0ZEcwz57z2yfWRds43b3TfQPPDSWmbjGmD43xRxLT41NDg@mail.gmail.com>
> Hi Kyotaro, ignoring the MakePGDirectory() failure will fix this database
> create redo error, but I suspect some other kind of redo, which depends on
> the files under the directory (they are not copied since the directory is
> not created) and also cannot be covered by the invalid page mechanism,
> could fail. Thanks.
If recovery starts from just after tablespace creation, that's
simple. The Symlink to the removed tablespace is already removed
in the case. Hence server innocently create files directly under
pg_tblspc, not in the tablespace. Finally all files that were
supposed to be created in the removed tablespace are removed
later in recovery.
If recovery starts from recalling page in a file that have been
in the tablespace, XLogReadBufferExtended creates one (perhaps
directly in pg_tblspc as described above) and the files are
removed later in recoery the same way to above. This case doen't
cause FATAL/PANIC during recovery even in master.
[email protected]
| * Create the target file if it doesn't already exist. This lets us cope
| * if the replay sequence contains writes to a relation that is later
| * deleted. (The original coding of this routine would instead suppress
| * the writes, but that seems like it risks losing valuable data if the
| * filesystem loses an inode during a crash. Better to write the data
| * until we are actually told to delete the file.)
So buffered access cannot be a problem for the reason above. The
remaining possible issue is non-buffered access to files in
removed tablespaces. This is what I mentioned upthread:
me> but I haven't checked this covers all places where need the same
me> treatment.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
@ 2019-04-24 08:13 ` Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Kyotaro HORIGUCHI @ 2019-04-24 08:13 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; pgsql-hackers
Mmm. I posted to wrong thread. Sorry.
At Tue, 23 Apr 2019 16:39:49 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <[email protected]> wrote in <[email protected]>
> At Tue, 23 Apr 2019 13:31:58 +0800, Paul Guo <[email protected]> wrote in <CAEET0ZEcwz57z2yfWRds43b3TfQPPDSWmbjGmD43xRxLT41NDg@mail.gmail.com>
> > Hi Kyotaro, ignoring the MakePGDirectory() failure will fix this database
> > create redo error, but I suspect some other kind of redo, which depends on
> > the files under the directory (they are not copied since the directory is
> > not created) and also cannot be covered by the invalid page mechanism,
> > could fail. Thanks.
>
> If recovery starts from just after tablespace creation, that's
> simple. The Symlink to the removed tablespace is already removed
> in the case. Hence server innocently create files directly under
> pg_tblspc, not in the tablespace. Finally all files that were
> supposed to be created in the removed tablespace are removed
> later in recovery.
>
> If recovery starts from recalling page in a file that have been
> in the tablespace, XLogReadBufferExtended creates one (perhaps
> directly in pg_tblspc as described above) and the files are
> removed later in recoery the same way to above. This case doen't
> cause FATAL/PANIC during recovery even in master.
>
> [email protected]
> | * Create the target file if it doesn't already exist. This lets us cope
> | * if the replay sequence contains writes to a relation that is later
> | * deleted. (The original coding of this routine would instead suppress
> | * the writes, but that seems like it risks losing valuable data if the
> | * filesystem loses an inode during a crash. Better to write the data
> | * until we are actually told to delete the file.)
>
> So buffered access cannot be a problem for the reason above. The
> remaining possible issue is non-buffered access to files in
> removed tablespaces. This is what I mentioned upthread:
>
> me> but I haven't checked this covers all places where need the same
> me> treatment.
RM_DBASE_ID is fixed by the patch.
XLOG/XACT/CLOG/MULTIXACT/RELMAP/STANDBY/COMMIT_TS/REPLORIGIN/LOGICALMSG:
- are not relevant.
HEAP/HEAP2/BTREE/HASH/GIN/GIST/SEQ/SPGIST/BRIN/GENERIC:
- Resources works on buffer is not affected.
SMGR:
- Both CREATE and TRUNCATE seems fine.
TBLSPC:
- We don't nest tablespace directories. No Problem.
I don't find a similar case.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
@ 2019-04-28 07:33 ` Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Paul Guo @ 2019-04-28 07:33 UTC (permalink / raw)
To: Kyotaro HORIGUCHI <[email protected]>; +Cc: Asim Praveen <[email protected]>; pgsql-hackers
On Wed, Apr 24, 2019 at 4:14 PM Kyotaro HORIGUCHI <
[email protected]> wrote:
> Mmm. I posted to wrong thread. Sorry.
>
> At Tue, 23 Apr 2019 16:39:49 +0900 (Tokyo Standard Time), Kyotaro
> HORIGUCHI <[email protected]> wrote in <
> [email protected]>
> > At Tue, 23 Apr 2019 13:31:58 +0800, Paul Guo <[email protected]> wrote in
> <CAEET0ZEcwz57z2yfWRds43b3TfQPPDSWmbjGmD43xRxLT41NDg@mail.gmail.com>
> > > Hi Kyotaro, ignoring the MakePGDirectory() failure will fix this
> database
> > > create redo error, but I suspect some other kind of redo, which
> depends on
> > > the files under the directory (they are not copied since the directory
> is
> > > not created) and also cannot be covered by the invalid page mechanism,
> > > could fail. Thanks.
> >
> > If recovery starts from just after tablespace creation, that's
> > simple. The Symlink to the removed tablespace is already removed
> > in the case. Hence server innocently create files directly under
> > pg_tblspc, not in the tablespace. Finally all files that were
> > supposed to be created in the removed tablespace are removed
> > later in recovery.
> >
> > If recovery starts from recalling page in a file that have been
> > in the tablespace, XLogReadBufferExtended creates one (perhaps
> > directly in pg_tblspc as described above) and the files are
> > removed later in recoery the same way to above. This case doen't
> > cause FATAL/PANIC during recovery even in master.
> >
> > [email protected]
> > | * Create the target file if it doesn't already exist. This lets us
> cope
> > | * if the replay sequence contains writes to a relation that is later
> > | * deleted. (The original coding of this routine would instead suppress
> > | * the writes, but that seems like it risks losing valuable data if the
> > | * filesystem loses an inode during a crash. Better to write the data
> > | * until we are actually told to delete the file.)
> >
> > So buffered access cannot be a problem for the reason above. The
> > remaining possible issue is non-buffered access to files in
> > removed tablespaces. This is what I mentioned upthread:
> >
> > me> but I haven't checked this covers all places where need the same
> > me> treatment.
>
> RM_DBASE_ID is fixed by the patch.
>
> XLOG/XACT/CLOG/MULTIXACT/RELMAP/STANDBY/COMMIT_TS/REPLORIGIN/LOGICALMSG:
> - are not relevant.
>
> HEAP/HEAP2/BTREE/HASH/GIN/GIST/SEQ/SPGIST/BRIN/GENERIC:
> - Resources works on buffer is not affected.
>
> SMGR:
> - Both CREATE and TRUNCATE seems fine.
>
> TBLSPC:
> - We don't nest tablespace directories. No Problem.
>
> I don't find a similar case.
I took some time in digging into the related code. It seems that ignoring
if the dst directory cannot be created directly
should be fine since smgr redo code creates tablespace path finally by
calling TablespaceCreateDbspace().
What's more, I found some more issues.
1) The below error message is actually misleading.
2019-04-17 14:52:14.951 CST [23030] FATAL: could not create directory
"pg_tblspc/65546/PG_12_201904072/65547": No such file or directory
2019-04-17 14:52:14.951 CST [23030] CONTEXT: WAL redo at 0/3011650 for
Database/CREATE: copy dir 1663/1 to 65546/65547
That should be due to dbase_desc(). It could be simply fixed following the
code logic in GetDatabasePath().
2) It seems that src directory could be missing then
dbase_redo()->copydir() could error out. For example,
\!rm -rf /tmp/tbspace1
\!mkdir /tmp/tbspace1
\!rm -rf /tmp/tbspace2
\!mkdir /tmp/tbspace2
create tablespace tbs1 location '/tmp/tbspace1';
create tablespace tbs2 location '/tmp/tbspace2';
create database db1 tablespace tbs1;
alter database db1 set tablespace tbs2;
drop tablespace tbs1;
Let's say, the standby finishes all replay but redo lsn on pg_control is
still the point at 'alter database', and then
kill postgres, then in theory when startup, dbase_redo()->copydir() will
ERROR since 'drop tablespace tbs1'
has removed the directories (and symlink) of tbs1. Below simple code change
could fix that.
diff --git a/src/backend/commands/dbcommands.c
b/src/backend/commands/dbcommands.c
index 9707afabd9..7d755c759e 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -2114,6 +2114,15 @@ dbase_redo(XLogReaderState *record)
*/
FlushDatabaseBuffers(xlrec->src_db_id);
+ /*
+ * It is possible that the source directory is missing if
+ * we are re-replaying the xlog while subsequent xlogs
+ * drop the tablespace in previous replaying. For this
+ * we just skip.
+ */
+ if (!(stat(src_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ return;
+
/*
* Copy this subdirectory to the new location
*
If we want to fix the issue by ignoring the dst path create failure, I do
not think we should do
that in copydir() since copydir() seems to be a common function. I'm not
sure whether it is
used by some extensions or not. If no maybe we should move the dst patch
create logic
out of copydir().
Also I'd suggest we should use pg_mkdir_p() in TablespaceCreateDbspace() to
replace
the code block includes a lot of get_parent_directory(), MakePGDirectory(),
etc even it
is not fixing a bug since pg_mkdir_p() code change seems to be more
graceful and simpler.
Whatever ignore mkdir failure or mkdir_p, I found that these steps seem to
be error-prone
along with postgre evolving since they are hard to test and also we are not
easy to think out
various potential bad cases. Is it possible that we should do real
checkpoint (flush & update
redo lsn) when seeing checkpoint xlogs for these operations? This will slow
down standby
but master also does this and also these operations are not usual,
espeically it seems that it
does not slow down wal receiving usually?
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
@ 2019-04-30 06:33 ` Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Paul Guo @ 2019-04-30 06:33 UTC (permalink / raw)
To: Kyotaro HORIGUCHI <[email protected]>; +Cc: Asim Praveen <[email protected]>; pgsql-hackers
I updated the original patch to
1) skip copydir() if either src path or dst parent path is missing in
dbase_redo(). Both missing cases seem to be possible. For the src path
missing case, mkdir_p() is meaningless. It seems that moving the directory
existence check step to dbase_redo() has less impact on other code.
2) Fixed dbase_desc(). Now the xlog output looks correct.
rmgr: Database len (rec/tot): 42/ 42, tx: 486, lsn:
0/016386A8, prev 0/01638630, desc: CREATE copy dir base/1 to
pg_tblspc/16384/PG_12_201904281/16386
rmgr: Database len (rec/tot): 34/ 34, tx: 487, lsn:
0/01638EB8, prev 0/01638E40, desc: DROP dir
pg_tblspc/16384/PG_12_201904281/16386
I'm not familiar with the TAP test details previously. I learned a lot
about how to test such case from Kyotaro's patch series.👍
On Sun, Apr 28, 2019 at 3:33 PM Paul Guo <[email protected]> wrote:
>
> On Wed, Apr 24, 2019 at 4:14 PM Kyotaro HORIGUCHI <
> [email protected]> wrote:
>
>> Mmm. I posted to wrong thread. Sorry.
>>
>> At Tue, 23 Apr 2019 16:39:49 +0900 (Tokyo Standard Time), Kyotaro
>> HORIGUCHI <[email protected]> wrote in <
>> [email protected]>
>> > At Tue, 23 Apr 2019 13:31:58 +0800, Paul Guo <[email protected]> wrote
>> in <CAEET0ZEcwz57z2yfWRds43b3TfQPPDSWmbjGmD43xRxLT41NDg@mail.gmail.com>
>> > > Hi Kyotaro, ignoring the MakePGDirectory() failure will fix this
>> database
>> > > create redo error, but I suspect some other kind of redo, which
>> depends on
>> > > the files under the directory (they are not copied since the
>> directory is
>> > > not created) and also cannot be covered by the invalid page mechanism,
>> > > could fail. Thanks.
>> >
>> > If recovery starts from just after tablespace creation, that's
>> > simple. The Symlink to the removed tablespace is already removed
>> > in the case. Hence server innocently create files directly under
>> > pg_tblspc, not in the tablespace. Finally all files that were
>> > supposed to be created in the removed tablespace are removed
>> > later in recovery.
>> >
>> > If recovery starts from recalling page in a file that have been
>> > in the tablespace, XLogReadBufferExtended creates one (perhaps
>> > directly in pg_tblspc as described above) and the files are
>> > removed later in recoery the same way to above. This case doen't
>> > cause FATAL/PANIC during recovery even in master.
>> >
>> > [email protected]
>> > | * Create the target file if it doesn't already exist. This lets us
>> cope
>> > | * if the replay sequence contains writes to a relation that is later
>> > | * deleted. (The original coding of this routine would instead
>> suppress
>> > | * the writes, but that seems like it risks losing valuable data if the
>> > | * filesystem loses an inode during a crash. Better to write the data
>> > | * until we are actually told to delete the file.)
>> >
>> > So buffered access cannot be a problem for the reason above. The
>> > remaining possible issue is non-buffered access to files in
>> > removed tablespaces. This is what I mentioned upthread:
>> >
>> > me> but I haven't checked this covers all places where need the same
>> > me> treatment.
>>
>> RM_DBASE_ID is fixed by the patch.
>>
>> XLOG/XACT/CLOG/MULTIXACT/RELMAP/STANDBY/COMMIT_TS/REPLORIGIN/LOGICALMSG:
>> - are not relevant.
>>
>> HEAP/HEAP2/BTREE/HASH/GIN/GIST/SEQ/SPGIST/BRIN/GENERIC:
>> - Resources works on buffer is not affected.
>>
>> SMGR:
>> - Both CREATE and TRUNCATE seems fine.
>>
>> TBLSPC:
>> - We don't nest tablespace directories. No Problem.
>>
>> I don't find a similar case.
>
>
> I took some time in digging into the related code. It seems that ignoring
> if the dst directory cannot be created directly
> should be fine since smgr redo code creates tablespace path finally by
> calling TablespaceCreateDbspace().
> What's more, I found some more issues.
>
> 1) The below error message is actually misleading.
>
> 2019-04-17 14:52:14.951 CST [23030] FATAL: could not create directory
> "pg_tblspc/65546/PG_12_201904072/65547": No such file or directory
> 2019-04-17 14:52:14.951 CST [23030] CONTEXT: WAL redo at 0/3011650 for
> Database/CREATE: copy dir 1663/1 to 65546/65547
>
> That should be due to dbase_desc(). It could be simply fixed following the
> code logic in GetDatabasePath().
>
> 2) It seems that src directory could be missing then
> dbase_redo()->copydir() could error out. For example,
>
> \!rm -rf /tmp/tbspace1
> \!mkdir /tmp/tbspace1
> \!rm -rf /tmp/tbspace2
> \!mkdir /tmp/tbspace2
> create tablespace tbs1 location '/tmp/tbspace1';
> create tablespace tbs2 location '/tmp/tbspace2';
> create database db1 tablespace tbs1;
> alter database db1 set tablespace tbs2;
> drop tablespace tbs1;
>
> Let's say, the standby finishes all replay but redo lsn on pg_control is
> still the point at 'alter database', and then
> kill postgres, then in theory when startup, dbase_redo()->copydir() will
> ERROR since 'drop tablespace tbs1'
> has removed the directories (and symlink) of tbs1. Below simple code
> change could fix that.
>
> diff --git a/src/backend/commands/dbcommands.c
> b/src/backend/commands/dbcommands.c
> index 9707afabd9..7d755c759e 100644
> --- a/src/backend/commands/dbcommands.c
> +++ b/src/backend/commands/dbcommands.c
> @@ -2114,6 +2114,15 @@ dbase_redo(XLogReaderState *record)
> */
> FlushDatabaseBuffers(xlrec->src_db_id);
>
> + /*
> + * It is possible that the source directory is missing if
> + * we are re-replaying the xlog while subsequent xlogs
> + * drop the tablespace in previous replaying. For this
> + * we just skip.
> + */
> + if (!(stat(src_path, &st) == 0 && S_ISDIR(st.st_mode)))
> + return;
> +
> /*
> * Copy this subdirectory to the new location
> *
>
> If we want to fix the issue by ignoring the dst path create failure, I do
> not think we should do
> that in copydir() since copydir() seems to be a common function. I'm not
> sure whether it is
> used by some extensions or not. If no maybe we should move the dst patch
> create logic
> out of copydir().
>
> Also I'd suggest we should use pg_mkdir_p() in TablespaceCreateDbspace()
> to replace
> the code block includes a lot of
> get_parent_directory(), MakePGDirectory(), etc even it
> is not fixing a bug since pg_mkdir_p() code change seems to be more
> graceful and simpler.
>
> Whatever ignore mkdir failure or mkdir_p, I found that these steps seem to
> be error-prone
> along with postgre evolving since they are hard to test and also we are
> not easy to think out
> various potential bad cases. Is it possible that we should do real
> checkpoint (flush & update
> redo lsn) when seeing checkpoint xlogs for these operations? This will
> slow down standby
> but master also does this and also these operations are not usual,
> espeically it seems that it
> does not slow down wal receiving usually?
>
>
>
>
Attachments:
[application/octet-stream] v2-0001-skip-copydir-if-either-src-directory-or-dst-direc.patch (5.7K, ../../CAEET0ZGhmDKrq7JJu2rLLqcJBR8pA4OYrKsirZ5Ft8-deG1e8A@mail.gmail.com/3-v2-0001-skip-copydir-if-either-src-directory-or-dst-direc.patch)
download | inline diff:
From 650221d14c1c34023aa66b1c22398eeae00dbcb8 Mon Sep 17 00:00:00 2001
From: Paul Guo <[email protected]>
Date: Tue, 30 Apr 2019 13:30:49 +0800
Subject: [PATCH v2] skip copydir() if either src directory or dst directory is
missing due to re-redoing create database but the tablespace is dropped.
---
src/backend/access/rmgrdesc/dbasedesc.c | 14 ++++++----
src/backend/commands/dbcommands.c | 35 ++++++++++++++++++++++++-
src/backend/commands/tablespace.c | 28 +-------------------
3 files changed, 44 insertions(+), 33 deletions(-)
diff --git a/src/backend/access/rmgrdesc/dbasedesc.c b/src/backend/access/rmgrdesc/dbasedesc.c
index c7d60ce10d..35092ffb0e 100644
--- a/src/backend/access/rmgrdesc/dbasedesc.c
+++ b/src/backend/access/rmgrdesc/dbasedesc.c
@@ -23,21 +23,25 @@ dbase_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ char *dbpath1, *dbpath2;
if (info == XLOG_DBASE_CREATE)
{
xl_dbase_create_rec *xlrec = (xl_dbase_create_rec *) rec;
- appendStringInfo(buf, "copy dir %u/%u to %u/%u",
- xlrec->src_tablespace_id, xlrec->src_db_id,
- xlrec->tablespace_id, xlrec->db_id);
+ dbpath1 = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
+ dbpath2 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
+ appendStringInfo(buf, "copy dir %s to %s", dbpath1, dbpath2);
+ pfree(dbpath2);
+ pfree(dbpath1);
}
else if (info == XLOG_DBASE_DROP)
{
xl_dbase_drop_rec *xlrec = (xl_dbase_drop_rec *) rec;
- appendStringInfo(buf, "dir %u/%u",
- xlrec->tablespace_id, xlrec->db_id);
+ dbpath1 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
+ appendStringInfo(buf, "dir %s", dbpath1);
+ pfree(dbpath1);
}
}
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index 9707afabd9..b7943529be 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -45,6 +45,7 @@
#include "commands/defrem.h"
#include "commands/seclabel.h"
#include "commands/tablespace.h"
+#include "common/file_perm.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "pgstat.h"
@@ -2089,7 +2090,9 @@ dbase_redo(XLogReaderState *record)
xl_dbase_create_rec *xlrec = (xl_dbase_create_rec *) XLogRecGetData(record);
char *src_path;
char *dst_path;
+ char *parent_path;
struct stat st;
+ bool do_copydir = true;
src_path = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
dst_path = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
@@ -2107,6 +2110,35 @@ dbase_redo(XLogReaderState *record)
(errmsg("some useless files may be left behind in old database directory \"%s\"",
dst_path)));
}
+ else
+ {
+ /*
+ * It is possible that the tablespace was later dropped, but we are
+ * re-redoing database create before that. In that case,
+ * the directory are missing, we simply skip the copydir step.
+ */
+ parent_path = pstrdup(dst_path);
+ get_parent_directory(parent_path);
+ if (!(stat(parent_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
+ do_copydir = false;
+ ereport(WARNING,
+ (errmsg("directory \"%s\" for copydir() does not exists."
+ "It is possibly expected. Skip copydir().",
+ parent_path)));
+ }
+ pfree(parent_path);
+ }
+
+ /* src directory is possibly missing also. See previous comment. */
+ if (!(stat(src_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
+ do_copydir = false;
+ ereport(WARNING,
+ (errmsg("source directory \"%s\" for copydir() does not exists."
+ "It is possibly expected. Skip copydir().",
+ src_path)));
+ }
/*
* Force dirty buffers out to disk, to ensure source database is
@@ -2119,7 +2151,8 @@ dbase_redo(XLogReaderState *record)
*
* We don't need to copy subdirectories
*/
- copydir(src_path, dst_path, false);
+ if (do_copydir)
+ copydir(src_path, dst_path, false);
}
else if (info == XLOG_DBASE_DROP)
{
diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c
index 8ec963f1cf..798c4586b8 100644
--- a/src/backend/commands/tablespace.c
+++ b/src/backend/commands/tablespace.c
@@ -154,8 +154,6 @@ TablespaceCreateDbspace(Oid spcNode, Oid dbNode, bool isRedo)
/* Directory creation failed? */
if (MakePGDirectory(dir) < 0)
{
- char *parentdir;
-
/* Failure other than not exists or not in WAL replay? */
if (errno != ENOENT || !isRedo)
ereport(ERROR,
@@ -168,32 +166,8 @@ TablespaceCreateDbspace(Oid spcNode, Oid dbNode, bool isRedo)
* continue by creating simple parent directories rather
* than a symlink.
*/
-
- /* create two parents up if not exist */
- parentdir = pstrdup(dir);
- get_parent_directory(parentdir);
- get_parent_directory(parentdir);
- /* Can't create parent and it doesn't already exist? */
- if (MakePGDirectory(parentdir) < 0 && errno != EEXIST)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not create directory \"%s\": %m",
- parentdir)));
- pfree(parentdir);
-
- /* create one parent up if not exist */
- parentdir = pstrdup(dir);
- get_parent_directory(parentdir);
- /* Can't create parent and it doesn't already exist? */
- if (MakePGDirectory(parentdir) < 0 && errno != EEXIST)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not create directory \"%s\": %m",
- parentdir)));
- pfree(parentdir);
-
/* Create database directory */
- if (MakePGDirectory(dir) < 0)
+ if (pg_mkdir_p(dir, pg_dir_create_mode) < 0)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not create directory \"%s\": %m",
--
2.17.2
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
@ 2019-05-07 06:47 ` Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Kyotaro HORIGUCHI @ 2019-05-07 06:47 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; pgsql-hackers
Hi.
At Tue, 30 Apr 2019 14:33:47 +0800, Paul Guo <[email protected]> wrote in <CAEET0ZGhmDKrq7JJu2rLLqcJBR8pA4OYrKsirZ5Ft8-deG1e8A@mail.gmail.com>
> I updated the original patch to
It's reasonable not to touch copydir.
> 1) skip copydir() if either src path or dst parent path is missing in
> dbase_redo(). Both missing cases seem to be possible. For the src path
> missing case, mkdir_p() is meaningless. It seems that moving the directory
> existence check step to dbase_redo() has less impact on other code.
Nice catch.
+ if (!(stat(parent_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
This patch is allowing missing source and destination directory
even in consistent state. I don't think it is safe.
+ ereport(WARNING,
+ (errmsg("directory \"%s\" for copydir() does not exists."
+ "It is possibly expected. Skip copydir().",
+ parent_path)));
This message seems unfriendly to users, or it seems like an elog
message. How about something like this. The same can be said for
the source directory.
| WARNING: skipped creating database directory: "%s"
| DETAIL: The tabelspace %u may have been removed just before crash.
# I'm not confident in this at all:(
> 2) Fixed dbase_desc(). Now the xlog output looks correct.
>
> rmgr: Database len (rec/tot): 42/ 42, tx: 486, lsn:
> 0/016386A8, prev 0/01638630, desc: CREATE copy dir base/1 to
> pg_tblspc/16384/PG_12_201904281/16386
>
> rmgr: Database len (rec/tot): 34/ 34, tx: 487, lsn:
> 0/01638EB8, prev 0/01638E40, desc: DROP dir
> pg_tblspc/16384/PG_12_201904281/16386
WAL records don't convey such information. The previous
description seems right to me.
> I'm not familiar with the TAP test details previously. I learned a lot
> about how to test such case from Kyotaro's patch series.👍
Yeah, good to hear.
> On Sun, Apr 28, 2019 at 3:33 PM Paul Guo <[email protected]> wrote:
> > If we want to fix the issue by ignoring the dst path create failure, I do
> > not think we should do
> > that in copydir() since copydir() seems to be a common function. I'm not
> > sure whether it is
> > used by some extensions or not. If no maybe we should move the dst patch
> > create logic
> > out of copydir().
Agreed to this.
> > Also I'd suggest we should use pg_mkdir_p() in TablespaceCreateDbspace()
> > to replace
> > the code block includes a lot of
> > get_parent_directory(), MakePGDirectory(), etc even it
> > is not fixing a bug since pg_mkdir_p() code change seems to be more
> > graceful and simpler.
But I don't agree to this. pg_mkdir_p goes above two-parents up,
which would be unwanted here.
> > Whatever ignore mkdir failure or mkdir_p, I found that these steps seem to
> > be error-prone
> > along with postgre evolving since they are hard to test and also we are
> > not easy to think out
> > various potential bad cases. Is it possible that we should do real
> > checkpoint (flush & update
> > redo lsn) when seeing checkpoint xlogs for these operations? This will
> > slow down standby
> > but master also does this and also these operations are not usual,
> > espeically it seems that it
> > does not slow down wal receiving usually?
That dramatically slows recovery (not replication) if databases
are created and deleted frequently. That wouldn't be acceptable.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
@ 2019-05-13 09:37 ` Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Paul Guo @ 2019-05-13 09:37 UTC (permalink / raw)
To: Kyotaro HORIGUCHI <[email protected]>; +Cc: pgsql-hackers
Thanks for the reply.
On Tue, May 7, 2019 at 2:47 PM Kyotaro HORIGUCHI <
[email protected]> wrote:
>
> + if (!(stat(parent_path, &st) == 0 && S_ISDIR(st.st_mode)))
> + {
>
> This patch is allowing missing source and destination directory
> even in consistent state. I don't think it is safe.
>
I do not understand this. Can you elaborate?
>
>
>
> + ereport(WARNING,
> + (errmsg("directory \"%s\" for copydir() does not exists."
> + "It is possibly expected. Skip copydir().",
> + parent_path)));
>
> This message seems unfriendly to users, or it seems like an elog
> message. How about something like this. The same can be said for
> the source directory.
>
> | WARNING: skipped creating database directory: "%s"
> | DETAIL: The tabelspace %u may have been removed just before crash.
>
Yeah. Looks better.
>
> # I'm not confident in this at all:(
>
> > 2) Fixed dbase_desc(). Now the xlog output looks correct.
> >
> > rmgr: Database len (rec/tot): 42/ 42, tx: 486, lsn:
> > 0/016386A8, prev 0/01638630, desc: CREATE copy dir base/1 to
> > pg_tblspc/16384/PG_12_201904281/16386
> >
> > rmgr: Database len (rec/tot): 34/ 34, tx: 487, lsn:
> > 0/01638EB8, prev 0/01638E40, desc: DROP dir
> > pg_tblspc/16384/PG_12_201904281/16386
>
> WAL records don't convey such information. The previous
> description seems right to me.
>
2019-04-17 14:52:14.951 CST [23030] CONTEXT: WAL redo at 0/3011650 for
Database/CREATE: copy dir 1663/1 to 65546/65547
The directories are definitely wrong and misleading.
> > > Also I'd suggest we should use pg_mkdir_p() in
> TablespaceCreateDbspace()
> > > to replace
> > > the code block includes a lot of
> > > get_parent_directory(), MakePGDirectory(), etc even it
> > > is not fixing a bug since pg_mkdir_p() code change seems to be more
> > > graceful and simpler.
>
> But I don't agree to this. pg_mkdir_p goes above two-parents up,
> which would be unwanted here.
>
> I do not understand this also. pg_mkdir_p() is similar to 'mkdir -p'.
This change just makes the code concise. Though in theory the change is not
needed.
> > > Whatever ignore mkdir failure or mkdir_p, I found that these steps
> seem to
> > > be error-prone
> > > along with postgre evolving since they are hard to test and also we are
> > > not easy to think out
> > > various potential bad cases. Is it possible that we should do real
> > > checkpoint (flush & update
> > > redo lsn) when seeing checkpoint xlogs for these operations? This will
> > > slow down standby
> > > but master also does this and also these operations are not usual,
> > > espeically it seems that it
> > > does not slow down wal receiving usually?
>
> That dramatically slows recovery (not replication) if databases
> are created and deleted frequently. That wouldn't be acceptable.
>
This behavior is rare and seems to have the same impact on master & standby
from checkpoint/restartpoint.
We do not worry about master so we should not worry about standby also.
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
@ 2019-05-14 03:06 ` Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Kyotaro HORIGUCHI @ 2019-05-14 03:06 UTC (permalink / raw)
To: [email protected]; +Cc: pgsql-hackers
Hello.
At Mon, 13 May 2019 17:37:50 +0800, Paul Guo <[email protected]> wrote in <CAEET0ZF9yN4DaXyuFLzOcAYyxuFF1Ms_OQWeA+Rwv3GhA5Q-SA@mail.gmail.com>
> Thanks for the reply.
>
> On Tue, May 7, 2019 at 2:47 PM Kyotaro HORIGUCHI <
> [email protected]> wrote:
>
> >
> > + if (!(stat(parent_path, &st) == 0 && S_ISDIR(st.st_mode)))
> > + {
> >
> > This patch is allowing missing source and destination directory
> > even in consistent state. I don't think it is safe.
> >
>
> I do not understand this. Can you elaborate?
Suppose we were recoverying based on a backup at LSN1 targeting
to LSN3 then it crashed at LSN2, where LSN1 < LSN2 <= LSN3. LSN2
is called as "consistency point", before where the database is
not consistent. It's because we are applying WAL recored older
than those that were already applied in the second trial. The
same can be said for crash recovery, where LSN1 is the latest
checkpoint ('s redo LSN) and LSN2=LSN3 is the crashed LSN.
Creation of an existing directory or dropping of a non-existent
directory are apparently inconsistent or "broken" so we should
stop recovery when seeing such WAL records while database is in
consistent state.
> > + ereport(WARNING,
> > + (errmsg("directory \"%s\" for copydir() does not exists."
> > + "It is possibly expected. Skip copydir().",
> > + parent_path)));
> >
> > This message seems unfriendly to users, or it seems like an elog
> > message. How about something like this. The same can be said for
> > the source directory.
> >
> > | WARNING: skipped creating database directory: "%s"
> > | DETAIL: The tabelspace %u may have been removed just before crash.
> >
>
> Yeah. Looks better.
>
>
> >
> > # I'm not confident in this at all:(
> >
> > > 2) Fixed dbase_desc(). Now the xlog output looks correct.
> > >
> > > rmgr: Database len (rec/tot): 42/ 42, tx: 486, lsn:
> > > 0/016386A8, prev 0/01638630, desc: CREATE copy dir base/1 to
> > > pg_tblspc/16384/PG_12_201904281/16386
> > >
> > > rmgr: Database len (rec/tot): 34/ 34, tx: 487, lsn:
> > > 0/01638EB8, prev 0/01638E40, desc: DROP dir
> > > pg_tblspc/16384/PG_12_201904281/16386
> >
> > WAL records don't convey such information. The previous
> > description seems right to me.
> >
>
> 2019-04-17 14:52:14.951 CST [23030] CONTEXT: WAL redo at 0/3011650 for
> Database/CREATE: copy dir 1663/1 to 65546/65547
> The directories are definitely wrong and misleading.
The original description is right in the light of how the server
recognizes. The record exactly says that "copy dir 1663/1 to
65546/65547" and the latter path is converted in filesystem layer
via a symlink.
> > > > Also I'd suggest we should use pg_mkdir_p() in
> > TablespaceCreateDbspace()
> > > > to replace
> > > > the code block includes a lot of
> > > > get_parent_directory(), MakePGDirectory(), etc even it
> > > > is not fixing a bug since pg_mkdir_p() code change seems to be more
> > > > graceful and simpler.
> >
> > But I don't agree to this. pg_mkdir_p goes above two-parents up,
> > which would be unwanted here.
> >
> > I do not understand this also. pg_mkdir_p() is similar to 'mkdir -p'.
> This change just makes the code concise. Though in theory the change is not
> needed.
We don't want to create tablespace direcotory after concurrent
DROPing, as the comment just above is saying:
| * Acquire TablespaceCreateLock to ensure that no DROP TABLESPACE
| * or TablespaceCreateDbspace is running concurrently.
If the concurrent DROP TABLESPACE destroyed the grand parent
directory, we mustn't create it again.
> > > > Whatever ignore mkdir failure or mkdir_p, I found that these steps
> > seem to
> > > > be error-prone
> > > > along with postgre evolving since they are hard to test and also we are
> > > > not easy to think out
> > > > various potential bad cases. Is it possible that we should do real
> > > > checkpoint (flush & update
> > > > redo lsn) when seeing checkpoint xlogs for these operations? This will
> > > > slow down standby
> > > > but master also does this and also these operations are not usual,
> > > > espeically it seems that it
> > > > does not slow down wal receiving usually?
> >
> > That dramatically slows recovery (not replication) if databases
> > are created and deleted frequently. That wouldn't be acceptable.
> >
>
> This behavior is rare and seems to have the same impact on master & standby
> from checkpoint/restartpoint.
> We do not worry about master so we should not worry about standby also.
I didn't mention replication. I said that that slows recovery,
which is not governed by master's speed.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
@ 2019-05-27 13:39 ` Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Paul Guo @ 2019-05-27 13:39 UTC (permalink / raw)
To: Kyotaro HORIGUCHI <[email protected]>; +Cc: pgsql-hackers
On Tue, May 14, 2019 at 11:06 AM Kyotaro HORIGUCHI <
[email protected]> wrote:
> Hello.
>
> At Mon, 13 May 2019 17:37:50 +0800, Paul Guo <[email protected]> wrote in <
> CAEET0ZF9yN4DaXyuFLzOcAYyxuFF1Ms_OQWeA+Rwv3GhA5Q-SA@mail.gmail.com>
> > Thanks for the reply.
> >
> > On Tue, May 7, 2019 at 2:47 PM Kyotaro HORIGUCHI <
> > [email protected]> wrote:
> >
> > >
> > > + if (!(stat(parent_path, &st) == 0 && S_ISDIR(st.st_mode)))
> > > + {
> > >
> > > This patch is allowing missing source and destination directory
> > > even in consistent state. I don't think it is safe.
> > >
> >
> > I do not understand this. Can you elaborate?
>
> Suppose we were recoverying based on a backup at LSN1 targeting
> to LSN3 then it crashed at LSN2, where LSN1 < LSN2 <= LSN3. LSN2
> is called as "consistency point", before where the database is
> not consistent. It's because we are applying WAL recored older
> than those that were already applied in the second trial. The
> same can be said for crash recovery, where LSN1 is the latest
> checkpoint ('s redo LSN) and LSN2=LSN3 is the crashed LSN.
>
> Creation of an existing directory or dropping of a non-existent
> directory are apparently inconsistent or "broken" so we should
> stop recovery when seeing such WAL records while database is in
> consistent state.
>
This seems to be hard to detect. I thought using invalid_page mechanism
long ago,
but it seems to be hard to fully detect a dropped tablespace.
> > > 2) Fixed dbase_desc(). Now the xlog output looks correct.
> > > >
> > > > rmgr: Database len (rec/tot): 42/ 42, tx: 486, lsn:
> > > > 0/016386A8, prev 0/01638630, desc: CREATE copy dir base/1 to
> > > > pg_tblspc/16384/PG_12_201904281/16386
> > > >
> > > > rmgr: Database len (rec/tot): 34/ 34, tx: 487, lsn:
> > > > 0/01638EB8, prev 0/01638E40, desc: DROP dir
> > > > pg_tblspc/16384/PG_12_201904281/16386
> > >
> > > WAL records don't convey such information. The previous
> > > description seems right to me.
> > >
> >
> > 2019-04-17 14:52:14.951 CST [23030] CONTEXT: WAL redo at 0/3011650 for
> > Database/CREATE: copy dir 1663/1 to 65546/65547
> > The directories are definitely wrong and misleading.
>
> The original description is right in the light of how the server
> recognizes. The record exactly says that "copy dir 1663/1 to
> 65546/65547" and the latter path is converted in filesystem layer
> via a symlink.
>
In either $PG_DATA/pg_tblspc or symlinked real tablespace directory,
there is an additional directory like PG_12_201905221 between
tablespace oid and database oid. See the directory layout as below,
so the directory info in xlog dump output was not correct.
$ ls -lh data/pg_tblspc/
total 0
lrwxrwxrwx. 1 gpadmin gpadmin 6 May 27 17:23 16384 -> /tmp/2
$ ls -lh /tmp/2
total 0
drwx------. 3 gpadmin gpadmin 18 May 27 17:24 PG_12_201905221
>
>
> > > > > Also I'd suggest we should use pg_mkdir_p() in
> > > TablespaceCreateDbspace()
> > > > > to replace
> > > > > the code block includes a lot of
> > > > > get_parent_directory(), MakePGDirectory(), etc even it
> > > > > is not fixing a bug since pg_mkdir_p() code change seems to be more
> > > > > graceful and simpler.
> > >
> > > But I don't agree to this. pg_mkdir_p goes above two-parents up,
> > > which would be unwanted here.
> > >
> > > I do not understand this also. pg_mkdir_p() is similar to 'mkdir -p'.
> > This change just makes the code concise. Though in theory the change is
> not
> > needed.
>
> We don't want to create tablespace direcotory after concurrent
> DROPing, as the comment just above is saying:
>
> | * Acquire TablespaceCreateLock to ensure that no DROP TABLESPACE
> | * or TablespaceCreateDbspace is running concurrently.
>
> If the concurrent DROP TABLESPACE destroyed the grand parent
> directory, we mustn't create it again.
>
Yes, this is a good reason to keep the original code. Thanks.
By the way, based on your previous test patch I added another test which
could easily detect
the missing src directory case.
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
@ 2019-06-19 07:21 ` Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Paul Guo @ 2019-06-19 07:21 UTC (permalink / raw)
To: Kyotaro HORIGUCHI <[email protected]>; +Cc: pgsql-hackers
On Mon, May 27, 2019 at 9:39 PM Paul Guo <[email protected]> wrote:
>
>
> On Tue, May 14, 2019 at 11:06 AM Kyotaro HORIGUCHI <
> [email protected]> wrote:
>
>> Hello.
>>
>> At Mon, 13 May 2019 17:37:50 +0800, Paul Guo <[email protected]> wrote in <
>> CAEET0ZF9yN4DaXyuFLzOcAYyxuFF1Ms_OQWeA+Rwv3GhA5Q-SA@mail.gmail.com>
>> > Thanks for the reply.
>> >
>> > On Tue, May 7, 2019 at 2:47 PM Kyotaro HORIGUCHI <
>> > [email protected]> wrote:
>> >
>> > >
>> > > + if (!(stat(parent_path, &st) == 0 && S_ISDIR(st.st_mode)))
>> > > + {
>> > >
>> > > This patch is allowing missing source and destination directory
>> > > even in consistent state. I don't think it is safe.
>> > >
>> >
>> > I do not understand this. Can you elaborate?
>>
>> Suppose we were recoverying based on a backup at LSN1 targeting
>> to LSN3 then it crashed at LSN2, where LSN1 < LSN2 <= LSN3. LSN2
>> is called as "consistency point", before where the database is
>> not consistent. It's because we are applying WAL recored older
>> than those that were already applied in the second trial. The
>> same can be said for crash recovery, where LSN1 is the latest
>> checkpoint ('s redo LSN) and LSN2=LSN3 is the crashed LSN.
>>
>> Creation of an existing directory or dropping of a non-existent
>> directory are apparently inconsistent or "broken" so we should
>> stop recovery when seeing such WAL records while database is in
>> consistent state.
>>
>
> This seems to be hard to detect. I thought using invalid_page mechanism
> long ago,
> but it seems to be hard to fully detect a dropped tablespace.
>
> > > > 2) Fixed dbase_desc(). Now the xlog output looks correct.
>> > > >
>> > > > rmgr: Database len (rec/tot): 42/ 42, tx: 486, lsn:
>> > > > 0/016386A8, prev 0/01638630, desc: CREATE copy dir base/1 to
>> > > > pg_tblspc/16384/PG_12_201904281/16386
>> > > >
>> > > > rmgr: Database len (rec/tot): 34/ 34, tx: 487, lsn:
>> > > > 0/01638EB8, prev 0/01638E40, desc: DROP dir
>> > > > pg_tblspc/16384/PG_12_201904281/16386
>> > >
>> > > WAL records don't convey such information. The previous
>> > > description seems right to me.
>> > >
>> >
>> > 2019-04-17 14:52:14.951 CST [23030] CONTEXT: WAL redo at 0/3011650 for
>> > Database/CREATE: copy dir 1663/1 to 65546/65547
>> > The directories are definitely wrong and misleading.
>>
>> The original description is right in the light of how the server
>> recognizes. The record exactly says that "copy dir 1663/1 to
>> 65546/65547" and the latter path is converted in filesystem layer
>> via a symlink.
>>
>
> In either $PG_DATA/pg_tblspc or symlinked real tablespace directory,
> there is an additional directory like PG_12_201905221 between
> tablespace oid and database oid. See the directory layout as below,
> so the directory info in xlog dump output was not correct.
>
> $ ls -lh data/pg_tblspc/
>
>
> total 0
>
>
> lrwxrwxrwx. 1 gpadmin gpadmin 6 May 27 17:23 16384 -> /tmp/2
>
>
> $ ls -lh /tmp/2
>
>
> total 0
>
>
> drwx------. 3 gpadmin gpadmin 18 May 27 17:24 PG_12_201905221
>
>>
>>
>> > > > > Also I'd suggest we should use pg_mkdir_p() in
>> > > TablespaceCreateDbspace()
>> > > > > to replace
>> > > > > the code block includes a lot of
>> > > > > get_parent_directory(), MakePGDirectory(), etc even it
>> > > > > is not fixing a bug since pg_mkdir_p() code change seems to be
>> more
>> > > > > graceful and simpler.
>> > >
>> > > But I don't agree to this. pg_mkdir_p goes above two-parents up,
>> > > which would be unwanted here.
>> > >
>> > > I do not understand this also. pg_mkdir_p() is similar to 'mkdir -p'.
>> > This change just makes the code concise. Though in theory the change is
>> not
>> > needed.
>>
>> We don't want to create tablespace direcotory after concurrent
>> DROPing, as the comment just above is saying:
>>
>> | * Acquire TablespaceCreateLock to ensure that no DROP TABLESPACE
>> | * or TablespaceCreateDbspace is running concurrently.
>>
>> If the concurrent DROP TABLESPACE destroyed the grand parent
>> directory, we mustn't create it again.
>>
>
> Yes, this is a good reason to keep the original code. Thanks.
>
> By the way, based on your previous test patch I added another test which
> could easily detect
> the missing src directory case.
>
>
I updated the patch to v3. In this version, we skip the error if copydir
fails due to missing src/dst directory,
but to make sure the ignoring is legal, I add a simple log/forget mechanism
(Using List) similar to the xlog invalid page
checking mechanism. Two tap tests are included. One is actually from a
previous patch by Kyotaro in this
email thread and another is added by me. In addition, dbase_desc() is fixed
to make the message accurate.
Thanks.
Attachments:
[application/octet-stream] v3-0001-skip-copydir-if-either-src-directory-or-dst-direc.patch (14.8K, ../../CAEET0ZF-0hGtkZGufHaqDzg-Ez89n=1T6raWv-03SP1FtB7wRw@mail.gmail.com/3-v3-0001-skip-copydir-if-either-src-directory-or-dst-direc.patch)
download | inline diff:
From 968adb96f7440386e5ee70f66fbd7495068a09e6 Mon Sep 17 00:00:00 2001
From: Paul Guo <[email protected]>
Date: Tue, 30 Apr 2019 13:30:49 +0800
Subject: [PATCH v3] skip copydir() if either src directory or dst directory is
missing due to re-redoing create database but the tablespace is dropped.
Also correct dbase_desc() so that related xlog description is not misleading.
Add tap tests for the previous tablespace patch.
One of the test and related change in PostgresNode.pm are actually
from community (Kyotaro HORIGUCHI). I added another test and modified
PostgresNode.pm further to suport my added test.
This patch uses the log/forget mechanism to avoid bad ignoring.
---
src/backend/access/rmgrdesc/dbasedesc.c | 14 ++-
src/backend/access/transam/xlog.c | 4 +
src/backend/commands/dbcommands.c | 107 +++++++++++++++++++++-
src/include/commands/dbcommands.h | 2 +
src/test/perl/PostgresNode.pm | 13 ++-
src/test/perl/RecursiveCopy.pm | 33 ++++++-
src/test/recovery/t/011_crash_recovery.pl | 99 +++++++++++++++++++-
7 files changed, 259 insertions(+), 13 deletions(-)
diff --git a/src/backend/access/rmgrdesc/dbasedesc.c b/src/backend/access/rmgrdesc/dbasedesc.c
index c7d60ce10d..35092ffb0e 100644
--- a/src/backend/access/rmgrdesc/dbasedesc.c
+++ b/src/backend/access/rmgrdesc/dbasedesc.c
@@ -23,21 +23,25 @@ dbase_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ char *dbpath1, *dbpath2;
if (info == XLOG_DBASE_CREATE)
{
xl_dbase_create_rec *xlrec = (xl_dbase_create_rec *) rec;
- appendStringInfo(buf, "copy dir %u/%u to %u/%u",
- xlrec->src_tablespace_id, xlrec->src_db_id,
- xlrec->tablespace_id, xlrec->db_id);
+ dbpath1 = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
+ dbpath2 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
+ appendStringInfo(buf, "copy dir %s to %s", dbpath1, dbpath2);
+ pfree(dbpath2);
+ pfree(dbpath1);
}
else if (info == XLOG_DBASE_DROP)
{
xl_dbase_drop_rec *xlrec = (xl_dbase_drop_rec *) rec;
- appendStringInfo(buf, "dir %u/%u",
- xlrec->tablespace_id, xlrec->db_id);
+ dbpath1 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
+ appendStringInfo(buf, "dir %s", dbpath1);
+ pfree(dbpath1);
}
}
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index e08320e829..5137a75efd 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -40,6 +40,7 @@
#include "catalog/pg_control.h"
#include "catalog/pg_database.h"
#include "commands/tablespace.h"
+#include "commands/dbcommands.h"
#include "common/controldata_utils.h"
#include "miscadmin.h"
#include "pgstat.h"
@@ -7842,6 +7843,9 @@ CheckRecoveryConsistency(void)
*/
XLogCheckInvalidPages();
+ /* Check whether some missing directories are unexpected. */
+ CheckMissingDirs4DbaseRedo();
+
reachedConsistency = true;
ereport(LOG,
(errmsg("consistent recovery state reached at %X/%X",
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index 15207bf75a..bc7ab3084e 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -45,6 +45,7 @@
#include "commands/defrem.h"
#include "commands/seclabel.h"
#include "commands/tablespace.h"
+#include "common/file_perm.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "pgstat.h"
@@ -92,6 +93,68 @@ static void remove_dbtablespaces(Oid db_id);
static bool check_db_file_conflict(Oid db_id);
static int errdetail_busy_db(int notherbackends, int npreparedxacts);
+static void log_missing_directory(char *dir);
+static void forget_missing_directory(char *dir);
+
+/*
+ * During XLOG replay, we may see either src directory or dst directory
+ * is missing during copying directory when creating database in dbase_redo()
+ * if the related tablespace was later dropped but we do re-redoing in
+ * recovery after abnormal shutdown. We do simply ignore copying in
+ * dbase_redo() but log those directories in memory and then sanity check
+ * the potential bug or bad user behaviors.
+ *
+ * We use List for simplicity since this should be ok for most cases - the
+ * list should be not long in usual case.
+ */
+static List *missing_dirs_dbase_redo = NIL;
+
+void
+CheckMissingDirs4DbaseRedo()
+{
+ ListCell *lc;
+
+ if (missing_dirs_dbase_redo == NIL)
+ return;
+
+ foreach(lc, missing_dirs_dbase_redo)
+ {
+ char *dir_entry = (char *) lfirst(lc);
+
+ elog(LOG, "Directory \"%s\" was missing during directory copying "
+ "when replaying 'database create'", dir_entry);
+ }
+
+ elog(PANIC, "WAL replay was wrong due to previous missing directories");
+}
+
+static void
+log_missing_directory(char *dir)
+{
+ elog(DEBUG2, "Logging missing directory for dbase_redo(): \"%s\"", dir);
+ missing_dirs_dbase_redo = lappend(missing_dirs_dbase_redo, pstrdup(dir));
+}
+
+static void
+forget_missing_directory(char *dir)
+{
+ ListCell *prev, *lc;
+
+ prev = NULL;
+ foreach(lc, missing_dirs_dbase_redo)
+ {
+ char *dir_entry = (char *) lfirst(lc);
+
+ if (strcmp(dir_entry, dir) == 0)
+ {
+ missing_dirs_dbase_redo = list_delete_cell(missing_dirs_dbase_redo, lc, prev);
+ elog(DEBUG2, "forgetting missing directory for dbase_redo(): \"%s\"", dir);
+ return;
+ }
+
+ prev = lc;
+ }
+}
/*
* CREATE DATABASE
@@ -2089,7 +2152,9 @@ dbase_redo(XLogReaderState *record)
xl_dbase_create_rec *xlrec = (xl_dbase_create_rec *) XLogRecGetData(record);
char *src_path;
char *dst_path;
+ char *parent_path;
struct stat st;
+ bool do_copydir = true;
src_path = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
dst_path = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
@@ -2107,6 +2172,43 @@ dbase_redo(XLogReaderState *record)
(errmsg("some useless files may be left behind in old database directory \"%s\"",
dst_path)));
}
+ else
+ {
+ /*
+ * It is possible that the tablespace was previously dropped, but
+ * we are re-redoing database create with that tablespace after
+ * an abnormal shutdown (e.g. immediate shutdown). In that case,
+ * the directory are missing, we simply skip the copydir step.
+ */
+ parent_path = pstrdup(dst_path);
+ get_parent_directory(parent_path);
+ if (!(stat(parent_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
+ do_copydir = false;
+ log_missing_directory(dst_path);
+ ereport(WARNING,
+ (errmsg("Skip creating database directory \"%s\". "
+ "The dest tablespace may have been removed "
+ "before abnormal shutdown. If the removal "
+ "is illegal after later checking we will panic.",
+ parent_path)));
+ }
+ pfree(parent_path);
+ }
+
+ /* src directory is possibly missing during redo also. */
+ if (!(stat(src_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
+ do_copydir = false;
+ log_missing_directory(src_path);
+ ereport(WARNING,
+ (errmsg("Skip creating database directory based on "
+ "\"%s\". The src tablespace may have been "
+ "removed before abnormal shutdown. If the removal "
+ "is illegal after later checking we will panic.",
+ src_path)));
+
+ }
/*
* Force dirty buffers out to disk, to ensure source database is
@@ -2119,7 +2221,8 @@ dbase_redo(XLogReaderState *record)
*
* We don't need to copy subdirectories
*/
- copydir(src_path, dst_path, false);
+ if (do_copydir)
+ copydir(src_path, dst_path, false);
}
else if (info == XLOG_DBASE_DROP)
{
@@ -2162,6 +2265,8 @@ dbase_redo(XLogReaderState *record)
(errmsg("some useless files may be left behind in old database directory \"%s\"",
dst_path)));
+ forget_missing_directory(dst_path);
+
if (InHotStandby)
{
/*
diff --git a/src/include/commands/dbcommands.h b/src/include/commands/dbcommands.h
index 28bf21153d..4893e0e289 100644
--- a/src/include/commands/dbcommands.h
+++ b/src/include/commands/dbcommands.h
@@ -19,6 +19,8 @@
#include "lib/stringinfo.h"
#include "nodes/parsenodes.h"
+extern void CheckMissingDirs4DbaseRedo(void);
+
extern Oid createdb(ParseState *pstate, const CreatedbStmt *stmt);
extern void dropdb(const char *dbname, bool missing_ok);
extern ObjectAddress RenameDatabase(const char *oldname, const char *newname);
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 8d5ad6bc16..e5b465dafd 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -551,13 +551,22 @@ target server since it isn't done by default.
sub backup
{
- my ($self, $backup_name) = @_;
+ my ($self, $backup_name, %params) = @_;
my $backup_path = $self->backup_dir . '/' . $backup_name;
my $name = $self->name;
+ my @rest = ();
+
+ if (defined $params{tablespace_mappings})
+ {
+ my @ts_mappings = split(/,/, $params{tablespace_mappings});
+ foreach my $elem (@ts_mappings) {
+ push(@rest, '--tablespace-mapping='.$elem);
+ }
+ }
print "# Taking pg_basebackup $backup_name from node \"$name\"\n";
TestLib::system_or_bail('pg_basebackup', '-D', $backup_path, '-h',
- $self->host, '-p', $self->port, '--no-sync');
+ $self->host, '-p', $self->port, '--no-sync', @rest);
print "# Backup finished\n";
return;
}
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
index baf5d0ac63..c912ce412d 100644
--- a/src/test/perl/RecursiveCopy.pm
+++ b/src/test/perl/RecursiveCopy.pm
@@ -22,6 +22,7 @@ use warnings;
use Carp;
use File::Basename;
use File::Copy;
+use TestLib;
=pod
@@ -97,14 +98,38 @@ sub _copypath_recurse
# invoke the filter and skip all further operation if it returns false
return 1 unless &$filterfn($curr_path);
- # Check for symlink -- needed only on source dir
- # (note: this will fall through quietly if file is already gone)
- croak "Cannot operate on symlink \"$srcpath\"" if -l $srcpath;
-
# Abort if destination path already exists. Should we allow directories
# to exist already?
croak "Destination path \"$destpath\" already exists" if -e $destpath;
+ # Check for symlink -- needed only on source dir
+ # (note: this will fall through quietly if file is already gone)
+ if (-l $srcpath)
+ {
+ croak "Cannot operate on symlink \"$srcpath\""
+ if ($srcpath !~ /\/(pg_tblspc\/[0-9]+)$/);
+
+ # We have mapped tablespaces. Copy them individually
+ my $linkname = $1;
+ my $tmpdir = TestLib::tempdir;
+ my $dstrealdir = TestLib::real_dir($tmpdir);
+ my $srcrealdir = readlink($srcpath);
+
+ opendir(my $dh, $srcrealdir);
+ while (readdir $dh)
+ {
+ next if (/^\.\.?$/);
+ my $spath = "$srcrealdir/$_";
+ my $dpath = "$dstrealdir/$_";
+
+ copypath($spath, $dpath);
+ }
+ closedir $dh;
+
+ symlink $dstrealdir, $destpath;
+ return 1;
+ }
+
# If this source path is a file, simply copy it to destination with the
# same name and we're done.
if (-f $srcpath)
diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl
index 5dc52412ca..fad6a64d2a 100644
--- a/src/test/recovery/t/011_crash_recovery.pl
+++ b/src/test/recovery/t/011_crash_recovery.pl
@@ -15,7 +15,7 @@ if ($Config{osname} eq 'MSWin32')
}
else
{
- plan tests => 3;
+ plan tests => 5;
}
my $node = get_new_node('master');
@@ -66,3 +66,100 @@ is($node->safe_psql('postgres', qq[SELECT txid_status('$xid');]),
'aborted', 'xid is aborted after crash');
$tx->kill_kill;
+
+# Ensure that tablespace removal doesn't cause error while recoverying
+# the preceding create datbase or objects.
+
+my $node_master = get_new_node('master2');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Create tablespace
+my $tspDir_master = TestLib::tempdir;
+my $realTSDir_master = TestLib::real_dir($tspDir_master);
+$node_master->safe_psql('postgres', "CREATE TABLESPACE ts1 LOCATION '$realTSDir_master'");
+
+my $tspDir_standby = TestLib::tempdir;
+my $realTSDir_standby = TestLib::real_dir($tspDir_standby);
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name,
+ tablespace_mappings =>
+ "$realTSDir_master=$realTSDir_standby");
+my $node_standby = get_new_node('standby2');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->start;
+
+# Make sure connection is made
+$node_master->poll_query_until(
+ 'postgres', 'SELECT count(*) = 1 FROM pg_stat_replication');
+
+# Make sure to perform restartpoint after tablespace creation
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->safe_psql('postgres', 'CHECKPOINT');
+
+# Do immediate shutdown just after a sequence of CREAT DATABASE / DROP
+# DATABASE / DROP TABLESPACE. This leaves a CREATE DATBASE WAL record
+# that is to be applied to already-removed tablespace.
+$node_master->safe_psql('postgres',
+ q[CREATE DATABASE db1 WITH TABLESPACE ts1;
+ DROP DATABASE db1;
+ DROP TABLESPACE ts1;]);
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->stop('immediate');
+
+# Should restart ignoring directory creation error.
+is($node_standby->start(fail_ok => 1), 1);
+
+
+# Ensure that tablespace removal doesn't cause error while recoverying
+# the preceding create datbase or objects.
+
+$node_master = get_new_node('master3');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Create tablespace
+$tspDir_master = TestLib::tempdir;
+$realTSDir_master = TestLib::real_dir($tspDir_master);
+mkdir "$realTSDir_master/1";
+mkdir "$realTSDir_master/2";
+$node_master->safe_psql('postgres', "CREATE TABLESPACE ts1 LOCATION '$realTSDir_master/1'");
+$node_master->safe_psql('postgres', "CREATE TABLESPACE ts2 LOCATION '$realTSDir_master/2'");
+
+$tspDir_standby = TestLib::tempdir;
+$realTSDir_standby = TestLib::real_dir($tspDir_standby);
+
+# Take backup
+$backup_name = 'my_backup';
+$node_master->backup($backup_name,
+ tablespace_mappings =>
+ "$realTSDir_master/1=$realTSDir_standby/1,$realTSDir_master/2=$realTSDir_standby/2");
+$node_standby = get_new_node('standby3');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->start;
+
+# Make sure connection is made
+$node_master->poll_query_until(
+ 'postgres', 'SELECT count(*) = 1 FROM pg_stat_replication');
+
+$node_master->safe_psql('postgres', "CREATE DATABASE db1 TABLESPACE ts1");
+
+# Make sure to perform restartpoint after tablespace creation
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->safe_psql('postgres', 'CHECKPOINT');
+
+# Do immediate shutdown ...
+$node_master->safe_psql('postgres',
+ q[ALTER DATABASE db1 SET TABLESPACE ts2;
+ DROP TABLESPACE ts1;]);
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->stop('immediate');
+
+# Should restart ignoring directory creation error.
+is($node_standby->start(fail_ok => 1), 1);
--
2.17.2
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
@ 2019-07-08 03:15 ` Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Thomas Munro @ 2019-07-08 03:15 UTC (permalink / raw)
To: Paul Guo <[email protected]>; +Cc: Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
On Wed, Jun 19, 2019 at 7:22 PM Paul Guo <[email protected]> wrote:
> I updated the patch to v3. In this version, we skip the error if copydir fails due to missing src/dst directory,
> but to make sure the ignoring is legal, I add a simple log/forget mechanism (Using List) similar to the xlog invalid page
> checking mechanism. Two tap tests are included. One is actually from a previous patch by Kyotaro in this
> email thread and another is added by me. In addition, dbase_desc() is fixed to make the message accurate.
Hello Paul,
FYI t/011_crash_recovery.pl is failing consistently on Travis CI with
this patch applied:
https://travis-ci.org/postgresql-cfbot/postgresql/builds/555368907
--
Thomas Munro
https://enterprisedb.com
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
@ 2019-07-15 10:52 ` Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Paul Guo @ 2019-07-15 10:52 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
On Mon, Jul 8, 2019 at 11:16 AM Thomas Munro <[email protected]> wrote:
> On Wed, Jun 19, 2019 at 7:22 PM Paul Guo <[email protected]> wrote:
> > I updated the patch to v3. In this version, we skip the error if copydir
> fails due to missing src/dst directory,
> > but to make sure the ignoring is legal, I add a simple log/forget
> mechanism (Using List) similar to the xlog invalid page
> > checking mechanism. Two tap tests are included. One is actually from a
> previous patch by Kyotaro in this
> > email thread and another is added by me. In addition, dbase_desc() is
> fixed to make the message accurate.
>
> Hello Paul,
>
> FYI t/011_crash_recovery.pl is failing consistently on Travis CI with
> this patch applied:
>
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__travis-2Dci.org_postgresql-2Dcfbot_postgresql_b...
>
>
>
This failure is because the previous v3 patch does not align with a recent
patch
commit 660a2b19038b2f6b9f6bcb2c3297a47d5e3557a8
Author: Noah Misch <[email protected]>
Date: Fri Jun 21 20:34:23 2019 -0700
Consolidate methods for translating a Perl path to a Windows path.
My patch uses TestLib::real_dir which is now replaced
with TestLib::perl2host in the above commit.
I've updated the patch to v4 to make my code align. Now the test passes in
my local environment.
Please see the attached v4 patch.
Thanks.
Attachments:
[application/octet-stream] v4-0001-skip-copydir-if-either-src-directory-or-dst-direc.patch (14.8K, ../../CAEET0ZFNE2-NQbaw70=EiuOw3JXSSQ_nkbbcXP91pY81vNE1zA@mail.gmail.com/3-v4-0001-skip-copydir-if-either-src-directory-or-dst-direc.patch)
download | inline diff:
From 7339a82089d2f363a55cb651d3651a2c01f2af7d Mon Sep 17 00:00:00 2001
From: Paul Guo <[email protected]>
Date: Tue, 30 Apr 2019 13:30:49 +0800
Subject: [PATCH v4] skip copydir() if either src directory or dst directory is
missing due to re-redoing create database but the tablespace is dropped.
Also correct dbase_desc() so that related xlog description is not misleading.
This patch uses the log/forget mechanism to avoid bad ignoring.
Kyotaro horiguchi added one test and did related change in PostgresNode.pm and
had a lot of discussion on this issue. I further added another test and
modified PostgresNode.pm again to support my new test.
---
src/backend/access/rmgrdesc/dbasedesc.c | 14 ++-
src/backend/access/transam/xlog.c | 4 +
src/backend/commands/dbcommands.c | 107 +++++++++++++++++++++-
src/include/commands/dbcommands.h | 2 +
src/test/perl/PostgresNode.pm | 13 ++-
src/test/perl/RecursiveCopy.pm | 33 ++++++-
src/test/recovery/t/011_crash_recovery.pl | 99 +++++++++++++++++++-
7 files changed, 259 insertions(+), 13 deletions(-)
diff --git a/src/backend/access/rmgrdesc/dbasedesc.c b/src/backend/access/rmgrdesc/dbasedesc.c
index c7d60ce10d..35092ffb0e 100644
--- a/src/backend/access/rmgrdesc/dbasedesc.c
+++ b/src/backend/access/rmgrdesc/dbasedesc.c
@@ -23,21 +23,25 @@ dbase_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ char *dbpath1, *dbpath2;
if (info == XLOG_DBASE_CREATE)
{
xl_dbase_create_rec *xlrec = (xl_dbase_create_rec *) rec;
- appendStringInfo(buf, "copy dir %u/%u to %u/%u",
- xlrec->src_tablespace_id, xlrec->src_db_id,
- xlrec->tablespace_id, xlrec->db_id);
+ dbpath1 = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
+ dbpath2 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
+ appendStringInfo(buf, "copy dir %s to %s", dbpath1, dbpath2);
+ pfree(dbpath2);
+ pfree(dbpath1);
}
else if (info == XLOG_DBASE_DROP)
{
xl_dbase_drop_rec *xlrec = (xl_dbase_drop_rec *) rec;
- appendStringInfo(buf, "dir %u/%u",
- xlrec->tablespace_id, xlrec->db_id);
+ dbpath1 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
+ appendStringInfo(buf, "dir %s", dbpath1);
+ pfree(dbpath1);
}
}
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index b6c9353cbd..aa3e5c726c 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -40,6 +40,7 @@
#include "catalog/pg_control.h"
#include "catalog/pg_database.h"
#include "commands/tablespace.h"
+#include "commands/dbcommands.h"
#include "common/controldata_utils.h"
#include "miscadmin.h"
#include "pgstat.h"
@@ -7855,6 +7856,9 @@ CheckRecoveryConsistency(void)
*/
XLogCheckInvalidPages();
+ /* Check whether some missing directories are unexpected. */
+ CheckMissingDirs4DbaseRedo();
+
reachedConsistency = true;
ereport(LOG,
(errmsg("consistent recovery state reached at %X/%X",
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index 863f89f19d..8ae804467e 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -45,6 +45,7 @@
#include "commands/defrem.h"
#include "commands/seclabel.h"
#include "commands/tablespace.h"
+#include "common/file_perm.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "pgstat.h"
@@ -92,6 +93,68 @@ static void remove_dbtablespaces(Oid db_id);
static bool check_db_file_conflict(Oid db_id);
static int errdetail_busy_db(int notherbackends, int npreparedxacts);
+static void log_missing_directory(char *dir);
+static void forget_missing_directory(char *dir);
+
+/*
+ * During XLOG replay, we may see either src directory or dst directory
+ * is missing during copying directory when creating database in dbase_redo()
+ * if the related tablespace was later dropped but we do re-redoing in
+ * recovery after abnormal shutdown. We do simply ignore copying in
+ * dbase_redo() but log those directories in memory and then sanity check
+ * the potential bug or bad user behaviors.
+ *
+ * We use List for simplicity since this should be ok for most cases - the
+ * list should be not long in usual case.
+ */
+static List *missing_dirs_dbase_redo = NIL;
+
+void
+CheckMissingDirs4DbaseRedo()
+{
+ ListCell *lc;
+
+ if (missing_dirs_dbase_redo == NIL)
+ return;
+
+ foreach(lc, missing_dirs_dbase_redo)
+ {
+ char *dir_entry = (char *) lfirst(lc);
+
+ elog(LOG, "Directory \"%s\" was missing during directory copying "
+ "when replaying 'database create'", dir_entry);
+ }
+
+ elog(PANIC, "WAL replay was wrong due to previous missing directories");
+}
+
+static void
+log_missing_directory(char *dir)
+{
+ elog(DEBUG2, "Logging missing directory for dbase_redo(): \"%s\"", dir);
+ missing_dirs_dbase_redo = lappend(missing_dirs_dbase_redo, pstrdup(dir));
+}
+
+static void
+forget_missing_directory(char *dir)
+{
+ ListCell *prev, *lc;
+
+ prev = NULL;
+ foreach(lc, missing_dirs_dbase_redo)
+ {
+ char *dir_entry = (char *) lfirst(lc);
+
+ if (strcmp(dir_entry, dir) == 0)
+ {
+ missing_dirs_dbase_redo = list_delete_cell(missing_dirs_dbase_redo, lc, prev);
+ elog(DEBUG2, "forgetting missing directory for dbase_redo(): \"%s\"", dir);
+ return;
+ }
+
+ prev = lc;
+ }
+}
/*
* CREATE DATABASE
@@ -2108,7 +2171,9 @@ dbase_redo(XLogReaderState *record)
xl_dbase_create_rec *xlrec = (xl_dbase_create_rec *) XLogRecGetData(record);
char *src_path;
char *dst_path;
+ char *parent_path;
struct stat st;
+ bool do_copydir = true;
src_path = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
dst_path = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
@@ -2126,6 +2191,43 @@ dbase_redo(XLogReaderState *record)
(errmsg("some useless files may be left behind in old database directory \"%s\"",
dst_path)));
}
+ else
+ {
+ /*
+ * It is possible that the tablespace was previously dropped, but
+ * we are re-redoing database create with that tablespace after
+ * an abnormal shutdown (e.g. immediate shutdown). In that case,
+ * the directory are missing, we simply skip the copydir step.
+ */
+ parent_path = pstrdup(dst_path);
+ get_parent_directory(parent_path);
+ if (!(stat(parent_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
+ do_copydir = false;
+ log_missing_directory(dst_path);
+ ereport(WARNING,
+ (errmsg("Skip creating database directory \"%s\". "
+ "The dest tablespace may have been removed "
+ "before abnormal shutdown. If the removal "
+ "is illegal after later checking we will panic.",
+ parent_path)));
+ }
+ pfree(parent_path);
+ }
+
+ /* src directory is possibly missing during redo also. */
+ if (!(stat(src_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
+ do_copydir = false;
+ log_missing_directory(src_path);
+ ereport(WARNING,
+ (errmsg("Skip creating database directory based on "
+ "\"%s\". The src tablespace may have been "
+ "removed before abnormal shutdown. If the removal "
+ "is illegal after later checking we will panic.",
+ src_path)));
+
+ }
/*
* Force dirty buffers out to disk, to ensure source database is
@@ -2138,7 +2240,8 @@ dbase_redo(XLogReaderState *record)
*
* We don't need to copy subdirectories
*/
- copydir(src_path, dst_path, false);
+ if (do_copydir)
+ copydir(src_path, dst_path, false);
}
else if (info == XLOG_DBASE_DROP)
{
@@ -2181,6 +2284,8 @@ dbase_redo(XLogReaderState *record)
(errmsg("some useless files may be left behind in old database directory \"%s\"",
dst_path)));
+ forget_missing_directory(dst_path);
+
if (InHotStandby)
{
/*
diff --git a/src/include/commands/dbcommands.h b/src/include/commands/dbcommands.h
index 28bf21153d..4893e0e289 100644
--- a/src/include/commands/dbcommands.h
+++ b/src/include/commands/dbcommands.h
@@ -19,6 +19,8 @@
#include "lib/stringinfo.h"
#include "nodes/parsenodes.h"
+extern void CheckMissingDirs4DbaseRedo(void);
+
extern Oid createdb(ParseState *pstate, const CreatedbStmt *stmt);
extern void dropdb(const char *dbname, bool missing_ok);
extern ObjectAddress RenameDatabase(const char *oldname, const char *newname);
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 6019f37f91..ba9b3f180f 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -542,13 +542,22 @@ target server since it isn't done by default.
sub backup
{
- my ($self, $backup_name) = @_;
+ my ($self, $backup_name, %params) = @_;
my $backup_path = $self->backup_dir . '/' . $backup_name;
my $name = $self->name;
+ my @rest = ();
+
+ if (defined $params{tablespace_mappings})
+ {
+ my @ts_mappings = split(/,/, $params{tablespace_mappings});
+ foreach my $elem (@ts_mappings) {
+ push(@rest, '--tablespace-mapping='.$elem);
+ }
+ }
print "# Taking pg_basebackup $backup_name from node \"$name\"\n";
TestLib::system_or_bail('pg_basebackup', '-D', $backup_path, '-h',
- $self->host, '-p', $self->port, '--no-sync');
+ $self->host, '-p', $self->port, '--no-sync', @rest);
print "# Backup finished\n";
return;
}
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
index baf5d0ac63..514ed90ae7 100644
--- a/src/test/perl/RecursiveCopy.pm
+++ b/src/test/perl/RecursiveCopy.pm
@@ -22,6 +22,7 @@ use warnings;
use Carp;
use File::Basename;
use File::Copy;
+use TestLib;
=pod
@@ -97,14 +98,38 @@ sub _copypath_recurse
# invoke the filter and skip all further operation if it returns false
return 1 unless &$filterfn($curr_path);
- # Check for symlink -- needed only on source dir
- # (note: this will fall through quietly if file is already gone)
- croak "Cannot operate on symlink \"$srcpath\"" if -l $srcpath;
-
# Abort if destination path already exists. Should we allow directories
# to exist already?
croak "Destination path \"$destpath\" already exists" if -e $destpath;
+ # Check for symlink -- needed only on source dir
+ # (note: this will fall through quietly if file is already gone)
+ if (-l $srcpath)
+ {
+ croak "Cannot operate on symlink \"$srcpath\""
+ if ($srcpath !~ /\/(pg_tblspc\/[0-9]+)$/);
+
+ # We have mapped tablespaces. Copy them individually
+ my $linkname = $1;
+ my $tmpdir = TestLib::tempdir;
+ my $dstrealdir = TestLib::perl2host($tmpdir);
+ my $srcrealdir = readlink($srcpath);
+
+ opendir(my $dh, $srcrealdir);
+ while (readdir $dh)
+ {
+ next if (/^\.\.?$/);
+ my $spath = "$srcrealdir/$_";
+ my $dpath = "$dstrealdir/$_";
+
+ copypath($spath, $dpath);
+ }
+ closedir $dh;
+
+ symlink $dstrealdir, $destpath;
+ return 1;
+ }
+
# If this source path is a file, simply copy it to destination with the
# same name and we're done.
if (-f $srcpath)
diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl
index 5dc52412ca..a769236438 100644
--- a/src/test/recovery/t/011_crash_recovery.pl
+++ b/src/test/recovery/t/011_crash_recovery.pl
@@ -15,7 +15,7 @@ if ($Config{osname} eq 'MSWin32')
}
else
{
- plan tests => 3;
+ plan tests => 5;
}
my $node = get_new_node('master');
@@ -66,3 +66,100 @@ is($node->safe_psql('postgres', qq[SELECT txid_status('$xid');]),
'aborted', 'xid is aborted after crash');
$tx->kill_kill;
+
+# Ensure that tablespace removal doesn't cause error while recovering
+# the preceding create database with that tablespace.
+
+my $node_master = get_new_node('master2');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Create tablespace
+my $tspDir_master = TestLib::tempdir;
+my $realTSDir_master = TestLib::perl2host($tspDir_master);
+$node_master->safe_psql('postgres', "CREATE TABLESPACE ts1 LOCATION '$realTSDir_master'");
+
+my $tspDir_standby = TestLib::tempdir;
+my $realTSDir_standby = TestLib::perl2host($tspDir_standby);
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name,
+ tablespace_mappings =>
+ "$realTSDir_master=$realTSDir_standby");
+my $node_standby = get_new_node('standby2');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->start;
+
+# Make sure connection is made
+$node_master->poll_query_until(
+ 'postgres', 'SELECT count(*) = 1 FROM pg_stat_replication');
+
+# Make sure to perform restartpoint after tablespace creation
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->safe_psql('postgres', 'CHECKPOINT');
+
+# Do immediate shutdown just after a sequence of CREAT DATABASE / DROP
+# DATABASE / DROP TABLESPACE. This leaves a CREATE DATBASE WAL record
+# that is to be applied to already-removed tablespace.
+$node_master->safe_psql('postgres',
+ q[CREATE DATABASE db1 WITH TABLESPACE ts1;
+ DROP DATABASE db1;
+ DROP TABLESPACE ts1;]);
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->stop('immediate');
+
+# Should restart ignoring directory creation error.
+is($node_standby->start(fail_ok => 1), 1);
+
+
+# Ensure that tablespace removal doesn't cause error while recovering the
+# preceding alter database set tablespace.
+
+$node_master = get_new_node('master3');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Create tablespace
+$tspDir_master = TestLib::tempdir;
+$realTSDir_master = TestLib::perl2host($tspDir_master);
+mkdir "$realTSDir_master/1";
+mkdir "$realTSDir_master/2";
+$node_master->safe_psql('postgres', "CREATE TABLESPACE ts1 LOCATION '$realTSDir_master/1'");
+$node_master->safe_psql('postgres', "CREATE TABLESPACE ts2 LOCATION '$realTSDir_master/2'");
+
+$tspDir_standby = TestLib::tempdir;
+$realTSDir_standby = TestLib::perl2host($tspDir_standby);
+
+# Take backup
+$backup_name = 'my_backup';
+$node_master->backup($backup_name,
+ tablespace_mappings =>
+ "$realTSDir_master/1=$realTSDir_standby/1,$realTSDir_master/2=$realTSDir_standby/2");
+$node_standby = get_new_node('standby3');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->start;
+
+# Make sure connection is made
+$node_master->poll_query_until(
+ 'postgres', 'SELECT count(*) = 1 FROM pg_stat_replication');
+
+$node_master->safe_psql('postgres', "CREATE DATABASE db1 TABLESPACE ts1");
+
+# Make sure to perform restartpoint after tablespace creation
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->safe_psql('postgres', 'CHECKPOINT');
+
+# Do immediate shutdown ...
+$node_master->safe_psql('postgres',
+ q[ALTER DATABASE db1 SET TABLESPACE ts2;
+ DROP TABLESPACE ts1;]);
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->stop('immediate');
+
+# Should restart ignoring directory creation error.
+is($node_standby->start(fail_ok => 1), 1);
--
2.17.2
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
@ 2019-08-01 22:37 ` Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Thomas Munro @ 2019-08-01 22:37 UTC (permalink / raw)
To: Paul Guo <[email protected]>; +Cc: Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
On Mon, Jul 15, 2019 at 10:52 PM Paul Guo <[email protected]> wrote:
> Please see the attached v4 patch.
While moving this to the next CF, I noticed that this needs updating
for the new pg_list.h API.
--
Thomas Munro
https://enterprisedb.com
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
@ 2019-08-22 13:13 ` Paul Guo <[email protected]>
2019-08-22 16:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
0 siblings, 2 replies; 109+ messages in thread
From: Paul Guo @ 2019-08-22 13:13 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
Thanks. I updated the patch to v5. It passes install-check testing and
recovery testing.
On Fri, Aug 2, 2019 at 6:38 AM Thomas Munro <[email protected]> wrote:
> On Mon, Jul 15, 2019 at 10:52 PM Paul Guo <[email protected]> wrote:
> > Please see the attached v4 patch.
>
> While moving this to the next CF, I noticed that this needs updating
> for the new pg_list.h API.
>
> --
> Thomas Munro
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__enterprisedb.com&d=DwIBaQ&c=lnl9vOaLMzs...
>
Attachments:
[application/octet-stream] v5-0001-skip-copydir-if-either-src-directory-or-dst-direc.patch (14.8K, ../../CAEET0ZGRf8oSjM2uq7Y1=bV2=p4=mJ1soHCsBj_E-_1xF6R8EQ@mail.gmail.com/3-v5-0001-skip-copydir-if-either-src-directory-or-dst-direc.patch)
download | inline diff:
From 1b6c0c2c67cfcdedb3de93d9a048cf86e4ae04f6 Mon Sep 17 00:00:00 2001
From: Paul Guo <[email protected]>
Date: Tue, 30 Apr 2019 13:30:49 +0800
Subject: [PATCH v5] skip copydir() if either src directory or dst directory is
missing due to re-redoing create database but the tablespace is dropped.
Also correct dbase_desc() so that related xlog description is not misleading.
This patch uses the log/forget mechanism to avoid bad ignoring.
Kyotaro horiguchi added one test and did related change in PostgresNode.pm and
had a lot of discussion on this issue. I further added another test and
modified PostgresNode.pm again to support my new test.
---
src/backend/access/rmgrdesc/dbasedesc.c | 14 +--
src/backend/access/transam/xlog.c | 4 +
src/backend/commands/dbcommands.c | 104 +++++++++++++++++++++-
src/include/commands/dbcommands.h | 2 +
src/test/perl/PostgresNode.pm | 13 ++-
src/test/perl/RecursiveCopy.pm | 33 ++++++-
src/test/recovery/t/011_crash_recovery.pl | 99 +++++++++++++++++++-
7 files changed, 256 insertions(+), 13 deletions(-)
diff --git a/src/backend/access/rmgrdesc/dbasedesc.c b/src/backend/access/rmgrdesc/dbasedesc.c
index c7d60ce10d..35092ffb0e 100644
--- a/src/backend/access/rmgrdesc/dbasedesc.c
+++ b/src/backend/access/rmgrdesc/dbasedesc.c
@@ -23,21 +23,25 @@ dbase_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ char *dbpath1, *dbpath2;
if (info == XLOG_DBASE_CREATE)
{
xl_dbase_create_rec *xlrec = (xl_dbase_create_rec *) rec;
- appendStringInfo(buf, "copy dir %u/%u to %u/%u",
- xlrec->src_tablespace_id, xlrec->src_db_id,
- xlrec->tablespace_id, xlrec->db_id);
+ dbpath1 = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
+ dbpath2 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
+ appendStringInfo(buf, "copy dir %s to %s", dbpath1, dbpath2);
+ pfree(dbpath2);
+ pfree(dbpath1);
}
else if (info == XLOG_DBASE_DROP)
{
xl_dbase_drop_rec *xlrec = (xl_dbase_drop_rec *) rec;
- appendStringInfo(buf, "dir %u/%u",
- xlrec->tablespace_id, xlrec->db_id);
+ dbpath1 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
+ appendStringInfo(buf, "dir %s", dbpath1);
+ pfree(dbpath1);
}
}
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index e651a841bb..0c4928a7c7 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -40,6 +40,7 @@
#include "catalog/pg_control.h"
#include "catalog/pg_database.h"
#include "commands/tablespace.h"
+#include "commands/dbcommands.h"
#include "common/controldata_utils.h"
#include "miscadmin.h"
#include "pgstat.h"
@@ -7858,6 +7859,9 @@ CheckRecoveryConsistency(void)
*/
XLogCheckInvalidPages();
+ /* Check whether some missing directories are unexpected. */
+ CheckMissingDirs4DbaseRedo();
+
reachedConsistency = true;
ereport(LOG,
(errmsg("consistent recovery state reached at %X/%X",
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index 95881a8550..bf85677bd4 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -45,6 +45,7 @@
#include "commands/defrem.h"
#include "commands/seclabel.h"
#include "commands/tablespace.h"
+#include "common/file_perm.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "pgstat.h"
@@ -92,6 +93,65 @@ static void remove_dbtablespaces(Oid db_id);
static bool check_db_file_conflict(Oid db_id);
static int errdetail_busy_db(int notherbackends, int npreparedxacts);
+static void log_missing_directory(char *dir);
+static void forget_missing_directory(char *dir);
+
+/*
+ * During XLOG replay, we may see either src directory or dst directory
+ * is missing during copying directory when creating database in dbase_redo()
+ * if the related tablespace was later dropped but we do re-redoing in
+ * recovery after abnormal shutdown. We do simply ignore copying in
+ * dbase_redo() but log those directories in memory and then sanity check
+ * the potential bug or bad user behaviors.
+ *
+ * We use List for simplicity since this should be ok for most cases - the
+ * list should be not long in usual case.
+ */
+static List *missing_dirs_dbase_redo = NIL;
+
+void
+CheckMissingDirs4DbaseRedo()
+{
+ ListCell *lc;
+
+ if (missing_dirs_dbase_redo == NIL)
+ return;
+
+ foreach(lc, missing_dirs_dbase_redo)
+ {
+ char *dir_entry = (char *) lfirst(lc);
+
+ elog(LOG, "Directory \"%s\" was missing during directory copying "
+ "when replaying 'database create'", dir_entry);
+ }
+
+ elog(PANIC, "WAL replay was wrong due to previous missing directories");
+}
+
+static void
+log_missing_directory(char *dir)
+{
+ elog(DEBUG2, "Logging missing directory for dbase_redo(): \"%s\"", dir);
+ missing_dirs_dbase_redo = lappend(missing_dirs_dbase_redo, pstrdup(dir));
+}
+
+static void
+forget_missing_directory(char *dir)
+{
+ ListCell *lc;
+
+ foreach(lc, missing_dirs_dbase_redo)
+ {
+ char *dir_entry = (char *) lfirst(lc);
+
+ if (strcmp(dir_entry, dir) == 0)
+ {
+ missing_dirs_dbase_redo = list_delete_cell(missing_dirs_dbase_redo, lc);
+ elog(DEBUG2, "forgetting missing directory for dbase_redo(): \"%s\"", dir);
+ return;
+ }
+ }
+}
/*
* CREATE DATABASE
@@ -2129,7 +2189,9 @@ dbase_redo(XLogReaderState *record)
xl_dbase_create_rec *xlrec = (xl_dbase_create_rec *) XLogRecGetData(record);
char *src_path;
char *dst_path;
+ char *parent_path;
struct stat st;
+ bool do_copydir = true;
src_path = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
dst_path = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
@@ -2147,6 +2209,43 @@ dbase_redo(XLogReaderState *record)
(errmsg("some useless files may be left behind in old database directory \"%s\"",
dst_path)));
}
+ else
+ {
+ /*
+ * It is possible that the tablespace was previously dropped, but
+ * we are re-redoing database create with that tablespace after
+ * an abnormal shutdown (e.g. immediate shutdown). In that case,
+ * the directory are missing, we simply skip the copydir step.
+ */
+ parent_path = pstrdup(dst_path);
+ get_parent_directory(parent_path);
+ if (!(stat(parent_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
+ do_copydir = false;
+ log_missing_directory(dst_path);
+ ereport(WARNING,
+ (errmsg("Skip creating database directory \"%s\". "
+ "The dest tablespace may have been removed "
+ "before abnormal shutdown. If the removal "
+ "is illegal after later checking we will panic.",
+ parent_path)));
+ }
+ pfree(parent_path);
+ }
+
+ /* src directory is possibly missing during redo also. */
+ if (!(stat(src_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
+ do_copydir = false;
+ log_missing_directory(src_path);
+ ereport(WARNING,
+ (errmsg("Skip creating database directory based on "
+ "\"%s\". The src tablespace may have been "
+ "removed before abnormal shutdown. If the removal "
+ "is illegal after later checking we will panic.",
+ src_path)));
+
+ }
/*
* Force dirty buffers out to disk, to ensure source database is
@@ -2159,7 +2258,8 @@ dbase_redo(XLogReaderState *record)
*
* We don't need to copy subdirectories
*/
- copydir(src_path, dst_path, false);
+ if (do_copydir)
+ copydir(src_path, dst_path, false);
}
else if (info == XLOG_DBASE_DROP)
{
@@ -2202,6 +2302,8 @@ dbase_redo(XLogReaderState *record)
(errmsg("some useless files may be left behind in old database directory \"%s\"",
dst_path)));
+ forget_missing_directory(dst_path);
+
if (InHotStandby)
{
/*
diff --git a/src/include/commands/dbcommands.h b/src/include/commands/dbcommands.h
index 154c8157ee..26e96b8957 100644
--- a/src/include/commands/dbcommands.h
+++ b/src/include/commands/dbcommands.h
@@ -19,6 +19,8 @@
#include "lib/stringinfo.h"
#include "nodes/parsenodes.h"
+extern void CheckMissingDirs4DbaseRedo(void);
+
extern Oid createdb(ParseState *pstate, const CreatedbStmt *stmt);
extern void dropdb(const char *dbname, bool missing_ok);
extern ObjectAddress RenameDatabase(const char *oldname, const char *newname);
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 270bd6c856..b5dc3a8918 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -546,13 +546,22 @@ target server since it isn't done by default.
sub backup
{
- my ($self, $backup_name) = @_;
+ my ($self, $backup_name, %params) = @_;
my $backup_path = $self->backup_dir . '/' . $backup_name;
my $name = $self->name;
+ my @rest = ();
+
+ if (defined $params{tablespace_mappings})
+ {
+ my @ts_mappings = split(/,/, $params{tablespace_mappings});
+ foreach my $elem (@ts_mappings) {
+ push(@rest, '--tablespace-mapping='.$elem);
+ }
+ }
print "# Taking pg_basebackup $backup_name from node \"$name\"\n";
TestLib::system_or_bail('pg_basebackup', '-D', $backup_path, '-h',
- $self->host, '-p', $self->port, '--no-sync');
+ $self->host, '-p', $self->port, '--no-sync', @rest);
print "# Backup finished\n";
return;
}
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
index baf5d0ac63..514ed90ae7 100644
--- a/src/test/perl/RecursiveCopy.pm
+++ b/src/test/perl/RecursiveCopy.pm
@@ -22,6 +22,7 @@ use warnings;
use Carp;
use File::Basename;
use File::Copy;
+use TestLib;
=pod
@@ -97,14 +98,38 @@ sub _copypath_recurse
# invoke the filter and skip all further operation if it returns false
return 1 unless &$filterfn($curr_path);
- # Check for symlink -- needed only on source dir
- # (note: this will fall through quietly if file is already gone)
- croak "Cannot operate on symlink \"$srcpath\"" if -l $srcpath;
-
# Abort if destination path already exists. Should we allow directories
# to exist already?
croak "Destination path \"$destpath\" already exists" if -e $destpath;
+ # Check for symlink -- needed only on source dir
+ # (note: this will fall through quietly if file is already gone)
+ if (-l $srcpath)
+ {
+ croak "Cannot operate on symlink \"$srcpath\""
+ if ($srcpath !~ /\/(pg_tblspc\/[0-9]+)$/);
+
+ # We have mapped tablespaces. Copy them individually
+ my $linkname = $1;
+ my $tmpdir = TestLib::tempdir;
+ my $dstrealdir = TestLib::perl2host($tmpdir);
+ my $srcrealdir = readlink($srcpath);
+
+ opendir(my $dh, $srcrealdir);
+ while (readdir $dh)
+ {
+ next if (/^\.\.?$/);
+ my $spath = "$srcrealdir/$_";
+ my $dpath = "$dstrealdir/$_";
+
+ copypath($spath, $dpath);
+ }
+ closedir $dh;
+
+ symlink $dstrealdir, $destpath;
+ return 1;
+ }
+
# If this source path is a file, simply copy it to destination with the
# same name and we're done.
if (-f $srcpath)
diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl
index 526a3481fb..5ac7303dd6 100644
--- a/src/test/recovery/t/011_crash_recovery.pl
+++ b/src/test/recovery/t/011_crash_recovery.pl
@@ -15,7 +15,7 @@ if ($Config{osname} eq 'MSWin32')
}
else
{
- plan tests => 3;
+ plan tests => 5;
}
my $node = get_new_node('master');
@@ -66,3 +66,100 @@ is($node->safe_psql('postgres', qq[SELECT txid_status('$xid');]),
'aborted', 'xid is aborted after crash');
$tx->kill_kill;
+
+# Ensure that tablespace removal doesn't cause error while recovering
+# the preceding create database with that tablespace.
+
+my $node_master = get_new_node('master2');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Create tablespace
+my $tspDir_master = TestLib::tempdir;
+my $realTSDir_master = TestLib::perl2host($tspDir_master);
+$node_master->safe_psql('postgres', "CREATE TABLESPACE ts1 LOCATION '$realTSDir_master'");
+
+my $tspDir_standby = TestLib::tempdir;
+my $realTSDir_standby = TestLib::perl2host($tspDir_standby);
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name,
+ tablespace_mappings =>
+ "$realTSDir_master=$realTSDir_standby");
+my $node_standby = get_new_node('standby2');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->start;
+
+# Make sure connection is made
+$node_master->poll_query_until(
+ 'postgres', 'SELECT count(*) = 1 FROM pg_stat_replication');
+
+# Make sure to perform restartpoint after tablespace creation
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->safe_psql('postgres', 'CHECKPOINT');
+
+# Do immediate shutdown just after a sequence of CREAT DATABASE / DROP
+# DATABASE / DROP TABLESPACE. This leaves a CREATE DATBASE WAL record
+# that is to be applied to already-removed tablespace.
+$node_master->safe_psql('postgres',
+ q[CREATE DATABASE db1 WITH TABLESPACE ts1;
+ DROP DATABASE db1;
+ DROP TABLESPACE ts1;]);
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->stop('immediate');
+
+# Should restart ignoring directory creation error.
+is($node_standby->start(fail_ok => 1), 1);
+
+
+# Ensure that tablespace removal doesn't cause error while recovering the
+# preceding alter database set tablespace.
+
+$node_master = get_new_node('master3');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Create tablespace
+$tspDir_master = TestLib::tempdir;
+$realTSDir_master = TestLib::perl2host($tspDir_master);
+mkdir "$realTSDir_master/1";
+mkdir "$realTSDir_master/2";
+$node_master->safe_psql('postgres', "CREATE TABLESPACE ts1 LOCATION '$realTSDir_master/1'");
+$node_master->safe_psql('postgres', "CREATE TABLESPACE ts2 LOCATION '$realTSDir_master/2'");
+
+$tspDir_standby = TestLib::tempdir;
+$realTSDir_standby = TestLib::perl2host($tspDir_standby);
+
+# Take backup
+$backup_name = 'my_backup';
+$node_master->backup($backup_name,
+ tablespace_mappings =>
+ "$realTSDir_master/1=$realTSDir_standby/1,$realTSDir_master/2=$realTSDir_standby/2");
+$node_standby = get_new_node('standby3');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->start;
+
+# Make sure connection is made
+$node_master->poll_query_until(
+ 'postgres', 'SELECT count(*) = 1 FROM pg_stat_replication');
+
+$node_master->safe_psql('postgres', "CREATE DATABASE db1 TABLESPACE ts1");
+
+# Make sure to perform restartpoint after tablespace creation
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->safe_psql('postgres', 'CHECKPOINT');
+
+# Do immediate shutdown ...
+$node_master->safe_psql('postgres',
+ q[ALTER DATABASE db1 SET TABLESPACE ts2;
+ DROP TABLESPACE ts1;]);
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->stop('immediate');
+
+# Should restart ignoring directory creation error.
+is($node_standby->start(fail_ok => 1), 1);
--
2.17.2
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
@ 2019-08-22 16:13 ` Anastasia Lubennikova <[email protected]>
2019-09-03 15:58 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2019-09-10 11:42 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
1 sibling, 2 replies; 109+ messages in thread
From: Anastasia Lubennikova @ 2019-08-22 16:13 UTC (permalink / raw)
To: Paul Guo <[email protected]>; Thomas Munro <[email protected]>; +Cc: Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
22.08.2019 16:13, Paul Guo wrote:
> Thanks. I updated the patch to v5. It passes install-check testing and
> recovery testing.
Hi,
Thank you for working on this fix.
The overall design of the latest version looks good to me.
But during the review, I found a bug in the current implementation.
New behavior must apply to crash-recovery only, now it applies to
archiveRecovery too.
That can cause a silent loss of a tablespace during regular standby
operation
since it never calls CheckRecoveryConsistency().
Steps to reproduce:
1) run master and replica
2) create dir for tablespace:
mkdir /tmp/tblspc1
3) create tablespace and database on the master:
create tablespace tblspc1 location '/tmp/tblspc1';
create database db1 tablespace tblspc1 ;
4) wait for replica to receive this changes and pause replication:
select pg_wal_replay_pause();
5) move replica's tablespace symlink to some empty directory, i.e.
/tmp/tblspc2
mkdir /tmp/tblspc2
ln -sfn /tmp/tblspc2 postgresql_data_replica/pg_tblspc/16384
6) create another database in tblspc1 on master:
create database db2 tablespace tblspc1 ;
7) resume replication on standby:
select pg_wal_replay_resume();
8) try to connect to db2 on standby
It's expected that dbase_redo() will fail because the directory on
standby is not found.
While with the patch it suppresses the error until we attempt to connect
db2 on the standby:
2019-08-22 18:34:39.178 MSK [21066] HINT: Execute
pg_wal_replay_resume() to continue.
2019-08-22 18:42:41.656 MSK [21066] WARNING: Skip creating database
directory "pg_tblspc/16384/PG_13_201908012". The dest tablespace may
have been removed before abnormal shutdown. If the removal is illegal
after later checking we will panic.
2019-08-22 18:42:41.656 MSK [21066] CONTEXT: WAL redo at 0/3027738 for
Database/CREATE: copy dir base/1 to pg_tblspc/16384/PG_13_201908012/16390
2019-08-22 18:42:46.096 MSK [21688] FATAL:
"pg_tblspc/16384/PG_13_201908012/16390" is not a valid data directory
2019-08-22 18:42:46.096 MSK [21688] DETAIL: File
"pg_tblspc/16384/PG_13_201908012/16390/PG_VERSION" is missing.
Also some nitpicking about code style:
1) Please, add comment to forget_missing_directory().
2) + elog(LOG, "Directory \"%s\" was missing during
directory copying "
I think we'd better update this message elevel to WARNING.
3) Shouldn't we also move FlushDatabaseBuffers(xlrec->src_db_id); call under
if (do_copydir) clause?
I don't see a reason to flush pages that we won't use later.
--
Anastasia Lubennikova
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-22 16:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
@ 2019-09-03 15:58 ` Alvaro Herrera <[email protected]>
2019-09-05 07:12 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
1 sibling, 1 reply; 109+ messages in thread
From: Alvaro Herrera @ 2019-09-03 15:58 UTC (permalink / raw)
To: Anastasia Lubennikova <[email protected]>; +Cc: Paul Guo <[email protected]>; Thomas Munro <[email protected]>; Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
On 2019-Aug-22, Anastasia Lubennikova wrote:
> 22.08.2019 16:13, Paul Guo wrote:
> > Thanks. I updated the patch to v5. It passes install-check testing and
> > recovery testing.
> Hi,
> Thank you for working on this fix.
> The overall design of the latest version looks good to me.
> But during the review, I found a bug in the current implementation.
> New behavior must apply to crash-recovery only, now it applies to
> archiveRecovery too.
Hello
Paul, Kyotaro, are you working on updating this bugfix? FWIW the latest
patch submitted by Paul is still current and CFbot says it passes its
own test, but from Anastasia's email it still needs a bit of work.
Also: it would be good to have this new bogus scenario described by
Anastasia covered by a new TAP test. Anastasia, can we enlist you to
write that? Maybe Kyotaro?
Thanks
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-22 16:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2019-09-03 15:58 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2019-09-05 07:12 ` Paul Guo <[email protected]>
0 siblings, 0 replies; 109+ messages in thread
From: Paul Guo @ 2019-09-05 07:12 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Anastasia Lubennikova <[email protected]>; Thomas Munro <[email protected]>; Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
On Tue, Sep 3, 2019 at 11:58 PM Alvaro Herrera <[email protected]>
wrote:
> On 2019-Aug-22, Anastasia Lubennikova wrote:
>
> > 22.08.2019 16:13, Paul Guo wrote:
> > > Thanks. I updated the patch to v5. It passes install-check testing and
> > > recovery testing.
> > Hi,
> > Thank you for working on this fix.
> > The overall design of the latest version looks good to me.
> > But during the review, I found a bug in the current implementation.
> > New behavior must apply to crash-recovery only, now it applies to
> > archiveRecovery too.
>
> Hello
>
> Paul, Kyotaro, are you working on updating this bugfix? FWIW the latest
> patch submitted by Paul is still current and CFbot says it passes its
> own test, but from Anastasia's email it still needs a bit of work.
>
> Also: it would be good to have this new bogus scenario described by
> Anastasia covered by a new TAP test. Anastasia, can we enlist you to
> write that? Maybe Kyotaro?
>
>
Thanks Anastasia and Alvaro for comment and suggestion. Sorry I've been busy
working on some non-PG stuffs recently. I've never worked on archive
recovery,
so I expect a bit more time after I'm free (hopefully several days later)
to take a look.
Of course Kyotaro, Anastasia or anyone feel free to address the concern
before that.
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-22 16:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
@ 2019-09-10 11:42 ` Asim R P <[email protected]>
2019-09-11 14:26 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
1 sibling, 1 reply; 109+ messages in thread
From: Asim R P @ 2019-09-10 11:42 UTC (permalink / raw)
To: Anastasia Lubennikova <[email protected]>; +Cc: Paul Guo <[email protected]>; Thomas Munro <[email protected]>; Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
Hi Anastasia
On Thu, Aug 22, 2019 at 9:43 PM Anastasia Lubennikova <
[email protected]> wrote:
>
> But during the review, I found a bug in the current implementation.
> New behavior must apply to crash-recovery only, now it applies to
archiveRecovery too.
> That can cause a silent loss of a tablespace during regular standby
operation
> since it never calls CheckRecoveryConsistency().
>
> Steps to reproduce:
> 1) run master and replica
> 2) create dir for tablespace:
> mkdir /tmp/tblspc1
>
> 3) create tablespace and database on the master:
> create tablespace tblspc1 location '/tmp/tblspc1';
> create database db1 tablespace tblspc1 ;
>
> 4) wait for replica to receive this changes and pause replication:
> select pg_wal_replay_pause();
>
> 5) move replica's tablespace symlink to some empty directory, i.e.
/tmp/tblspc2
> mkdir /tmp/tblspc2
> ln -sfn /tmp/tblspc2 postgresql_data_replica/pg_tblspc/16384
>
By changing the tablespace symlink target, we are silently nullifying
effects of a committed transaction from the standby data directory - the
directory structure created by the standby for create tablespace
transaction. This step, therefore, does not look like a valid test case to
me. Can you share a sequence of steps that does not involve changing data
directory manually?
>
> Also some nitpicking about code style:
> 1) Please, add comment to forget_missing_directory().
>
> 2) + elog(LOG, "Directory \"%s\" was missing during
directory copying "
> I think we'd better update this message elevel to WARNING.
>
> 3) Shouldn't we also move FlushDatabaseBuffers(xlrec->src_db_id); call
under
> if (do_copydir) clause?
> I don't see a reason to flush pages that we won't use later.
>
Thank you for the review feedback. I agree with all the points. Let me
incorporate them (I plan to pick this work up and drive it to completion as
Paul got busy with other things).
But before that I'm revisiting another solution upthread, that of creating
restart points when replaying create/drop database commands before making
filesystem changes such as removing a directory. The restart points should
align with checkpoints on master. The concern against this solution was
creation of restart points will slow down recovery. I don't think crash
recovery is affected by this solution because of the already existing
enforcement of checkpoints. WAL records prior to a create/drop database
will not be seen by crash recovery due to the checkpoint enforced during
the command's normal execution.
Asim
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-22 16:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2019-09-10 11:42 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
@ 2019-09-11 14:26 ` Anastasia Lubennikova <[email protected]>
2019-09-12 08:35 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Anastasia Lubennikova @ 2019-09-11 14:26 UTC (permalink / raw)
To: Asim R P <[email protected]>; +Cc: Paul Guo <[email protected]>; Thomas Munro <[email protected]>; Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
10.09.2019 14:42, Asim R P wrote:
> Hi Anastasia
>
> On Thu, Aug 22, 2019 at 9:43 PM Anastasia Lubennikova
> <[email protected] <mailto:[email protected]>>
> wrote:
> >
> > But during the review, I found a bug in the current implementation.
> > New behavior must apply to crash-recovery only, now it applies to
> archiveRecovery too.
> > That can cause a silent loss of a tablespace during regular standby
> operation
> > since it never calls CheckRecoveryConsistency().
> >
> > Steps to reproduce:
> > 1) run master and replica
> > 2) create dir for tablespace:
> > mkdir /tmp/tblspc1
> >
> > 3) create tablespace and database on the master:
> > create tablespace tblspc1 location '/tmp/tblspc1';
> > create database db1 tablespace tblspc1 ;
> >
> > 4) wait for replica to receive this changes and pause replication:
> > select pg_wal_replay_pause();
> >
> > 5) move replica's tablespace symlink to some empty directory, i.e.
> /tmp/tblspc2
> > mkdir /tmp/tblspc2
> > ln -sfn /tmp/tblspc2 postgresql_data_replica/pg_tblspc/16384
> >
>
> By changing the tablespace symlink target, we are silently nullifying
> effects of a committed transaction from the standby data directory -
> the directory structure created by the standby for create tablespace
> transaction. This step, therefore, does not look like a valid test
> case to me. Can you share a sequence of steps that does not involve
> changing data directory manually?
>
Hi, the whole idea of the test is to reproduce a data loss. For example,
if the disk containing this tablespace failed.
Probably, simply deleting the directory
'postgresql_data_replica/pg_tblspc/16384'
would work as well, though I was afraid that it can be caught by some
earlier checks and my example won't be so illustrative.
>
> Thank you for the review feedback. I agree with all the points. Let
> me incorporate them (I plan to pick this work up and drive it to
> completion as Paul got busy with other things).
>
> But before that I'm revisiting another solution upthread, that of
> creating restart points when replaying create/drop database commands
> before making filesystem changes such as removing a directory. The
> restart points should align with checkpoints on master. The concern
> against this solution was creation of restart points will slow down
> recovery. I don't think crash recovery is affected by this solution
> because of the already existing enforcement of checkpoints. WAL
> records prior to a create/drop database will not be seen by crash
> recovery due to the checkpoint enforced during the command's normal
> execution.
>
I haven't measured the impact of generating extra restart points in
previous solution, so I cannot tell whether concerns upthread are
justified. Still, I enjoy latest design more, since it is clear and
similar with the code of checking unexpected uninitialized pages. In
principle it works. And the issue, I described in previous review can be
easily fixed by several additional checks of InHotStandby macro.
--
Anastasia Lubennikova
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-22 16:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2019-09-10 11:42 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-11 14:26 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
@ 2019-09-12 08:35 ` Kyotaro Horiguchi <[email protected]>
2019-09-12 12:02 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Kyotaro Horiguchi @ 2019-09-12 08:35 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers
Hello.
At Wed, 11 Sep 2019 17:26:44 +0300, Anastasia Lubennikova <[email protected]> wrote in <[email protected]>
> 10.09.2019 14:42, Asim R P wrote:
> > Hi Anastasia
> >
> > On Thu, Aug 22, 2019 at 9:43 PM Anastasia Lubennikova
> > <[email protected] <mailto:[email protected]>>
> > wrote:
> > >
> > > But during the review, I found a bug in the current implementation.
> > > New behavior must apply to crash-recovery only, now it applies to
> > > archiveRecovery too.
> > > That can cause a silent loss of a tablespace during regular standby
> > > operation
> > > since it never calls CheckRecoveryConsistency().
Yeah. We should take the same steps with redo operations on
missing pages. Just record failure during inconsistent state then
forget it if underlying tablespace is gone. If we had a record
when we reached concsistency, we're in a serious situation and
should stop recovery. log_invalid_page forget_invalid_pages and
CheckRecoveryConsistency are the entry points of the feature to
understand.
> > > Steps to reproduce:
> > > 1) run master and replica
> > > 2) create dir for tablespace:
> > > mkdir /tmp/tblspc1
> > >
> > > 3) create tablespace and database on the master:
> > > create tablespace tblspc1 location '/tmp/tblspc1';
> > > create database db1 tablespace tblspc1 ;
> > >
> > > 4) wait for replica to receive this changes and pause replication:
> > > select pg_wal_replay_pause();
> > >
> > > 5) move replica's tablespace symlink to some empty directory,
> > > i.e. /tmp/tblspc2
> > > mkdir /tmp/tblspc2
> > > ln -sfn /tmp/tblspc2 postgresql_data_replica/pg_tblspc/16384
> > >
> >
> > By changing the tablespace symlink target, we are silently nullifying
> > effects of a committed transaction from the standby data directory -
> > the directory structure created by the standby for create tablespace
> > transaction. This step, therefore, does not look like a valid test
> > case to me. Can you share a sequence of steps that does not involve
> > changing data directory manually?
I see it as the same. WAL is inconsistent with what happend on
storage with the steps. Database is just broken.
> Hi, the whole idea of the test is to reproduce a data loss. For
> example, if the disk containing this tablespace failed.
So, apparently we must start recovery from a backup before that
failure happened in that case, and that should ends in success.
# I remember that the start point of this patch is a crash after
# table space drop subsequent to several operations within the
# table space. Then, crash recovery fails at an operation in the
# finally-removed tablespace. Is it right?
> Probably, simply deleting the directory
> 'postgresql_data_replica/pg_tblspc/16384'
> would work as well, though I was afraid that it can be caught by some
> earlier checks and my example won't be so illustrative.
> >
> > Thank you for the review feedback. I agree with all the points. Let
> > me incorporate them (I plan to pick this work up and drive it to
> > completion as Paul got busy with other things).
> >
> > But before that I'm revisiting another solution upthread, that of
> > creating restart points when replaying create/drop database commands
> > before making filesystem changes such as removing a directory. The
> > restart points should align with checkpoints on master. The concern
> > against this solution was creation of restart points will slow down
> > recovery. I don't think crash recovery is affected by this solution
> > because of the already existing enforcement of checkpoints. WAL
> > records prior to a create/drop database will not be seen by crash
> > recovery due to the checkpoint enforced during the command's normal
> > execution.
> >
>
> I haven't measured the impact of generating extra restart points in
> previous solution, so I cannot tell whether concerns upthread are
> justified. Still, I enjoy latest design more, since it is clear and
> similar with the code of checking unexpected uninitialized pages. In
> principle it works. And the issue, I described in previous review can
> be easily fixed by several additional checks of InHotStandby macro.
Generally we shouldn't trigger useless restart point for
uncertain reasons. If standby crashes, it starts the next
recovery from the latest *restart point*. Even in that case what
we should do is the same.
Of course, for testing, we *should* establish a restartpoint
manually in order to establish the prerequisite state.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-22 16:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2019-09-10 11:42 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-11 14:26 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2019-09-12 08:35 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
@ 2019-09-12 12:02 ` Asim R P <[email protected]>
0 siblings, 0 replies; 109+ messages in thread
From: Asim R P @ 2019-09-12 12:02 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: Anastasia Lubennikova <[email protected]>; Paul Guo <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
On Thu, Sep 12, 2019 at 2:05 PM Kyotaro Horiguchi <[email protected]>
wrote:
>
> Hello.
>
> At Wed, 11 Sep 2019 17:26:44 +0300, Anastasia Lubennikova <
[email protected]> wrote in <
[email protected]>
> > 10.09.2019 14:42, Asim R P wrote:
> > > Hi Anastasia
> > >
> > > On Thu, Aug 22, 2019 at 9:43 PM Anastasia Lubennikova
> > > <[email protected] <mailto:[email protected]>>
> > > wrote:
> > > >
> > > > But during the review, I found a bug in the current implementation.
> > > > New behavior must apply to crash-recovery only, now it applies to
> > > > archiveRecovery too.
> > > > That can cause a silent loss of a tablespace during regular standby
> > > > operation
> > > > since it never calls CheckRecoveryConsistency().
>
> Yeah. We should take the same steps with redo operations on
> missing pages. Just record failure during inconsistent state then
> forget it if underlying tablespace is gone. If we had a record
> when we reached concsistency, we're in a serious situation and
> should stop recovery. log_invalid_page forget_invalid_pages and
> CheckRecoveryConsistency are the entry points of the feature to
> understand.
>
Yes, I get it now. I will adjust the patch written by Paul accordingly.
>
> # I remember that the start point of this patch is a crash after
> # table space drop subsequent to several operations within the
> # table space. Then, crash recovery fails at an operation in the
> # finally-removed tablespace. Is it right?
>
That's correct. Once the directories are removed from filesystem, any
attempt to replay WAL records that depend on their existence fails.
> > > But before that I'm revisiting another solution upthread, that of
> > > creating restart points when replaying create/drop database commands
> > > before making filesystem changes such as removing a directory. The
> > > restart points should align with checkpoints on master. The concern
> > > against this solution was creation of restart points will slow down
> > > recovery. I don't think crash recovery is affected by this solution
> > > because of the already existing enforcement of checkpoints. WAL
> > > records prior to a create/drop database will not be seen by crash
> > > recovery due to the checkpoint enforced during the command's normal
> > > execution.
> > >
> >
> > I haven't measured the impact of generating extra restart points in
> > previous solution, so I cannot tell whether concerns upthread are
> > justified. Still, I enjoy latest design more, since it is clear and
> > similar with the code of checking unexpected uninitialized pages. In
> > principle it works. And the issue, I described in previous review can
> > be easily fixed by several additional checks of InHotStandby macro.
>
> Generally we shouldn't trigger useless restart point for
> uncertain reasons. If standby crashes, it starts the next
> recovery from the latest *restart point*. Even in that case what
> we should do is the same.
>
The reason is quite clear to me - removing directories from filesystem
break the ability to replay WAL records second time. And we already create
checkpoints during normal operation in such a case, so crash recovery on a
master node does not suffer from this bug. I've attached a patch that
performs restart points during drop database replay, just for reference.
It passes both the TAP tests written by Kyotaro and Paul. I had to modify
drop database WAL record a bit.
Asim
Attachments:
[application/octet-stream] v1-0001-Create-restartpoint-when-replaying-drop-database.patch (6.2K, ../../CANXE4TcG68ksC5DCNatr1MZxgvuMpJi_ZfKN2f1ZG2BLyG2H_Q@mail.gmail.com/3-v1-0001-Create-restartpoint-when-replaying-drop-database.patch)
download | inline diff:
From bd98ac1a2e98bc67238103b0f2764cf7fe0edc58 Mon Sep 17 00:00:00 2001
From: Asim R P <[email protected]>
Date: Thu, 12 Sep 2019 17:17:29 +0530
Subject: [PATCH v1 1/2] Create restartpoint when replaying drop database
Drop database replay involves removing the database directory. We do
not have a mechanism similar to invalid page detection for directories
during WAL replay. If, due to a crash, WAL replay must resume from a
checkpoint, we should avoid replaying WAL records second time, that
precede the drop database.
Proposed by Paul Guo.
---
src/backend/access/rmgrdesc/dbasedesc.c | 6 ++-
src/backend/commands/dbcommands.c | 84 +++++++++++++++++++++------------
src/include/access/xlog.h | 1 -
src/include/commands/dbcommands_xlog.h | 3 +-
4 files changed, 61 insertions(+), 33 deletions(-)
diff --git a/src/backend/access/rmgrdesc/dbasedesc.c b/src/backend/access/rmgrdesc/dbasedesc.c
index c7d60ce10d..b97bac2411 100644
--- a/src/backend/access/rmgrdesc/dbasedesc.c
+++ b/src/backend/access/rmgrdesc/dbasedesc.c
@@ -36,8 +36,10 @@ dbase_desc(StringInfo buf, XLogReaderState *record)
{
xl_dbase_drop_rec *xlrec = (xl_dbase_drop_rec *) rec;
- appendStringInfo(buf, "dir %u/%u",
- xlrec->tablespace_id, xlrec->db_id);
+ int i;
+ for (i = 0; i < xlrec->nspcids; i++)
+ appendStringInfo(buf, "\ndir %u/%u",
+ xlrec->db_id, xlrec->tablespace_ids[i]);
}
}
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index 95881a8550..c0c8726698 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -1400,16 +1400,19 @@ movedb(const char *dbname, const char *tblspcname)
* Record the filesystem change in XLOG
*/
{
- xl_dbase_drop_rec xlrec;
+ size_t xlrec_size = sizeof(xl_dbase_drop_rec) + sizeof(Oid);
+ xl_dbase_drop_rec *xlrec = palloc(xlrec_size);
- xlrec.db_id = db_id;
- xlrec.tablespace_id = src_tblspcoid;
+ xlrec->db_id = db_id;
+ xlrec->nspcids = 1;
+ xlrec->tablespace_ids[0] = src_tblspcoid;
XLogBeginInsert();
- XLogRegisterData((char *) &xlrec, sizeof(xl_dbase_drop_rec));
+ XLogRegisterData((char *) &xlrec, xlrec_size);
(void) XLogInsert(RM_DBASE_ID,
XLOG_DBASE_DROP | XLR_SPECIAL_REL_UPDATE);
+ pfree(xlrec);
}
/* Now it's safe to release the database lock */
@@ -1914,6 +1917,8 @@ remove_dbtablespaces(Oid db_id)
Relation rel;
TableScanDesc scan;
HeapTuple tuple;
+ List *dstpaths = NIL;
+ List *spcoids = NIL;
rel = table_open(TableSpaceRelationId, AccessShareLock);
scan = table_beginscan_catalog(rel, 0, NULL);
@@ -1936,31 +1941,39 @@ remove_dbtablespaces(Oid db_id)
pfree(dstpath);
continue;
}
+ dstpaths = lappend(dstpaths, dstpath);
+ spcoids = lappend_oid(spcoids, dsttablespace);
+ }
+
+ table_endscan(scan);
+ table_close(rel, AccessShareLock);
+
+ size_t xlrec_size = (sizeof(xl_dbase_drop_rec) +
+ sizeof(Oid)*list_length(spcoids));
+ xl_dbase_drop_rec *xlrec = palloc(xlrec_size);
+ xlrec->db_id = db_id;
+ xlrec->nspcids = list_length(spcoids);
- if (!rmtree(dstpath, true))
+ int i=0;
+ const ListCell *cell1, *cell2;
+ forboth(cell1, dstpaths, cell2, spcoids)
+ {
+ char *path = lfirst(cell1);
+ if (!rmtree(path, true))
ereport(WARNING,
(errmsg("some useless files may be left behind in old database directory \"%s\"",
- dstpath)));
-
+ path)));
+ pfree(path);
/* Record the filesystem change in XLOG */
- {
- xl_dbase_drop_rec xlrec;
-
- xlrec.db_id = db_id;
- xlrec.tablespace_id = dsttablespace;
-
- XLogBeginInsert();
- XLogRegisterData((char *) &xlrec, sizeof(xl_dbase_drop_rec));
-
- (void) XLogInsert(RM_DBASE_ID,
- XLOG_DBASE_DROP | XLR_SPECIAL_REL_UPDATE);
- }
-
- pfree(dstpath);
+ xlrec->tablespace_ids[i++] = lfirst_oid(cell2);
}
- table_endscan(scan);
- table_close(rel, AccessShareLock);
+ XLogBeginInsert();
+ XLogRegisterData((char *) &xlrec, xlrec_size);
+ (void) XLogInsert(RM_DBASE_ID,
+ XLOG_DBASE_DROP | XLR_SPECIAL_REL_UPDATE);
+
+ pfree(xlrec);
}
/*
@@ -2166,8 +2179,6 @@ dbase_redo(XLogReaderState *record)
xl_dbase_drop_rec *xlrec = (xl_dbase_drop_rec *) XLogRecGetData(record);
char *dst_path;
- dst_path = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
-
if (InHotStandby)
{
/*
@@ -2196,11 +2207,26 @@ dbase_redo(XLogReaderState *record)
/* Clean out the xlog relcache too */
XLogDropDatabase(xlrec->db_id);
+ /*
+ * If we crash after removing directories, we should avoid replaying
+ * WAL records prior to the current WAL record (drop database).
+ * Creating a restartpoint ensures that recovery will start from at
+ * least this point onwards in the event of a crash / immediate
+ * shutdown.
+ */
+ RequestCheckpoint(CHECKPOINT_IMMEDIATE | CHECKPOINT_FORCE |
+ CHECKPOINT_WAIT);
+
/* And remove the physical files */
- if (!rmtree(dst_path, true))
- ereport(WARNING,
- (errmsg("some useless files may be left behind in old database directory \"%s\"",
- dst_path)));
+ int i;
+ for (i = 0; i < xlrec->nspcids; i++)
+ {
+ dst_path = GetDatabasePath(xlrec->db_id, xlrec->tablespace_ids[i]);
+ if (!rmtree(dst_path, true))
+ ereport(WARNING,
+ (errmsg("some useless files may be left behind in old database directory \"%s\"",
+ dst_path)));
+ }
if (InHotStandby)
{
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index d519252aad..d0582a726b 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -221,7 +221,6 @@ extern bool XLOG_DEBUG;
/* These indicate the cause of a checkpoint request */
#define CHECKPOINT_CAUSE_XLOG 0x0080 /* XLOG consumption */
#define CHECKPOINT_CAUSE_TIME 0x0100 /* Elapsed time */
-
/*
* Flag bits for the record being inserted, set using XLogSetRecordFlags().
*/
diff --git a/src/include/commands/dbcommands_xlog.h b/src/include/commands/dbcommands_xlog.h
index 46be8a615a..a1654f25e0 100644
--- a/src/include/commands/dbcommands_xlog.h
+++ b/src/include/commands/dbcommands_xlog.h
@@ -34,7 +34,8 @@ typedef struct xl_dbase_drop_rec
{
/* Records dropping of a single subdirectory incl. contents */
Oid db_id;
- Oid tablespace_id;
+ uint16 nspcids;
+ Oid tablespace_ids[0];
} xl_dbase_drop_rec;
extern void dbase_redo(XLogReaderState *rptr);
--
2.14.3 (Apple Git-98)
[application/octet-stream] v1-0002-Test-to-validate-replay-of-WAL-records-involving-.patch (7.2K, ../../CANXE4TcG68ksC5DCNatr1MZxgvuMpJi_ZfKN2f1ZG2BLyG2H_Q@mail.gmail.com/4-v1-0002-Test-to-validate-replay-of-WAL-records-involving-.patch)
download | inline diff:
From 7c311632576bc8f48ac43a9dd0fa07e86e6b47e6 Mon Sep 17 00:00:00 2001
From: Asim R P <[email protected]>
Date: Thu, 12 Sep 2019 17:19:45 +0530
Subject: [PATCH v1 2/2] Test to validate replay of WAL records involving drop
database
Authored by Paul Guo and Kyotaro Horiguchi.
---
src/test/perl/PostgresNode.pm | 13 +++-
src/test/perl/RecursiveCopy.pm | 33 +++++++++--
src/test/recovery/t/011_crash_recovery.pl | 99 ++++++++++++++++++++++++++++++-
3 files changed, 138 insertions(+), 7 deletions(-)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 270bd6c856..b5dc3a8918 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -546,13 +546,22 @@ target server since it isn't done by default.
sub backup
{
- my ($self, $backup_name) = @_;
+ my ($self, $backup_name, %params) = @_;
my $backup_path = $self->backup_dir . '/' . $backup_name;
my $name = $self->name;
+ my @rest = ();
+
+ if (defined $params{tablespace_mappings})
+ {
+ my @ts_mappings = split(/,/, $params{tablespace_mappings});
+ foreach my $elem (@ts_mappings) {
+ push(@rest, '--tablespace-mapping='.$elem);
+ }
+ }
print "# Taking pg_basebackup $backup_name from node \"$name\"\n";
TestLib::system_or_bail('pg_basebackup', '-D', $backup_path, '-h',
- $self->host, '-p', $self->port, '--no-sync');
+ $self->host, '-p', $self->port, '--no-sync', @rest);
print "# Backup finished\n";
return;
}
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
index baf5d0ac63..514ed90ae7 100644
--- a/src/test/perl/RecursiveCopy.pm
+++ b/src/test/perl/RecursiveCopy.pm
@@ -22,6 +22,7 @@ use warnings;
use Carp;
use File::Basename;
use File::Copy;
+use TestLib;
=pod
@@ -97,14 +98,38 @@ sub _copypath_recurse
# invoke the filter and skip all further operation if it returns false
return 1 unless &$filterfn($curr_path);
- # Check for symlink -- needed only on source dir
- # (note: this will fall through quietly if file is already gone)
- croak "Cannot operate on symlink \"$srcpath\"" if -l $srcpath;
-
# Abort if destination path already exists. Should we allow directories
# to exist already?
croak "Destination path \"$destpath\" already exists" if -e $destpath;
+ # Check for symlink -- needed only on source dir
+ # (note: this will fall through quietly if file is already gone)
+ if (-l $srcpath)
+ {
+ croak "Cannot operate on symlink \"$srcpath\""
+ if ($srcpath !~ /\/(pg_tblspc\/[0-9]+)$/);
+
+ # We have mapped tablespaces. Copy them individually
+ my $linkname = $1;
+ my $tmpdir = TestLib::tempdir;
+ my $dstrealdir = TestLib::perl2host($tmpdir);
+ my $srcrealdir = readlink($srcpath);
+
+ opendir(my $dh, $srcrealdir);
+ while (readdir $dh)
+ {
+ next if (/^\.\.?$/);
+ my $spath = "$srcrealdir/$_";
+ my $dpath = "$dstrealdir/$_";
+
+ copypath($spath, $dpath);
+ }
+ closedir $dh;
+
+ symlink $dstrealdir, $destpath;
+ return 1;
+ }
+
# If this source path is a file, simply copy it to destination with the
# same name and we're done.
if (-f $srcpath)
diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl
index 526a3481fb..5ac7303dd6 100644
--- a/src/test/recovery/t/011_crash_recovery.pl
+++ b/src/test/recovery/t/011_crash_recovery.pl
@@ -15,7 +15,7 @@ if ($Config{osname} eq 'MSWin32')
}
else
{
- plan tests => 3;
+ plan tests => 5;
}
my $node = get_new_node('master');
@@ -66,3 +66,100 @@ is($node->safe_psql('postgres', qq[SELECT txid_status('$xid');]),
'aborted', 'xid is aborted after crash');
$tx->kill_kill;
+
+# Ensure that tablespace removal doesn't cause error while recovering
+# the preceding create database with that tablespace.
+
+my $node_master = get_new_node('master2');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Create tablespace
+my $tspDir_master = TestLib::tempdir;
+my $realTSDir_master = TestLib::perl2host($tspDir_master);
+$node_master->safe_psql('postgres', "CREATE TABLESPACE ts1 LOCATION '$realTSDir_master'");
+
+my $tspDir_standby = TestLib::tempdir;
+my $realTSDir_standby = TestLib::perl2host($tspDir_standby);
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name,
+ tablespace_mappings =>
+ "$realTSDir_master=$realTSDir_standby");
+my $node_standby = get_new_node('standby2');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->start;
+
+# Make sure connection is made
+$node_master->poll_query_until(
+ 'postgres', 'SELECT count(*) = 1 FROM pg_stat_replication');
+
+# Make sure to perform restartpoint after tablespace creation
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->safe_psql('postgres', 'CHECKPOINT');
+
+# Do immediate shutdown just after a sequence of CREAT DATABASE / DROP
+# DATABASE / DROP TABLESPACE. This leaves a CREATE DATBASE WAL record
+# that is to be applied to already-removed tablespace.
+$node_master->safe_psql('postgres',
+ q[CREATE DATABASE db1 WITH TABLESPACE ts1;
+ DROP DATABASE db1;
+ DROP TABLESPACE ts1;]);
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->stop('immediate');
+
+# Should restart ignoring directory creation error.
+is($node_standby->start(fail_ok => 1), 1);
+
+
+# Ensure that tablespace removal doesn't cause error while recovering the
+# preceding alter database set tablespace.
+
+$node_master = get_new_node('master3');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Create tablespace
+$tspDir_master = TestLib::tempdir;
+$realTSDir_master = TestLib::perl2host($tspDir_master);
+mkdir "$realTSDir_master/1";
+mkdir "$realTSDir_master/2";
+$node_master->safe_psql('postgres', "CREATE TABLESPACE ts1 LOCATION '$realTSDir_master/1'");
+$node_master->safe_psql('postgres', "CREATE TABLESPACE ts2 LOCATION '$realTSDir_master/2'");
+
+$tspDir_standby = TestLib::tempdir;
+$realTSDir_standby = TestLib::perl2host($tspDir_standby);
+
+# Take backup
+$backup_name = 'my_backup';
+$node_master->backup($backup_name,
+ tablespace_mappings =>
+ "$realTSDir_master/1=$realTSDir_standby/1,$realTSDir_master/2=$realTSDir_standby/2");
+$node_standby = get_new_node('standby3');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->start;
+
+# Make sure connection is made
+$node_master->poll_query_until(
+ 'postgres', 'SELECT count(*) = 1 FROM pg_stat_replication');
+
+$node_master->safe_psql('postgres', "CREATE DATABASE db1 TABLESPACE ts1");
+
+# Make sure to perform restartpoint after tablespace creation
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->safe_psql('postgres', 'CHECKPOINT');
+
+# Do immediate shutdown ...
+$node_master->safe_psql('postgres',
+ q[ALTER DATABASE db1 SET TABLESPACE ts2;
+ DROP TABLESPACE ts1;]);
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->stop('immediate');
+
+# Should restart ignoring directory creation error.
+is($node_standby->start(fail_ok => 1), 1);
--
2.14.3 (Apple Git-98)
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
@ 2019-09-19 11:59 ` Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
1 sibling, 1 reply; 109+ messages in thread
From: Asim R P @ 2019-09-19 11:59 UTC (permalink / raw)
To: Paul Guo <[email protected]>; Anastasia Lubennikova <[email protected]>; Alexandra Wang <[email protected]>; +Cc: Thomas Munro <[email protected]>; Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
On Thu, Aug 22, 2019 at 6:44 PM Paul Guo <[email protected]> wrote:
>
> Thanks. I updated the patch to v5. It passes install-check testing and
recovery testing.
>
This patch contains one more bug, in addition to what Anastasia has found.
If the test case in the patch is tweaked slightly, as follows, the standby
crashes due to PANIC.
--- a/src/test/recovery/t/011_crash_recovery.pl
+++ b/src/test/recovery/t/011_crash_recovery.pl
@@ -147,8 +147,6 @@ $node_standby->start;
$node_master->poll_query_until(
'postgres', 'SELECT count(*) = 1 FROM pg_stat_replication');
-$node_master->safe_psql('postgres', "CREATE DATABASE db1 TABLESPACE ts1");
-
# Make sure to perform restartpoint after tablespace creation
$node_master->wait_for_catchup($node_standby, 'replay',
$node_master->lsn('replay'));
@@ -156,7 +154,8 @@ $node_standby->safe_psql('postgres', 'CHECKPOINT');
# Do immediate shutdown ...
$node_master->safe_psql('postgres',
- q[ALTER DATABASE db1 SET
TABLESPACE ts2;
+ q[CREATE DATABASE db1
TABLESPACE ts1;
+ ALTER DATABASE db1 SET
TABLESPACE ts2;
DROP TABLESPACE ts1;]);
$node_master->wait_for_catchup($node_standby, 'replay',
$node_master->lsn('replay'));
Notice the create additional create database in the above change. That
causes the same tablespace directory (ts1) logged twice in the list of
missing directories. At the end of crash recovery, there is one unmatched
entry in the missing dirs list and the standby PANICs.
Please find attached a couple of tests that are built on top of what was
already written by Paul, Kyotaro. The patch includes a test to demonstrate
the above mentioned failure and a test case that my friend Alexandra wrote
to implement the archive recovery scenario noted by Anastasia.
In order to fix the test failures, we need to distinguish between a missing
database directory and a missing tablespace directory. And also add logic
to forget missing directories during tablespace drop. I am working on it.
Asim
Attachments:
[application/octet-stream] 0001-Tests-for-replay-of-create-database-operation-on-sta.patch (7.7K, ../../CANXE4TdHyvpOyHVnw=goiazD7d4CiR7=MxtsvSUDphmasSVzvg@mail.gmail.com/3-0001-Tests-for-replay-of-create-database-operation-on-sta.patch)
download | inline diff:
From 26b34d171d2bb185a2d927b88a77a4b0cacb0c88 Mon Sep 17 00:00:00 2001
From: Asim R P <[email protected]>
Date: Thu, 19 Sep 2019 17:14:55 +0530
Subject: [PATCH] Tests for replay of create database operation on standby
A couple of tests to demonstrate that standby fails to replay a create
database WAL record during crash recovery, if one or more of underlying
directories are missing from the file system. This can happen if a drop
tablespace or drop database WAL record has been replayed in archive
recovery, before a crash. And then the create database record happens
to be replayed again during crash recovery. The failures indicate bugs
that need to be fixed.
The first test, TEST 4, performs several DDL operations resulting in a
database directory being removed, along with a few create database
operations. It expects crash recovery to succeed because for each
missing directory encountered during create database replay, a matching
drop tablespace or drop database WAL record is found later.
Second test, TEST 5, validates that a standby rightfully aborts replay
during archive recovery, if a missing directory is encountered when
replaying create database WAL record.
These tests have been proposed and implemented in various ways by
Alexandra, Anastasia, Kyotaro, Paul and me.
---
src/test/recovery/t/011_crash_recovery.pl | 150 +++++++++++++++++++++++++++++-
1 file changed, 149 insertions(+), 1 deletion(-)
diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl
index 526a3481fb..1cea17c7d4 100644
--- a/src/test/recovery/t/011_crash_recovery.pl
+++ b/src/test/recovery/t/011_crash_recovery.pl
@@ -6,6 +6,7 @@ use warnings;
use PostgresNode;
use TestLib;
use Test::More;
+use File::Path qw(rmtree);
use Config;
if ($Config{osname} eq 'MSWin32')
{
@@ -15,7 +16,7 @@ if ($Config{osname} eq 'MSWin32')
}
else
{
- plan tests => 3;
+ plan tests => 5;
}
my $node = get_new_node('master');
@@ -66,3 +67,150 @@ is($node->safe_psql('postgres', qq[SELECT txid_status('$xid');]),
'aborted', 'xid is aborted after crash');
$tx->kill_kill;
+
+# TEST 4
+#
+# Ensure that a missing tablespace directory during crash recovery on
+# a standby is hangled correctly. The standby should finish crash
+# recovery successfully because a matching drop database record is
+# found in the WAL. The following scnearios are covered:
+#
+# 1. Create a database against a user-defined tablespace then drop the
+# tablespace.
+#
+# 2. Move a database from source tablespace to target tablespace then
+# drop the source tablespace.
+#
+# 3. Create a datbase from another database as template then drop the
+# template database.
+
+my $node_master = get_new_node('master2');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Create tablespace
+my $dropme_ts_master = TestLib::tempdir;
+$dropme_ts_master = TestLib::perl2host($dropme_ts_master);
+my $source_ts_master = TestLib::tempdir;
+$source_ts_master = TestLib::perl2host($source_ts_master);
+my $target_ts_master = TestLib::tempdir;
+$target_ts_master = TestLib::perl2host($target_ts_master);
+
+$node_master->safe_psql('postgres',
+ qq[CREATE TABLESPACE dropme_ts location '$dropme_ts_master';
+ CREATE TABLESPACE source_ts location '$source_ts_master';
+ CREATE TABLESPACE target_ts location '$target_ts_master';
+ CREATE DATABASE template_db IS_TEMPLATE = true;]);
+
+my $dropme_ts_standby = TestLib::tempdir;
+$dropme_ts_standby = TestLib::perl2host($dropme_ts_standby);
+my $source_ts_standby = TestLib::tempdir;
+$source_ts_standby = TestLib::perl2host($source_ts_standby);
+my $target_ts_standby = TestLib::tempdir;
+$target_ts_standby = TestLib::perl2host($target_ts_standby);
+
+# Take backup
+my $backup_name = 'my_backup';
+my $ts_mapping = "$dropme_ts_master=$dropme_ts_standby," .
+ "$source_ts_master=$source_ts_standby," .
+ "$target_ts_master=$target_ts_standby";
+$node_master->backup($backup_name, tablespace_mappings => $ts_mapping);
+
+my $node_standby = get_new_node('standby2');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->start;
+
+# Make sure connection is made
+$node_master->poll_query_until(
+ 'postgres', 'SELECT count(*) = 1 FROM pg_stat_replication');
+
+# Make sure to perform restartpoint after tablespace creation
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->safe_psql('postgres', 'CHECKPOINT');
+
+# Do immediate shutdown just after a sequence of CREAT DATABASE / DROP
+# DATABASE / DROP TABLESPACE. This causes CREATE DATBASE WAL records
+# to be applied to already-removed directories.
+$node_master->safe_psql('postgres',
+ q[CREATE DATABASE dropme_db WITH TABLESPACE dropme_ts;
+ CREATE DATABASE moveme_db TABLESPACE source_ts;
+ ALTER DATABASE moveme_db SET TABLESPACE target_ts;
+ DROP DATABASE dropme_db;
+ CREATE DATABASE newdb TEMPLATE template_db;
+ ALTER DATABASE template_db IS_TEMPLATE = false;
+ DROP TABLESPACE source_ts;
+ DROP TABLESPACE dropme_ts;
+ DROP DATABASE template_db;]);
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->stop('immediate');
+
+# Should restart ignoring directory creation error.
+is($node_standby->start(fail_ok => 1), 1);
+
+# TEST 5
+#
+# Ensure that a missing tablespace directory during create database
+# replay immediately causes panic if the standby has already reached
+# consistent state (archive recovery is in progress).
+
+$node_master = get_new_node('master4');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Create tablespace
+my $tspDir_master = TestLib::tempdir;
+my $realTSDir_master = TestLib::perl2host($tspDir_master);
+$node_master->safe_psql('postgres', "CREATE TABLESPACE ts1 LOCATION '$realTSDir_master'");
+$node_master->safe_psql('postgres', "CREATE DATABASE db1 TABLESPACE ts1");
+
+my $tspDir_standby = TestLib::tempdir;
+my $realTSDir_standby = TestLib::perl2host($tspDir_standby);
+
+# Take backup
+$backup_name = 'my_backup';
+$node_master->backup($backup_name,
+ tablespace_mappings =>
+ "$realTSDir_master=$realTSDir_standby");
+$node_standby = get_new_node('standby4');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->start;
+
+# Make sure standby reached consistency and starts accepting connections
+$node_standby->poll_query_until('postgres', 'SELECT 1', '1');
+
+# Remove standby tablespace directory so it will be missing when
+# replay resumes.
+#
+# XXX For some reason, the tablespace mapping is not honored and the
+# standby ends up getting a different temp dir than what was specified
+# in the tablepsace mapping. So get the tablespace directory by
+# querying standby.
+$realTSDir_standby = $node_standby->safe_psql(
+ 'postgres',
+ "select pg_tablespace_location(oid) from pg_tablespace where spcname = 'ts1'");
+rmtree($realTSDir_standby);
+
+# Create a database in the tablespace and a table in default tablespace
+$node_master->safe_psql('postgres',
+ q[CREATE TABLE should_not_replay_insertion(a int);
+ CREATE DATABASE db2 WITH TABLESPACE ts1;
+ INSERT INTO should_not_replay_insertion VALUES (1);]);
+
+# Standby should fail and should not silently skip replaying the wal
+if ($node_master->poll_query_until_params(
+ 'postgres',
+ 'SELECT count(*) = 0 FROM pg_stat_replication',
+ timeout => 5) == 1)
+{
+ pass('standby failed as expected');
+ # We know that the standby has failed. Setting its pid to
+ # undefined avoids error when PostgreNode module tries to stop the
+ # standby node as part of tear_down sequence.
+ $node_standby->{_pid} = undef;
+}
+else
+{
+ fail('standby did not fail within 5 seconds');
+}
--
2.14.3 (Apple Git-98)
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
@ 2019-09-20 12:23 ` Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Asim R P @ 2019-09-20 12:23 UTC (permalink / raw)
To: Paul Guo <[email protected]>; Anastasia Lubennikova <[email protected]>; Alexandra Wang <[email protected]>; +Cc: Thomas Munro <[email protected]>; Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
On Thu, Sep 19, 2019 at 5:29 PM Asim R P <[email protected]> wrote:
>
> In order to fix the test failures, we need to distinguish between a
missing database directory and a missing tablespace directory. And also
add logic to forget missing directories during tablespace drop. I am
working on it.
Please find attached a solution that builds on what Paul has propose. A
hash table, similar to the invalid page hash table is used to track missing
directory references. A missing directory may be a tablespace or a
database, based on whether the tablespace is found missing or the source
database is found missing. The crash recovery succeeds if the hash table
is empty at the end.
Asim
Attachments:
[application/octet-stream] v6-0001-Support-node-initialization-from-backup-with-tabl.patch (2.1K, ../../CANXE4Td1udkoCX5tyBUKiax5r_qFZDwR6iG1Ymm3t2CfW9zTuw@mail.gmail.com/3-v6-0001-Support-node-initialization-from-backup-with-tabl.patch)
download | inline diff:
From f676cd4c9d698b0f9e4f86d6251253ac6ddda36f Mon Sep 17 00:00:00 2001
From: Asim R P <[email protected]>
Date: Fri, 20 Sep 2019 17:31:25 +0530
Subject: [PATCH v6 1/3] Support node initialization from backup with
tablespaces
User defined tablespaces appear as symlinks in in the backup. This
commit tweaks recursive copy subroutine to allow for symlinks specific
to tablespaces.
Authored by Kyotaro
---
src/test/perl/RecursiveCopy.pm | 33 +++++++++++++++++++++++++++++----
1 file changed, 29 insertions(+), 4 deletions(-)
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
index baf5d0ac63..514ed90ae7 100644
--- a/src/test/perl/RecursiveCopy.pm
+++ b/src/test/perl/RecursiveCopy.pm
@@ -22,6 +22,7 @@ use warnings;
use Carp;
use File::Basename;
use File::Copy;
+use TestLib;
=pod
@@ -97,14 +98,38 @@ sub _copypath_recurse
# invoke the filter and skip all further operation if it returns false
return 1 unless &$filterfn($curr_path);
- # Check for symlink -- needed only on source dir
- # (note: this will fall through quietly if file is already gone)
- croak "Cannot operate on symlink \"$srcpath\"" if -l $srcpath;
-
# Abort if destination path already exists. Should we allow directories
# to exist already?
croak "Destination path \"$destpath\" already exists" if -e $destpath;
+ # Check for symlink -- needed only on source dir
+ # (note: this will fall through quietly if file is already gone)
+ if (-l $srcpath)
+ {
+ croak "Cannot operate on symlink \"$srcpath\""
+ if ($srcpath !~ /\/(pg_tblspc\/[0-9]+)$/);
+
+ # We have mapped tablespaces. Copy them individually
+ my $linkname = $1;
+ my $tmpdir = TestLib::tempdir;
+ my $dstrealdir = TestLib::perl2host($tmpdir);
+ my $srcrealdir = readlink($srcpath);
+
+ opendir(my $dh, $srcrealdir);
+ while (readdir $dh)
+ {
+ next if (/^\.\.?$/);
+ my $spath = "$srcrealdir/$_";
+ my $dpath = "$dstrealdir/$_";
+
+ copypath($spath, $dpath);
+ }
+ closedir $dh;
+
+ symlink $dstrealdir, $destpath;
+ return 1;
+ }
+
# If this source path is a file, simply copy it to destination with the
# same name and we're done.
if (-f $srcpath)
--
2.14.3 (Apple Git-98)
[application/octet-stream] v6-0002-Tests-to-replay-create-database-operation-on-stan.patch (9.8K, ../../CANXE4Td1udkoCX5tyBUKiax5r_qFZDwR6iG1Ymm3t2CfW9zTuw@mail.gmail.com/4-v6-0002-Tests-to-replay-create-database-operation-on-stan.patch)
download | inline diff:
From 45f2eb5ddd2c819942cc299e97eb017cd04b5181 Mon Sep 17 00:00:00 2001
From: Asim R P <[email protected]>
Date: Fri, 20 Sep 2019 17:34:19 +0530
Subject: [PATCH v6 2/3] Tests to replay create database operation on standby
The tests demonstrate that standby fails to replay a create database
WAL record during crash recovery, if one or more of underlying
directories are missing from the file system. This can happen if a
drop tablespace or drop database WAL record has been replayed in
archive recovery, before a crash. And then the create database record
happens to be replayed again during crash recovery. The failures
indicate bugs that need to be fixed.
The first test, TEST 4, performs several DDL operations resulting in a
database directory being removed, along with a few create database
operations. It expects crash recovery to succeed because for each
missing directory encountered during create database replay, a matching
drop tablespace or drop database WAL record is found later.
Second test, TEST 5, validates that a standby rightfully aborts replay
during archive recovery, if a missing directory is encountered when
replaying create database WAL record.
These tests have been proposed and implemented in various ways by
Alexandra, Anastasia, Kyotaro, Paul and me.
---
src/test/perl/PostgresNode.pm | 31 +++++-
src/test/recovery/t/011_crash_recovery.pl | 152 +++++++++++++++++++++++++++++-
2 files changed, 177 insertions(+), 6 deletions(-)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 270bd6c856..959fa24bba 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -546,13 +546,22 @@ target server since it isn't done by default.
sub backup
{
- my ($self, $backup_name) = @_;
+ my ($self, $backup_name, %params) = @_;
my $backup_path = $self->backup_dir . '/' . $backup_name;
my $name = $self->name;
+ my @rest = ();
+
+ if (defined $params{tablespace_mappings})
+ {
+ my @ts_mappings = split(/,/, $params{tablespace_mappings});
+ foreach my $elem (@ts_mappings) {
+ push(@rest, '--tablespace-mapping='.$elem);
+ }
+ }
print "# Taking pg_basebackup $backup_name from node \"$name\"\n";
TestLib::system_or_bail('pg_basebackup', '-D', $backup_path, '-h',
- $self->host, '-p', $self->port, '--no-sync');
+ $self->host, '-p', $self->port, '--no-sync', @rest);
print "# Backup finished\n";
return;
}
@@ -1550,9 +1559,21 @@ sub poll_query_until
$expected = 't' unless defined($expected); # default value
+ $self->poll_query_until_params(
+ $dbname, $query,
+ expected => $expected, timeout => 180);
+}
+
+sub poll_query_until_params
+{
+ my ($self, $dbname, $query, %params) = @_;
+
+ $params{expected} = 't' unless defined($params{expected});
+ $params{timeout} = 180 unless defined($params{timeout});
+
my $cmd = [ 'psql', '-XAt', '-c', $query, '-d', $self->connstr($dbname) ];
my ($stdout, $stderr);
- my $max_attempts = 180 * 10;
+ my $max_attempts = $params{timeout} * 10;
my $attempts = 0;
while ($attempts < $max_attempts)
@@ -1562,7 +1583,7 @@ sub poll_query_until
chomp($stdout);
$stdout =~ s/\r//g if $TestLib::windows_os;
- if ($stdout eq $expected)
+ if ($stdout eq $params{expected})
{
return 1;
}
@@ -1580,7 +1601,7 @@ sub poll_query_until
diag qq(poll_query_until timed out executing this query:
$query
expecting this output:
-$expected
+$params{expected}
last actual query output:
$stdout
with stderr:
diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl
index 526a3481fb..23397232b8 100644
--- a/src/test/recovery/t/011_crash_recovery.pl
+++ b/src/test/recovery/t/011_crash_recovery.pl
@@ -6,6 +6,7 @@ use warnings;
use PostgresNode;
use TestLib;
use Test::More;
+use File::Path qw(rmtree);
use Config;
if ($Config{osname} eq 'MSWin32')
{
@@ -15,7 +16,7 @@ if ($Config{osname} eq 'MSWin32')
}
else
{
- plan tests => 3;
+ plan tests => 5;
}
my $node = get_new_node('master');
@@ -66,3 +67,152 @@ is($node->safe_psql('postgres', qq[SELECT txid_status('$xid');]),
'aborted', 'xid is aborted after crash');
$tx->kill_kill;
+
+# TEST 4
+#
+# Ensure that a missing tablespace directory during crash recovery on
+# a standby is hangled correctly. The standby should finish crash
+# recovery successfully because a matching drop database record is
+# found in the WAL. The following scnearios are covered:
+#
+# 1. Create a database against a user-defined tablespace then drop the
+# tablespace.
+#
+# 2. Move a database from source tablespace to target tablespace then
+# drop the source tablespace.
+#
+# 3. Create a datbase from another database as template then drop the
+# template database.
+
+my $node_master = get_new_node('master2');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Create tablespace
+my $dropme_ts_master = TestLib::tempdir;
+$dropme_ts_master = TestLib::perl2host($dropme_ts_master);
+my $source_ts_master = TestLib::tempdir;
+$source_ts_master = TestLib::perl2host($source_ts_master);
+my $target_ts_master = TestLib::tempdir;
+$target_ts_master = TestLib::perl2host($target_ts_master);
+
+$node_master->safe_psql('postgres',
+ qq[CREATE TABLESPACE dropme_ts location '$dropme_ts_master';
+ CREATE TABLESPACE source_ts location '$source_ts_master';
+ CREATE TABLESPACE target_ts location '$target_ts_master';
+ CREATE DATABASE template_db IS_TEMPLATE = true;]);
+
+my $dropme_ts_standby = TestLib::tempdir;
+$dropme_ts_standby = TestLib::perl2host($dropme_ts_standby);
+my $source_ts_standby = TestLib::tempdir;
+$source_ts_standby = TestLib::perl2host($source_ts_standby);
+my $target_ts_standby = TestLib::tempdir;
+$target_ts_standby = TestLib::perl2host($target_ts_standby);
+
+# Take backup
+my $backup_name = 'my_backup';
+my $ts_mapping = "$dropme_ts_master=$dropme_ts_standby," .
+ "$source_ts_master=$source_ts_standby," .
+ "$target_ts_master=$target_ts_standby";
+$node_master->backup($backup_name, tablespace_mappings => $ts_mapping);
+
+my $node_standby = get_new_node('standby2');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->start;
+
+# Make sure connection is made
+$node_master->poll_query_until(
+ 'postgres', 'SELECT count(*) = 1 FROM pg_stat_replication');
+
+# Make sure to perform restartpoint after tablespace creation
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->safe_psql('postgres', 'CHECKPOINT');
+
+# Do immediate shutdown just after a sequence of CREAT DATABASE / DROP
+# DATABASE / DROP TABLESPACE. This causes CREATE DATBASE WAL records
+# to be applied to already-removed directories.
+$node_master->safe_psql('postgres',
+ q[CREATE DATABASE dropme_db1 WITH TABLESPACE dropme_ts;
+ CREATE DATABASE dropme_db2 WITH TABLESPACE dropme_ts;
+ CREATE DATABASE moveme_db TABLESPACE source_ts;
+ ALTER DATABASE moveme_db SET TABLESPACE target_ts;
+ DROP DATABASE dropme_db1;
+ CREATE DATABASE newdb TEMPLATE template_db;
+ ALTER DATABASE template_db IS_TEMPLATE = false;
+ DROP TABLESPACE source_ts;
+ DROP DATABASE dropme_db2;
+ DROP TABLESPACE dropme_ts;
+ DROP DATABASE template_db;]);
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->stop('immediate');
+
+# Should restart ignoring directory creation error.
+is($node_standby->start(fail_ok => 1), 1);
+
+# TEST 5
+#
+# Ensure that a missing tablespace directory during create database
+# replay immediately causes panic if the standby has already reached
+# consistent state (archive recovery is in progress).
+
+$node_master = get_new_node('master3');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Create tablespace
+my $ts_master = TestLib::tempdir;
+$ts_master = TestLib::perl2host($ts_master);
+$node_master->safe_psql('postgres', "CREATE TABLESPACE ts1 LOCATION '$ts_master'");
+$node_master->safe_psql('postgres', "CREATE DATABASE db1 TABLESPACE ts1");
+
+my $ts_standby = TestLib::tempdir("standby");
+$ts_standby = TestLib::perl2host($ts_standby);
+
+# Take backup
+$backup_name = 'my_backup';
+$node_master->backup($backup_name,
+ tablespace_mappings =>
+ "$ts_master=$ts_standby");
+$node_standby = get_new_node('standby3');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->start;
+
+# Make sure standby reached consistency and starts accepting connections
+$node_standby->poll_query_until('postgres', 'SELECT 1', '1');
+
+# Remove standby tablespace directory so it will be missing when
+# replay resumes.
+#
+# The tablespace mapping is lost when the standby node is initialized
+# from basebackup because RecursiveCopy::copypath creates a new temp
+# directory for each tablspace symlink found in backup. We must
+# obtain the correct tablespace directory by querying standby.
+$ts_standby = $node_standby->safe_psql(
+ 'postgres',
+ "select pg_tablespace_location(oid) from pg_tablespace where spcname = 'ts1'");
+rmtree($ts_standby);
+
+# Create a database in the tablespace and a table in default tablespace
+$node_master->safe_psql('postgres',
+ q[CREATE TABLE should_not_replay_insertion(a int);
+ CREATE DATABASE db2 WITH TABLESPACE ts1;
+ INSERT INTO should_not_replay_insertion VALUES (1);]);
+
+# Standby should fail and should not silently skip replaying the wal
+if ($node_master->poll_query_until_params(
+ 'postgres',
+ 'SELECT count(*) = 0 FROM pg_stat_replication',
+ timeout => 5) == 1)
+{
+ pass('standby failed as expected');
+ # We know that the standby has failed. Setting its pid to
+ # undefined avoids error when PostgreNode module tries to stop the
+ # standby node as part of tear_down sequence.
+ $node_standby->{_pid} = undef;
+}
+else
+{
+ fail('standby did not fail within 5 seconds');
+}
--
2.14.3 (Apple Git-98)
[application/octet-stream] v6-0003-Fix-replay-of-create-database-records-on-standby.patch (12.4K, ../../CANXE4Td1udkoCX5tyBUKiax5r_qFZDwR6iG1Ymm3t2CfW9zTuw@mail.gmail.com/5-v6-0003-Fix-replay-of-create-database-records-on-standby.patch)
download | inline diff:
From ee35455705ac7ea81b5db6e090591971d277c5a8 Mon Sep 17 00:00:00 2001
From: Asim R P <[email protected]>
Date: Fri, 20 Sep 2019 17:43:25 +0530
Subject: [PATCH v6 3/3] Fix replay of create database records on standby
Crash recovery on standby may encounter missing directories when
replaying create database WAL records. Prior to this patch, the
standby would fail to recover in such a case. However, the
directories could be legitimately missing. Consider a sequence of WAL
records as follows:
CREATE DATABASE
DROP DATABASE
DROP TABLESPACE
If, after replaying the last WAL record and removing the tablespace
directory, the standby crashes and has to replay the create database
record again, the crash recovery must be able to move on.
This patch adds mechanism similar to invalid page hash table, to track
missing directories during crash recovery. If all the missing
directory references are matched with corresponding drop records at
the end of crash recovery, the standby can safely enter archive
recovery.
Bug identified by Paul.
Authored by Paul, Kyotaro and me.
Discussion: CAEET0ZGx9AvioViLf7nbR_8tH9-%3D27DN5xWJ2P9-ROH16e4JUA%40mail.gmail.com
---
src/backend/access/rmgrdesc/dbasedesc.c | 14 ++--
src/backend/access/transam/xlog.c | 6 ++
src/backend/access/transam/xlogutils.c | 130 ++++++++++++++++++++++++++++++++
src/backend/commands/dbcommands.c | 53 +++++++++++++
src/backend/commands/tablespace.c | 3 +
src/include/access/xlogutils.h | 4 +
src/include/commands/dbcommands.h | 2 +
7 files changed, 207 insertions(+), 5 deletions(-)
diff --git a/src/backend/access/rmgrdesc/dbasedesc.c b/src/backend/access/rmgrdesc/dbasedesc.c
index c7d60ce10d..35092ffb0e 100644
--- a/src/backend/access/rmgrdesc/dbasedesc.c
+++ b/src/backend/access/rmgrdesc/dbasedesc.c
@@ -23,21 +23,25 @@ dbase_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ char *dbpath1, *dbpath2;
if (info == XLOG_DBASE_CREATE)
{
xl_dbase_create_rec *xlrec = (xl_dbase_create_rec *) rec;
- appendStringInfo(buf, "copy dir %u/%u to %u/%u",
- xlrec->src_tablespace_id, xlrec->src_db_id,
- xlrec->tablespace_id, xlrec->db_id);
+ dbpath1 = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
+ dbpath2 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
+ appendStringInfo(buf, "copy dir %s to %s", dbpath1, dbpath2);
+ pfree(dbpath2);
+ pfree(dbpath1);
}
else if (info == XLOG_DBASE_DROP)
{
xl_dbase_drop_rec *xlrec = (xl_dbase_drop_rec *) rec;
- appendStringInfo(buf, "dir %u/%u",
- xlrec->tablespace_id, xlrec->db_id);
+ dbpath1 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
+ appendStringInfo(buf, "dir %s", dbpath1);
+ pfree(dbpath1);
}
}
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index b7ff004234..749e8d5961 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -7858,6 +7858,12 @@ CheckRecoveryConsistency(void)
*/
XLogCheckInvalidPages();
+ /*
+ * Check if the XLOG sequence contained any unresolved references to
+ * missing directories.
+ */
+ XLogCheckMissingDirs();
+
reachedConsistency = true;
ereport(LOG,
(errmsg("consistent recovery state reached at %X/%X",
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index 1fc39333f1..e1384afe30 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -56,6 +56,136 @@ typedef struct xl_invalid_page
static HTAB *invalid_page_tab = NULL;
+/*
+ * If a create database WAL record is being replayed more than once during
+ * crash recovery on a standby, it is possible that either the tablespace
+ * directory or the template database directory is missing. This happens when
+ * the directories are removed by replay of subsequent drop records. Note
+ * that this problem happens only on standby and not on master. On master, a
+ * checkpoint is created at the end of create database operation. On standby,
+ * however, such a strategy (creating restart points during replay) is not
+ * viable because it will slow down WAL replay.
+ *
+ * The alternative is to track references to each missing directory
+ * encountered when performing crash recovery in the following hash table.
+ * Similar to invalid page table above, the expectation is that each missing
+ * directory entry should be matched with a drop database or drop tablespace
+ * WAL record by the end of crash recovery.
+ */
+typedef struct xl_missing_dir_key
+{
+ Oid spcNode;
+ Oid dbNode;
+} xl_missing_dir_key;
+
+typedef struct xl_missing_dir
+{
+ xl_missing_dir_key key;
+ char path[MAXPGPATH];
+} xl_missing_dir;
+
+static HTAB *missing_dir_tab = NULL;
+
+void
+XLogLogMissingDir(Oid spcNode, Oid dbNode, char *path)
+{
+ xl_missing_dir_key key;
+ bool found;
+ xl_missing_dir *entry;
+
+ /*
+ * Database OID may be invalid but tablespace OID must be valid. If
+ * dbNode is InvalidOid, we are logging a missing tablespace directory,
+ * otherwise we are logging a missing database directory.
+ */
+ Assert(OidIsValid(spcNode));
+
+ if (reachedConsistency)
+ elog(PANIC, "cannot find directory %s tablespace %d database %d",
+ path, spcNode, dbNode);
+
+ if (missing_dir_tab == NULL)
+ {
+ /* create hash table when first needed */
+ HASHCTL ctl;
+
+ memset(&ctl, 0, sizeof(ctl));
+ ctl.keysize = sizeof(xl_missing_dir_key);
+ ctl.entrysize = sizeof(xl_missing_dir);
+
+ missing_dir_tab = hash_create("XLOG missing directory table",
+ 100,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS);
+ }
+
+ key.spcNode = spcNode;
+ key.dbNode = dbNode;
+
+ entry = hash_search(missing_dir_tab, &key, HASH_ENTER, &found);
+
+ if (found)
+ elog(DEBUG2, "missing directory %s tablespace %d database %d already exists: %s",
+ path, spcNode, dbNode, entry->path);
+ else
+ {
+ strlcpy(entry->path, path, sizeof(entry->path));
+ elog(DEBUG2, "logged missing dir %s tablespace %d database %d",
+ path, spcNode, dbNode);
+ }
+}
+
+void
+XLogForgetMissingDir(Oid spcNode, Oid dbNode, char *path)
+{
+ xl_missing_dir_key key;
+
+ key.spcNode = spcNode;
+ key.dbNode = dbNode;
+
+ /* Database OID may be invalid but tablespace OID must be valid. */
+ Assert(OidIsValid(spcNode));
+
+ if (missing_dir_tab == NULL)
+ return;
+
+ if (hash_search(missing_dir_tab, &key, HASH_REMOVE, NULL) == NULL)
+ elog(DEBUG2, "dir %s tablespace %d database %d is not missing",
+ path, spcNode, dbNode);
+ else
+ elog(DEBUG2, "forgot missing dir %s for tablespace %d database %d",
+ path, spcNode, dbNode);
+}
+
+/*
+ * This is called at the end of crash recovery, before entering archive
+ * recovery on a standby. PANIC if the hash table is not empty.
+ */
+void
+XLogCheckMissingDirs(void)
+{
+ HASH_SEQ_STATUS status;
+ xl_missing_dir *hentry;
+ bool foundone = false;
+
+ if (missing_dir_tab == NULL)
+ return; /* nothing to do */
+
+ hash_seq_init(&status, missing_dir_tab);
+
+ while ((hentry = (xl_missing_dir *) hash_seq_search(&status)) != NULL)
+ {
+ elog(WARNING, "missing directory \"%s\" tablespace %d database %d",
+ hentry->path, hentry->key.spcNode, hentry->key.dbNode);
+ foundone = true;
+ }
+
+ if (foundone)
+ elog(PANIC, "WAL contains references to missing directories");
+
+ hash_destroy(missing_dir_tab);
+ missing_dir_tab = NULL;
+}
/* Report a reference to an invalid page */
static void
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index 95881a8550..5388800660 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -45,6 +45,7 @@
#include "commands/defrem.h"
#include "commands/seclabel.h"
#include "commands/tablespace.h"
+#include "common/file_perm.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "pgstat.h"
@@ -2129,7 +2130,9 @@ dbase_redo(XLogReaderState *record)
xl_dbase_create_rec *xlrec = (xl_dbase_create_rec *) XLogRecGetData(record);
char *src_path;
char *dst_path;
+ char *parent_path;
struct stat st;
+ bool skip = false;
src_path = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
dst_path = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
@@ -2147,6 +2150,54 @@ dbase_redo(XLogReaderState *record)
(errmsg("some useless files may be left behind in old database directory \"%s\"",
dst_path)));
}
+ else
+ {
+ /*
+ * It is possible that drop tablespace record appearing later in
+ * the WAL as already been replayed. That means we are replaying
+ * the create database record second time, as part of crash
+ * recovery. In that case, the tablespace directory has already
+ * been removed and the create database operation cannot be
+ * replayed. We should skip the replay but remember the missing
+ * tablespace directory, to be matched with a drop tablespace
+ * record later.
+ */
+ parent_path = pstrdup(dst_path);
+ get_parent_directory(parent_path);
+ if (!(stat(parent_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
+ XLogLogMissingDir(xlrec->tablespace_id, InvalidOid, dst_path);
+ skip = true;
+ ereport(WARNING,
+ (errmsg("skipping create database WAL record"),
+ errdetail("Target tablespace \"%s\" not found. We "
+ "expect to encounter a WAL record that "
+ "removes this directory before reaching "
+ "consistent state.", parent_path)));
+ }
+ pfree(parent_path);
+ }
+
+ /*
+ * Source directory may be missing. E.g. the template database used
+ * for creating this database may have been dropped, due to reasons
+ * noted above. Moving a database from one tablespace may also be a
+ * partner in the crime.
+ */
+ if (!(stat(src_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
+ XLogLogMissingDir(xlrec->src_tablespace_id, xlrec->src_db_id, src_path);
+ skip = true;
+ ereport(WARNING,
+ (errmsg("skipping create database WAL record"),
+ errdetail("Source database \"%s\" not found. We expect "
+ "to encounter a WAL record that removes this "
+ "directory before reaching consistent state.",
+ src_path)));
+ }
+
+ if (skip)
+ return;
/*
* Force dirty buffers out to disk, to ensure source database is
@@ -2202,6 +2253,8 @@ dbase_redo(XLogReaderState *record)
(errmsg("some useless files may be left behind in old database directory \"%s\"",
dst_path)));
+ XLogForgetMissingDir(xlrec->tablespace_id, xlrec->db_id, dst_path);
+
if (InHotStandby)
{
/*
diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c
index 84efb414d8..0d553c39c8 100644
--- a/src/backend/commands/tablespace.c
+++ b/src/backend/commands/tablespace.c
@@ -58,6 +58,7 @@
#include "access/xact.h"
#include "access/xlog.h"
#include "access/xloginsert.h"
+#include "access/xlogutils.h"
#include "catalog/catalog.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
@@ -1517,6 +1518,8 @@ tblspc_redo(XLogReaderState *record)
{
xl_tblspc_drop_rec *xlrec = (xl_tblspc_drop_rec *) XLogRecGetData(record);
+ XLogForgetMissingDir(xlrec->ts_id, InvalidOid, "");
+
/*
* If we issued a WAL record for a drop tablespace it implies that
* there were no files in it at all when the DROP was done. That means
diff --git a/src/include/access/xlogutils.h b/src/include/access/xlogutils.h
index 4105b59904..7034b691bd 100644
--- a/src/include/access/xlogutils.h
+++ b/src/include/access/xlogutils.h
@@ -23,6 +23,10 @@ extern void XLogDropDatabase(Oid dbid);
extern void XLogTruncateRelation(RelFileNode rnode, ForkNumber forkNum,
BlockNumber nblocks);
+extern void XLogLogMissingDir(Oid spcNode, Oid dbNode, char *path);
+extern void XLogForgetMissingDir(Oid spcNode, Oid dbNode, char *path);
+extern void XLogCheckMissingDirs(void);
+
/* Result codes for XLogReadBufferForRedo[Extended] */
typedef enum
{
diff --git a/src/include/commands/dbcommands.h b/src/include/commands/dbcommands.h
index 154c8157ee..26e96b8957 100644
--- a/src/include/commands/dbcommands.h
+++ b/src/include/commands/dbcommands.h
@@ -19,6 +19,8 @@
#include "lib/stringinfo.h"
#include "nodes/parsenodes.h"
+extern void CheckMissingDirs4DbaseRedo(void);
+
extern Oid createdb(ParseState *pstate, const CreatedbStmt *stmt);
extern void dropdb(const char *dbname, bool missing_ok);
extern ObjectAddress RenameDatabase(const char *oldname, const char *newname);
--
2.14.3 (Apple Git-98)
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
@ 2019-12-26 14:45 ` Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Anastasia Lubennikova @ 2019-12-26 14:45 UTC (permalink / raw)
To: Asim R P <[email protected]>; Paul Guo <[email protected]>; Alexandra Wang <[email protected]>; +Cc: Thomas Munro <[email protected]>; Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
20.09.2019 15:23, Asim R P wrote:
> On Thu, Sep 19, 2019 at 5:29 PM Asim R P <[email protected]
> <mailto:[email protected]>> wrote:
> >
> > In order to fix the test failures, we need to distinguish between a
> missing database directory and a missing tablespace directory. And
> also add logic to forget missing directories during tablespace drop.
> I am working on it.
>
> Please find attached a solution that builds on what Paul has propose.
> A hash table, similar to the invalid page hash table is used to track
> missing directory references. A missing directory may be a tablespace
> or a database, based on whether the tablespace is found missing or the
> source database is found missing. The crash recovery succeeds if the
> hash table is empty at the end.
>
The v6-0003 patch had merge conflicts due to the recent
'xl_dbase_drop_rec' change, so I rebased it.
See v7-0003 in attachment. Changes are pretty straightforward, though It
would be great, if you could check them once more.
Newly introduced test 4 in t/011_crash_recovery.pl fails without the
patch and passes with it.
It seems to me that everything is fine, so I mark it "Ready For Committer"
--
Anastasia Lubennikova
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
@ 2020-01-10 00:22 ` Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Alvaro Herrera @ 2020-01-10 00:22 UTC (permalink / raw)
To: Anastasia Lubennikova <[email protected]>; +Cc: Asim R P <[email protected]>; Paul Guo <[email protected]>; Alexandra Wang <[email protected]>; Thomas Munro <[email protected]>; Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
I looked at this a little while and was bothered by the perl changes; it
seems out of place to have RecursiveCopy be thinking about tablespaces,
which is way out of its league. So I rewrote that to use a callback:
the PostgresNode code passes a callback that's in charge to handle the
case of a symlink. Things look much more in place with that. I didn't
verify that all places that should use this are filled.
In 0002 I found adding a new function unnecessary: we can keep backwards
compat by checking 'ref' of the third argument. With that we don't have
to add a new function. (POD changes pending.)
I haven't reviewed 0003.
v8 of all these patches attached.
"git am" told me your 0001 was in unrecognized format. It applied fine
with "patch". I suggest that if you're going to submit a series with
commit messages and all, please use "git format-patch" with the same
"-v" argument (9 in this case) for all patches.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2020-01-10 13:43 ` Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Alvaro Herrera @ 2020-01-10 13:43 UTC (permalink / raw)
To: Anastasia Lubennikova <[email protected]>; +Cc: Asim R P <[email protected]>; Paul Guo <[email protected]>; Alexandra Wang <[email protected]>; Thomas Munro <[email protected]>; Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
On 2020-Jan-09, Alvaro Herrera wrote:
> I looked at this a little while and was bothered by the perl changes; it
> seems out of place to have RecursiveCopy be thinking about tablespaces,
> which is way out of its league. So I rewrote that to use a callback:
> the PostgresNode code passes a callback that's in charge to handle the
> case of a symlink. Things look much more in place with that. I didn't
> verify that all places that should use this are filled.
>
> In 0002 I found adding a new function unnecessary: we can keep backwards
> compat by checking 'ref' of the third argument. With that we don't have
> to add a new function. (POD changes pending.)
I forgot to add that something in these changes is broken (probably the
symlink handling callback) so the tests fail, but I couldn't stay away
from my daughter's birthday long enough to figure out what or how. I'm
on something else today, so if one of you can research and submit fixed
versions, that'd be great.
Thanks,
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2020-01-13 10:27 ` Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Paul Guo @ 2020-01-13 10:27 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Anastasia Lubennikova <[email protected]>; Asim R P <[email protected]>; Alexandra Wang <[email protected]>; Thomas Munro <[email protected]>; Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
On Fri, Jan 10, 2020 at 9:43 PM Alvaro Herrera <[email protected]>
wrote:
> On 2020-Jan-09, Alvaro Herrera wrote:
>
> > I looked at this a little while and was bothered by the perl changes; it
> > seems out of place to have RecursiveCopy be thinking about tablespaces,
> > which is way out of its league. So I rewrote that to use a callback:
> > the PostgresNode code passes a callback that's in charge to handle the
> > case of a symlink. Things look much more in place with that. I didn't
> > verify that all places that should use this are filled.
> >
> > In 0002 I found adding a new function unnecessary: we can keep backwards
> > compat by checking 'ref' of the third argument. With that we don't have
> > to add a new function. (POD changes pending.)
>
> I forgot to add that something in these changes is broken (probably the
> symlink handling callback) so the tests fail, but I couldn't stay away
> from my daughter's birthday long enough to figure out what or how. I'm
> on something else today, so if one of you can research and submit fixed
> versions, that'd be great.
>
> Thanks,
>
I spent some time on this before getting off work today.
With below fix, the 4th test is now ok but the 5th (last one) hangs due to
panic.
(gdb) bt
#0 0x0000003397e32625 in raise () from /lib64/libc.so.6
#1 0x0000003397e33e05 in abort () from /lib64/libc.so.6
#2 0x0000000000a90506 in errfinish (dummy=0) at elog.c:590
#3 0x0000000000a92b4b in elog_finish (elevel=22, fmt=0xb2d580 "cannot find
directory %s tablespace %d database %d") at elog.c:1465
#4 0x000000000057aa0a in XLogLogMissingDir (spcNode=16384, dbNode=0,
path=0x1885100 "pg_tblspc/16384/PG_13_202001091/16389") at xlogutils.c:104
#5 0x000000000065e92e in dbase_redo (record=0x1841568) at dbcommands.c:2225
#6 0x000000000056ac94 in StartupXLOG () at xlog.c:7200
diff --git a/src/include/commands/dbcommands.h
b/src/include/commands/dbcommands.h
index b71b400e700..f8f6d5ffd03 100644
--- a/src/include/commands/dbcommands.h
+++ b/src/include/commands/dbcommands.h
@@ -19,8 +19,6 @@
#include "lib/stringinfo.h"
#include "nodes/parsenodes.h"
-extern void CheckMissingDirs4DbaseRedo(void);
-
extern Oid createdb(ParseState *pstate, const CreatedbStmt *stmt);
extern void dropdb(const char *dbname, bool missing_ok, bool force);
extern void DropDatabase(ParseState *pstate, DropdbStmt *stmt);
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index e6e7ea505d9..4eef8bb1985 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -615,11 +615,11 @@ sub _srcsymlink
my $srcrealdir = readlink($srcpath);
opendir(my $dh, $srcrealdir);
- while (readdir $dh)
+ while (my $entry = (readdir $dh))
{
- next if (/^\.\.?$/);
- my $spath = "$srcrealdir/$_";
- my $dpath = "$dstrealdir/$_";
+ next if ($entry eq '.' or $entry eq '..');
+ my $spath = "$srcrealdir/$entry";
+ my $dpath = "$dstrealdir/$entry";
RecursiveCopy::copypath($spath, $dpath);
}
closedir $dh;
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
@ 2020-01-15 10:18 ` Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Paul Guo @ 2020-01-15 10:18 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Anastasia Lubennikova <[email protected]>; Asim R P <[email protected]>; Alexandra Wang <[email protected]>; Thomas Munro <[email protected]>; Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
I further fixed the last test failure (due to a small bug in the test, not
in code). Attached are the new patch series. Let's see the CI pipeline
result.
Attachments:
[application/octet-stream] v9-0001-Support-node-initialization-from-backup-with-tabl.patch (4.4K, ../../CAEET0ZGYHZ1h=N_V-fGkfSjpwu2xLST3NUOwzvHjdg2H23UafQ@mail.gmail.com/3-v9-0001-Support-node-initialization-from-backup-with-tabl.patch)
download | inline diff:
From fa8b7c964d170bbf7aeb1a7b0e94de28f651f388 Mon Sep 17 00:00:00 2001
From: Asim R P <[email protected]>
Date: Fri, 20 Sep 2019 17:31:25 +0530
Subject: [PATCH v9 1/3] Support node initialization from backup with
tablespaces
User defined tablespaces appear as symlinks in in the backup. This
commit tweaks recursive copy subroutine to allow for symlinks specific
to tablespaces.
Authored by Kyotaro
---
src/test/perl/PostgresNode.pm | 29 ++++++++++++++++++++++++++++-
src/test/perl/RecursiveCopy.pm | 40 +++++++++++++++++++++++++++++++++-------
2 files changed, 61 insertions(+), 8 deletions(-)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 2e0cf4a2f3e..587a12ef405 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -593,6 +593,32 @@ sub backup_fs_cold
return;
}
+sub _srcsymlink
+{
+ my ($srcpath, $destpath) = @_;
+
+ croak "Cannot operate on symlink \"$srcpath\""
+ if ($srcpath !~ qr{/(pg_tblspc/[0-9]+)$});
+
+ # We have mapped tablespaces. Copy them individually
+ my $tmpdir = TestLib::tempdir;
+ my $dstrealdir = TestLib::perl2host($tmpdir);
+ my $srcrealdir = readlink($srcpath);
+
+ opendir(my $dh, $srcrealdir);
+ while (my $entry = (readdir $dh))
+ {
+ next if ($entry eq '.' or $entry eq '..');
+ my $spath = "$srcrealdir/$entry";
+ my $dpath = "$dstrealdir/$entry";
+ RecursiveCopy::copypath($spath, $dpath);
+ }
+ closedir $dh;
+
+ symlink $dstrealdir, $destpath;
+
+ return 1;
+}
# Common sub of backup_fs_hot and backup_fs_cold
sub _backup_fs
@@ -680,7 +706,8 @@ sub init_from_backup
my $data_path = $self->data_dir;
rmdir($data_path);
- RecursiveCopy::copypath($backup_path, $data_path);
+ RecursiveCopy::copypath($backup_path, $data_path,
+ srcsymlinkfn => \&_srcsymlink);
chmod(0700, $data_path);
# Base configuration for this node
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
index baf5d0ac63b..715edcdedd0 100644
--- a/src/test/perl/RecursiveCopy.pm
+++ b/src/test/perl/RecursiveCopy.pm
@@ -66,6 +66,7 @@ sub copypath
{
my ($base_src_dir, $base_dest_dir, %params) = @_;
my $filterfn;
+ my $srcsymlinkfn;
if (defined $params{filterfn})
{
@@ -80,31 +81,55 @@ sub copypath
$filterfn = sub { return 1; };
}
+ if (defined $params{srcsymlinkfn})
+ {
+ croak "if specified, srcsymlinkfn must be a subroutine reference"
+ unless defined(ref $params{srcsymlinkfn})
+ and (ref $params{srcsymlinkfn} eq 'CODE');
+
+ $srcsymlinkfn = $params{srcsymlinkfn};
+ }
+ else
+ {
+ $srcsymlinkfn = undef;
+ }
+
# Complain if original path is bogus, because _copypath_recurse won't.
croak "\"$base_src_dir\" does not exist" if !-e $base_src_dir;
# Start recursive copy from current directory
- return _copypath_recurse($base_src_dir, $base_dest_dir, "", $filterfn);
+ return _copypath_recurse($base_src_dir, $base_dest_dir, "", $filterfn, $srcsymlinkfn);
}
# Recursive private guts of copypath
sub _copypath_recurse
{
- my ($base_src_dir, $base_dest_dir, $curr_path, $filterfn) = @_;
+ my ($base_src_dir, $base_dest_dir, $curr_path, $filterfn,
+ $srcsymlinkfn) = @_;
my $srcpath = "$base_src_dir/$curr_path";
my $destpath = "$base_dest_dir/$curr_path";
# invoke the filter and skip all further operation if it returns false
return 1 unless &$filterfn($curr_path);
- # Check for symlink -- needed only on source dir
- # (note: this will fall through quietly if file is already gone)
- croak "Cannot operate on symlink \"$srcpath\"" if -l $srcpath;
-
# Abort if destination path already exists. Should we allow directories
# to exist already?
croak "Destination path \"$destpath\" already exists" if -e $destpath;
+ # Check for symlink -- needed only on source dir
+ # If caller provided us with a callback, call it; otherwise we're out.
+ if (-l $srcpath)
+ {
+ if (defined $srcsymlinkfn)
+ {
+ return &$srcsymlinkfn($srcpath, $destpath);
+ }
+ else
+ {
+ croak "Cannot operate on symlink \"$srcpath\"";
+ }
+ }
+
# If this source path is a file, simply copy it to destination with the
# same name and we're done.
if (-f $srcpath)
@@ -137,7 +162,8 @@ sub _copypath_recurse
{
next if ($entry eq '.' or $entry eq '..');
_copypath_recurse($base_src_dir, $base_dest_dir,
- $curr_path eq '' ? $entry : "$curr_path/$entry", $filterfn)
+ $curr_path eq '' ? $entry : "$curr_path/$entry", $filterfn,
+ $srcsymlinkfn)
or die "copypath $srcpath/$entry -> $destpath/$entry failed";
}
--
2.14.3
[application/octet-stream] v9-0003-Fix-replay-of-create-database-records-on-standby.patch (11.9K, ../../CAEET0ZGYHZ1h=N_V-fGkfSjpwu2xLST3NUOwzvHjdg2H23UafQ@mail.gmail.com/4-v9-0003-Fix-replay-of-create-database-records-on-standby.patch)
download | inline diff:
From 57c4d4f31e501e5b25bb36919e666855fd2b0789 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <[email protected]>
Date: Thu, 9 Jan 2020 17:54:40 -0300
Subject: [PATCH v9 3/3] Fix replay of create database records on standby
Crash recovery on standby may encounter missing directories when
replaying create database WAL records. Prior to this patch, the
standby would fail to recover in such a case. However, the
directories could be legitimately missing. Consider a sequence of WAL
records as follows:
CREATE DATABASE
DROP DATABASE
DROP TABLESPACE
If, after replaying the last WAL record and removing the tablespace
directory, the standby crashes and has to replay the create database
record again, the crash recovery must be able to move on.
This patch adds mechanism similar to invalid page hash table, to track
missing directories during crash recovery. If all the missing
directory references are matched with corresponding drop records at
the end of crash recovery, the standby can safely enter archive
recovery.
Bug identified by Paul Guo.
Authored by Paul Guo, Kyotaro Horiguchi and Asim R P.
---
src/backend/access/rmgrdesc/dbasedesc.c | 16 ++--
src/backend/access/transam/xlog.c | 6 ++
src/backend/access/transam/xlogutils.c | 130 ++++++++++++++++++++++++++++++++
src/backend/commands/dbcommands.c | 54 +++++++++++++
src/backend/commands/tablespace.c | 3 +
src/include/access/xlogutils.h | 4 +
6 files changed, 208 insertions(+), 5 deletions(-)
diff --git a/src/backend/access/rmgrdesc/dbasedesc.c b/src/backend/access/rmgrdesc/dbasedesc.c
index 73d2a4ca34b..f7117873d73 100644
--- a/src/backend/access/rmgrdesc/dbasedesc.c
+++ b/src/backend/access/rmgrdesc/dbasedesc.c
@@ -23,14 +23,17 @@ dbase_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ char *dbpath1, *dbpath2;
if (info == XLOG_DBASE_CREATE)
{
xl_dbase_create_rec *xlrec = (xl_dbase_create_rec *) rec;
- appendStringInfo(buf, "copy dir %u/%u to %u/%u",
- xlrec->src_tablespace_id, xlrec->src_db_id,
- xlrec->tablespace_id, xlrec->db_id);
+ dbpath1 = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
+ dbpath2 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
+ appendStringInfo(buf, "copy dir %s to %s", dbpath1, dbpath2);
+ pfree(dbpath2);
+ pfree(dbpath1);
}
else if (info == XLOG_DBASE_DROP)
{
@@ -39,8 +42,11 @@ dbase_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "dir");
for (i = 0; i < xlrec->ntablespaces; i++)
- appendStringInfo(buf, " %u/%u",
- xlrec->tablespace_ids[i], xlrec->db_id);
+ {
+ dbpath1 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_ids[i]);
+ appendStringInfo(buf, "%s", dbpath1);
+ pfree(dbpath1);
+ }
}
}
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 7f4f784c0eb..d97e48f369c 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -7890,6 +7890,12 @@ CheckRecoveryConsistency(void)
*/
XLogCheckInvalidPages();
+ /*
+ * Check if the XLOG sequence contained any unresolved references to
+ * missing directories.
+ */
+ XLogCheckMissingDirs();
+
reachedConsistency = true;
ereport(LOG,
(errmsg("consistent recovery state reached at %X/%X",
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index b55c3833703..6c2dd5aba1b 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -56,6 +56,136 @@ typedef struct xl_invalid_page
static HTAB *invalid_page_tab = NULL;
+/*
+ * If a create database WAL record is being replayed more than once during
+ * crash recovery on a standby, it is possible that either the tablespace
+ * directory or the template database directory is missing. This happens when
+ * the directories are removed by replay of subsequent drop records. Note
+ * that this problem happens only on standby and not on master. On master, a
+ * checkpoint is created at the end of create database operation. On standby,
+ * however, such a strategy (creating restart points during replay) is not
+ * viable because it will slow down WAL replay.
+ *
+ * The alternative is to track references to each missing directory
+ * encountered when performing crash recovery in the following hash table.
+ * Similar to invalid page table above, the expectation is that each missing
+ * directory entry should be matched with a drop database or drop tablespace
+ * WAL record by the end of crash recovery.
+ */
+typedef struct xl_missing_dir_key
+{
+ Oid spcNode;
+ Oid dbNode;
+} xl_missing_dir_key;
+
+typedef struct xl_missing_dir
+{
+ xl_missing_dir_key key;
+ char path[MAXPGPATH];
+} xl_missing_dir;
+
+static HTAB *missing_dir_tab = NULL;
+
+void
+XLogLogMissingDir(Oid spcNode, Oid dbNode, char *path)
+{
+ xl_missing_dir_key key;
+ bool found;
+ xl_missing_dir *entry;
+
+ /*
+ * Database OID may be invalid but tablespace OID must be valid. If
+ * dbNode is InvalidOid, we are logging a missing tablespace directory,
+ * otherwise we are logging a missing database directory.
+ */
+ Assert(OidIsValid(spcNode));
+
+ if (reachedConsistency)
+ elog(PANIC, "cannot find directory %s tablespace %d database %d",
+ path, spcNode, dbNode);
+
+ if (missing_dir_tab == NULL)
+ {
+ /* create hash table when first needed */
+ HASHCTL ctl;
+
+ memset(&ctl, 0, sizeof(ctl));
+ ctl.keysize = sizeof(xl_missing_dir_key);
+ ctl.entrysize = sizeof(xl_missing_dir);
+
+ missing_dir_tab = hash_create("XLOG missing directory table",
+ 100,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS);
+ }
+
+ key.spcNode = spcNode;
+ key.dbNode = dbNode;
+
+ entry = hash_search(missing_dir_tab, &key, HASH_ENTER, &found);
+
+ if (found)
+ elog(DEBUG2, "missing directory %s tablespace %d database %d already exists: %s",
+ path, spcNode, dbNode, entry->path);
+ else
+ {
+ strlcpy(entry->path, path, sizeof(entry->path));
+ elog(DEBUG2, "logged missing dir %s tablespace %d database %d",
+ path, spcNode, dbNode);
+ }
+}
+
+void
+XLogForgetMissingDir(Oid spcNode, Oid dbNode, char *path)
+{
+ xl_missing_dir_key key;
+
+ key.spcNode = spcNode;
+ key.dbNode = dbNode;
+
+ /* Database OID may be invalid but tablespace OID must be valid. */
+ Assert(OidIsValid(spcNode));
+
+ if (missing_dir_tab == NULL)
+ return;
+
+ if (hash_search(missing_dir_tab, &key, HASH_REMOVE, NULL) == NULL)
+ elog(DEBUG2, "dir %s tablespace %d database %d is not missing",
+ path, spcNode, dbNode);
+ else
+ elog(DEBUG2, "forgot missing dir %s for tablespace %d database %d",
+ path, spcNode, dbNode);
+}
+
+/*
+ * This is called at the end of crash recovery, before entering archive
+ * recovery on a standby. PANIC if the hash table is not empty.
+ */
+void
+XLogCheckMissingDirs(void)
+{
+ HASH_SEQ_STATUS status;
+ xl_missing_dir *hentry;
+ bool foundone = false;
+
+ if (missing_dir_tab == NULL)
+ return; /* nothing to do */
+
+ hash_seq_init(&status, missing_dir_tab);
+
+ while ((hentry = (xl_missing_dir *) hash_seq_search(&status)) != NULL)
+ {
+ elog(WARNING, "missing directory \"%s\" tablespace %d database %d",
+ hentry->path, hentry->key.spcNode, hentry->key.dbNode);
+ foundone = true;
+ }
+
+ if (foundone)
+ elog(PANIC, "WAL contains references to missing directories");
+
+ hash_destroy(missing_dir_tab);
+ missing_dir_tab = NULL;
+}
/* Report a reference to an invalid page */
static void
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index 367c30adb01..6d6668e4f88 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -46,6 +46,7 @@
#include "commands/defrem.h"
#include "commands/seclabel.h"
#include "commands/tablespace.h"
+#include "common/file_perm.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "pgstat.h"
@@ -2185,7 +2186,9 @@ dbase_redo(XLogReaderState *record)
xl_dbase_create_rec *xlrec = (xl_dbase_create_rec *) XLogRecGetData(record);
char *src_path;
char *dst_path;
+ char *parent_path;
struct stat st;
+ bool skip = false;
src_path = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
dst_path = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
@@ -2203,6 +2206,54 @@ dbase_redo(XLogReaderState *record)
(errmsg("some useless files may be left behind in old database directory \"%s\"",
dst_path)));
}
+ else
+ {
+ /*
+ * It is possible that drop tablespace record appearing later in
+ * the WAL as already been replayed. That means we are replaying
+ * the create database record second time, as part of crash
+ * recovery. In that case, the tablespace directory has already
+ * been removed and the create database operation cannot be
+ * replayed. We should skip the replay but remember the missing
+ * tablespace directory, to be matched with a drop tablespace
+ * record later.
+ */
+ parent_path = pstrdup(dst_path);
+ get_parent_directory(parent_path);
+ if (!(stat(parent_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
+ XLogLogMissingDir(xlrec->tablespace_id, InvalidOid, dst_path);
+ skip = true;
+ ereport(WARNING,
+ (errmsg("skipping create database WAL record"),
+ errdetail("Target tablespace \"%s\" not found. We "
+ "expect to encounter a WAL record that "
+ "removes this directory before reaching "
+ "consistent state.", parent_path)));
+ }
+ pfree(parent_path);
+ }
+
+ /*
+ * Source directory may be missing. E.g. the template database used
+ * for creating this database may have been dropped, due to reasons
+ * noted above. Moving a database from one tablespace may also be a
+ * partner in the crime.
+ */
+ if (!(stat(src_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
+ XLogLogMissingDir(xlrec->src_tablespace_id, xlrec->src_db_id, src_path);
+ skip = true;
+ ereport(WARNING,
+ (errmsg("skipping create database WAL record"),
+ errdetail("Source database \"%s\" not found. We expect "
+ "to encounter a WAL record that removes this "
+ "directory before reaching consistent state.",
+ src_path)));
+ }
+
+ if (skip)
+ return;
/*
* Force dirty buffers out to disk, to ensure source database is
@@ -2260,6 +2311,9 @@ dbase_redo(XLogReaderState *record)
ereport(WARNING,
(errmsg("some useless files may be left behind in old database directory \"%s\"",
dst_path)));
+
+ XLogForgetMissingDir(xlrec->tablespace_ids[i], xlrec->db_id, dst_path);
+
pfree(dst_path);
}
diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c
index 051478057f6..33407dceeb2 100644
--- a/src/backend/commands/tablespace.c
+++ b/src/backend/commands/tablespace.c
@@ -58,6 +58,7 @@
#include "access/xact.h"
#include "access/xlog.h"
#include "access/xloginsert.h"
+#include "access/xlogutils.h"
#include "catalog/catalog.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
@@ -1516,6 +1517,8 @@ tblspc_redo(XLogReaderState *record)
{
xl_tblspc_drop_rec *xlrec = (xl_tblspc_drop_rec *) XLogRecGetData(record);
+ XLogForgetMissingDir(xlrec->ts_id, InvalidOid, "");
+
/*
* If we issued a WAL record for a drop tablespace it implies that
* there were no files in it at all when the DROP was done. That means
diff --git a/src/include/access/xlogutils.h b/src/include/access/xlogutils.h
index 5181a077d96..41067350069 100644
--- a/src/include/access/xlogutils.h
+++ b/src/include/access/xlogutils.h
@@ -23,6 +23,10 @@ extern void XLogDropDatabase(Oid dbid);
extern void XLogTruncateRelation(RelFileNode rnode, ForkNumber forkNum,
BlockNumber nblocks);
+extern void XLogLogMissingDir(Oid spcNode, Oid dbNode, char *path);
+extern void XLogForgetMissingDir(Oid spcNode, Oid dbNode, char *path);
+extern void XLogCheckMissingDirs(void);
+
/* Result codes for XLogReadBufferForRedo[Extended] */
typedef enum
{
--
2.14.3
[application/octet-stream] v9-0002-Tests-to-replay-create-database-operation-on-stan.patch (10.0K, ../../CAEET0ZGYHZ1h=N_V-fGkfSjpwu2xLST3NUOwzvHjdg2H23UafQ@mail.gmail.com/5-v9-0002-Tests-to-replay-create-database-operation-on-stan.patch)
download | inline diff:
From f1c9874eb75a9a2979e56402b4a5a363a6f061bc Mon Sep 17 00:00:00 2001
From: Asim R P <[email protected]>
Date: Fri, 20 Sep 2019 17:34:19 +0530
Subject: [PATCH v9 2/3] Tests to replay create database operation on standby
The tests demonstrate that standby fails to replay a create database
WAL record during crash recovery, if one or more of underlying
directories are missing from the file system. This can happen if a
drop tablespace or drop database WAL record has been replayed in
archive recovery, before a crash. And then the create database record
happens to be replayed again during crash recovery. The failures
indicate bugs that need to be fixed.
The first test, TEST 4, performs several DDL operations resulting in a
database directory being removed, along with a few create database
operations. It expects crash recovery to succeed because for each
missing directory encountered during create database replay, a matching
drop tablespace or drop database WAL record is found later.
Second test, TEST 5, validates that a standby rightfully aborts replay
during archive recovery, if a missing directory is encountered when
replaying create database WAL record.
These tests have been proposed and implemented in various ways by
Alexandra Wang, Anastasia Lubennikova, Kyotaro Horiguchi, Paul Guo and me.
---
src/test/perl/PostgresNode.pm | 34 +++++--
src/test/recovery/t/011_crash_recovery.pl | 152 +++++++++++++++++++++++++++++-
2 files changed, 178 insertions(+), 8 deletions(-)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 587a12ef405..4eef8bb1985 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -546,13 +546,22 @@ target server since it isn't done by default.
sub backup
{
- my ($self, $backup_name) = @_;
+ my ($self, $backup_name, %params) = @_;
my $backup_path = $self->backup_dir . '/' . $backup_name;
my $name = $self->name;
+ my @rest = ();
+
+ if (defined $params{tablespace_mappings})
+ {
+ my @ts_mappings = split(/,/, $params{tablespace_mappings});
+ foreach my $elem (@ts_mappings) {
+ push(@rest, '--tablespace-mapping='.$elem);
+ }
+ }
print "# Taking pg_basebackup $backup_name from node \"$name\"\n";
TestLib::system_or_bail('pg_basebackup', '-D', $backup_path, '-h',
- $self->host, '-p', $self->port, '--no-sync');
+ $self->host, '-p', $self->port, '--no-sync', @rest);
print "# Backup finished\n";
return;
}
@@ -1640,13 +1649,24 @@ Returns 1 if successful, 0 if timed out.
sub poll_query_until
{
- my ($self, $dbname, $query, $expected) = @_;
+ my ($self, $dbname, $query, $params) = @_;
+ my $expected;
+
+ # Be backwards-compatible
+ if (defined $params and ref $params eq '')
+ {
+ $params = {
+ expected => $params,
+ timeout => 180
+ };
+ }
- $expected = 't' unless defined($expected); # default value
+ $params->{expected} = 't' unless defined($params->{expected});
+ $params->{timeout} = 180 unless defined($params->{timeout});
my $cmd = [ 'psql', '-XAt', '-c', $query, '-d', $self->connstr($dbname) ];
my ($stdout, $stderr);
- my $max_attempts = 180 * 10;
+ my $max_attempts = $params->{timeout} * 10;
my $attempts = 0;
while ($attempts < $max_attempts)
@@ -1656,7 +1676,7 @@ sub poll_query_until
chomp($stdout);
$stdout =~ s/\r//g if $TestLib::windows_os;
- if ($stdout eq $expected)
+ if ($stdout eq $params->{expected})
{
return 1;
}
@@ -1674,7 +1694,7 @@ sub poll_query_until
diag qq(poll_query_until timed out executing this query:
$query
expecting this output:
-$expected
+$params->{expected}
last actual query output:
$stdout
with stderr:
diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl
index 526a3481fb5..3690865e07b 100644
--- a/src/test/recovery/t/011_crash_recovery.pl
+++ b/src/test/recovery/t/011_crash_recovery.pl
@@ -6,6 +6,7 @@ use warnings;
use PostgresNode;
use TestLib;
use Test::More;
+use File::Path qw(rmtree);
use Config;
if ($Config{osname} eq 'MSWin32')
{
@@ -15,7 +16,7 @@ if ($Config{osname} eq 'MSWin32')
}
else
{
- plan tests => 3;
+ plan tests => 5;
}
my $node = get_new_node('master');
@@ -66,3 +67,152 @@ is($node->safe_psql('postgres', qq[SELECT txid_status('$xid');]),
'aborted', 'xid is aborted after crash');
$tx->kill_kill;
+
+# TEST 4
+#
+# Ensure that a missing tablespace directory during crash recovery on
+# a standby is hangled correctly. The standby should finish crash
+# recovery successfully because a matching drop database record is
+# found in the WAL. The following scnearios are covered:
+#
+# 1. Create a database against a user-defined tablespace then drop the
+# tablespace.
+#
+# 2. Move a database from source tablespace to target tablespace then
+# drop the source tablespace.
+#
+# 3. Create a datbase from another database as template then drop the
+# template database.
+
+my $node_master = get_new_node('master2');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Create tablespace
+my $dropme_ts_master = TestLib::tempdir;
+$dropme_ts_master = TestLib::perl2host($dropme_ts_master);
+my $source_ts_master = TestLib::tempdir;
+$source_ts_master = TestLib::perl2host($source_ts_master);
+my $target_ts_master = TestLib::tempdir;
+$target_ts_master = TestLib::perl2host($target_ts_master);
+
+$node_master->safe_psql('postgres',
+ qq[CREATE TABLESPACE dropme_ts location '$dropme_ts_master';
+ CREATE TABLESPACE source_ts location '$source_ts_master';
+ CREATE TABLESPACE target_ts location '$target_ts_master';
+ CREATE DATABASE template_db IS_TEMPLATE = true;]);
+
+my $dropme_ts_standby = TestLib::tempdir;
+$dropme_ts_standby = TestLib::perl2host($dropme_ts_standby);
+my $source_ts_standby = TestLib::tempdir;
+$source_ts_standby = TestLib::perl2host($source_ts_standby);
+my $target_ts_standby = TestLib::tempdir;
+$target_ts_standby = TestLib::perl2host($target_ts_standby);
+
+# Take backup
+my $backup_name = 'my_backup';
+my $ts_mapping = "$dropme_ts_master=$dropme_ts_standby," .
+ "$source_ts_master=$source_ts_standby," .
+ "$target_ts_master=$target_ts_standby";
+$node_master->backup($backup_name, tablespace_mappings => $ts_mapping);
+
+my $node_standby = get_new_node('standby2');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->start;
+
+# Make sure connection is made
+$node_master->poll_query_until(
+ 'postgres', 'SELECT count(*) = 1 FROM pg_stat_replication');
+
+# Make sure to perform restartpoint after tablespace creation
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->safe_psql('postgres', 'CHECKPOINT');
+
+# Do immediate shutdown just after a sequence of CREAT DATABASE / DROP
+# DATABASE / DROP TABLESPACE. This causes CREATE DATBASE WAL records
+# to be applied to already-removed directories.
+$node_master->safe_psql('postgres',
+ q[CREATE DATABASE dropme_db1 WITH TABLESPACE dropme_ts;
+ CREATE DATABASE dropme_db2 WITH TABLESPACE dropme_ts;
+ CREATE DATABASE moveme_db TABLESPACE source_ts;
+ ALTER DATABASE moveme_db SET TABLESPACE target_ts;
+ DROP DATABASE dropme_db1;
+ CREATE DATABASE newdb TEMPLATE template_db;
+ ALTER DATABASE template_db IS_TEMPLATE = false;
+ DROP TABLESPACE source_ts;
+ DROP DATABASE dropme_db2;
+ DROP TABLESPACE dropme_ts;
+ DROP DATABASE template_db;]);
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->stop('immediate');
+
+# Should restart ignoring directory creation error.
+is($node_standby->start(fail_ok => 1), 1);
+
+# TEST 5
+#
+# Ensure that a missing tablespace directory during create database
+# replay immediately causes panic if the standby has already reached
+# consistent state (archive recovery is in progress).
+
+$node_master = get_new_node('master3');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Create tablespace
+my $ts_master = TestLib::tempdir;
+$ts_master = TestLib::perl2host($ts_master);
+$node_master->safe_psql('postgres', "CREATE TABLESPACE ts1 LOCATION '$ts_master'");
+$node_master->safe_psql('postgres', "CREATE DATABASE db1 TABLESPACE ts1");
+
+my $ts_standby = TestLib::tempdir("standby");
+$ts_standby = TestLib::perl2host($ts_standby);
+
+# Take backup
+$backup_name = 'my_backup';
+$node_master->backup($backup_name,
+ tablespace_mappings =>
+ "$ts_master=$ts_standby");
+$node_standby = get_new_node('standby3');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->start;
+
+# Make sure standby reached consistency and starts accepting connections
+$node_standby->poll_query_until('postgres', 'SELECT 1', '1');
+
+# Remove standby tablespace directory so it will be missing when
+# replay resumes.
+#
+# The tablespace mapping is lost when the standby node is initialized
+# from basebackup because RecursiveCopy::copypath creates a new temp
+# directory for each tablspace symlink found in backup. We must
+# obtain the correct tablespace directory by querying standby.
+$ts_standby = $node_standby->safe_psql(
+ 'postgres',
+ "select pg_tablespace_location(oid) from pg_tablespace where spcname = 'ts1'");
+rmtree($ts_standby);
+
+# Create a database in the tablespace and a table in default tablespace
+$node_master->safe_psql('postgres',
+ q[CREATE TABLE should_not_replay_insertion(a int);
+ CREATE DATABASE db2 WITH TABLESPACE ts1;
+ INSERT INTO should_not_replay_insertion VALUES (1);]);
+
+# Standby should fail and should not silently skip replaying the wal
+if ($node_master->poll_query_until(
+ 'postgres',
+ 'SELECT count(*) = 0 FROM pg_stat_replication',
+ 't') == 1)
+{
+ pass('standby failed as expected');
+ # We know that the standby has failed. Setting its pid to
+ # undefined avoids error when PostgreNode module tries to stop the
+ # standby node as part of tear_down sequence.
+ $node_standby->{_pid} = undef;
+}
+else
+{
+ fail('standby did not fail within 5 seconds');
+}
--
2.14.3
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
@ 2020-01-27 15:24 ` Fujii Masao <[email protected]>
2020-03-25 05:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-07-08 12:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
0 siblings, 2 replies; 109+ messages in thread
From: Fujii Masao @ 2020-01-27 15:24 UTC (permalink / raw)
To: Paul Guo <[email protected]>; Alvaro Herrera <[email protected]>; +Cc: Anastasia Lubennikova <[email protected]>; Asim R P <[email protected]>; Alexandra Wang <[email protected]>; Thomas Munro <[email protected]>; Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
On 2020/01/15 19:18, Paul Guo wrote:
> I further fixed the last test failure (due to a small bug in the test, not in code). Attached are the new patch series. Let's see the CI pipeline result.
Thanks for updating the patches!
I started reading the 0003 patch.
The approach that the 0003 patch uses is not the perfect solution.
If the standby crashes after tblspc_redo() removes the directory and before
its subsequent COMMIT record is replayed, PANIC error would occur since
there can be some unresolved missing directory entries when we reach the
consistent state. The problem would very rarely happen, though...
Just idea; calling XLogFlush() to update the minimum recovery point just
before tblspc_redo() performs destroy_tablespace_directories() may be
safe and helpful for the problem?
- appendStringInfo(buf, "copy dir %u/%u to %u/%u",
- xlrec->src_tablespace_id, xlrec->src_db_id,
- xlrec->tablespace_id, xlrec->db_id);
+ dbpath1 = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
+ dbpath2 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
+ appendStringInfo(buf, "copy dir %s to %s", dbpath1, dbpath2);
+ pfree(dbpath2);
+ pfree(dbpath1);
If the patch is for the bug fix and would be back-ported, the above change
would lead to change pg_waldump's output for CREATE/DROP DATABASE between
minor versions. IMO it's better to avoid such change and separate the above
as a separate patch only for master.
- appendStringInfo(buf, " %u/%u",
- xlrec->tablespace_ids[i], xlrec->db_id);
+ {
+ dbpath1 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_ids[i]);
+ appendStringInfo(buf, "%s", dbpath1);
+ pfree(dbpath1);
+ }
Same as above.
BTW, the above "%s" should be " %s", i.e., a space character needs to be
appended to the head of "%s".
+ get_parent_directory(parent_path);
+ if (!(stat(parent_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
+ XLogLogMissingDir(xlrec->tablespace_id, InvalidOid, dst_path);
The third argument of XLogLogMissingDir() should be parent_path instead of
dst_path?
+ if (hash_search(missing_dir_tab, &key, HASH_REMOVE, NULL) == NULL)
+ elog(DEBUG2, "dir %s tablespace %d database %d is not missing",
+ path, spcNode, dbNode);
I think that this elog() is useless and rather confusing.
+ XLogForgetMissingDir(xlrec->ts_id, InvalidOid, "");
The third argument should be set to the actual path instead of an empty
string. Otherwise XLogForgetMissingDir() may emit a confusing DEBUG2
message. Or the third argument of XLogForgetMissingDir() should be removed
and the path in the DEBUG2 message should be calculated from the spcNode
and dbNode in the hash entry in XLogForgetMissingDir().
+#include "common/file_perm.h"
This seems not necessary.
Regards,
--
Fujii Masao
NTT DATA CORPORATION
Advanced Platform Technology Group
Research and Development Headquarters
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
@ 2020-03-25 05:52 ` Fujii Masao <[email protected]>
2020-07-07 21:12 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Daniel Gustafsson <[email protected]>
1 sibling, 1 reply; 109+ messages in thread
From: Fujii Masao @ 2020-03-25 05:52 UTC (permalink / raw)
To: Paul Guo <[email protected]>; Alvaro Herrera <[email protected]>; +Cc: Anastasia Lubennikova <[email protected]>; Asim R P <[email protected]>; Alexandra Wang <[email protected]>; Thomas Munro <[email protected]>; Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
On 2020/01/28 0:24, Fujii Masao wrote:
>
>
> On 2020/01/15 19:18, Paul Guo wrote:
>> I further fixed the last test failure (due to a small bug in the test, not in code). Attached are the new patch series. Let's see the CI pipeline result.
>
> Thanks for updating the patches!
>
> I started reading the 0003 patch.
I marked this patch as Waiting on Author in CF because there is no update
since my last review comments. Could you mark it as Needs Review again
if you post the updated version of the patch.
Regards,
--
Fujii Masao
NTT DATA CORPORATION
Advanced Platform Technology Group
Research and Development Headquarters
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-03-25 05:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
@ 2020-07-07 21:12 ` Daniel Gustafsson <[email protected]>
0 siblings, 0 replies; 109+ messages in thread
From: Daniel Gustafsson @ 2020-07-07 21:12 UTC (permalink / raw)
To: Paul Guo <[email protected]>; +Cc: Fujii Masao <[email protected]>; Alvaro Herrera <[email protected]>; Anastasia Lubennikova <[email protected]>; Asim R P <[email protected]>; Alexandra Wang <[email protected]>; Thomas Munro <[email protected]>; Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
> On 25 Mar 2020, at 06:52, Fujii Masao <[email protected]> wrote:
>
> On 2020/01/28 0:24, Fujii Masao wrote:
>> On 2020/01/15 19:18, Paul Guo wrote:
>>> I further fixed the last test failure (due to a small bug in the test, not in code). Attached are the new patch series. Let's see the CI pipeline result.
>> Thanks for updating the patches!
>> I started reading the 0003 patch.
>
> I marked this patch as Waiting on Author in CF because there is no update
> since my last review comments. Could you mark it as Needs Review again
> if you post the updated version of the patch.
This thread has been stalled since effectively January, so I'm marking this
patch Returned with Feedback. Feel free to open a new entry once the review
comments have been addressed.
cheers ./daniel
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
@ 2020-07-08 12:56 ` Paul Guo <[email protected]>
2021-01-05 01:07 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
1 sibling, 1 reply; 109+ messages in thread
From: Paul Guo @ 2020-07-08 12:56 UTC (permalink / raw)
To: Fujii Masao <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Anastasia Lubennikova <[email protected]>; Asim Praveen (Pivotal) <[email protected]>; Lei Wang (Pivotal) <[email protected]>; Thomas Munro <[email protected]>; Kyotaro HORIGUCHI <[email protected]>; pgsql-hackers
Looks like my previous reply was held for moderation (maybe due to my new email address).
I configured my pg account today using the new email address. I guess this email would be
held for moderation.
I’m now replying my previous reply email and attaching the new patch series.
On Jul 6, 2020, at 10:18 AM, Paul Guo <[email protected]<mailto:[email protected]>> wrote:
Thanks for the review. I’m now re-picking up the work. I modified the code following the comments.
Besides, I tweaked the test code a bit. There are several things I’m not 100% sure. Please see
my replies below.
On Jan 27, 2020, at 11:24 PM, Fujii Masao <[email protected]<mailto:[email protected]>> wrote:
On 2020/01/15 19:18, Paul Guo wrote:
I further fixed the last test failure (due to a small bug in the test, not in code). Attached are the new patch series. Let's see the CI pipeline result.
Thanks for updating the patches!
I started reading the 0003 patch.
The approach that the 0003 patch uses is not the perfect solution.
If the standby crashes after tblspc_redo() removes the directory and before
its subsequent COMMIT record is replayed, PANIC error would occur since
there can be some unresolved missing directory entries when we reach the
consistent state. The problem would very rarely happen, though...
Just idea; calling XLogFlush() to update the minimum recovery point just
before tblspc_redo() performs destroy_tablespace_directories() may be
safe and helpful for the problem?
Yes looks like an issue. My understanding is the below scenario.
XLogLogMissingDir()
XLogFlush() in redo (e.g. in a commit redo). <- create a minimum recovery point (we call it LSN_A).
tblspc_redo()->XLogForgetMissingDir()
<- If we panic immediately after we remove the directory in tblspc_redo()
<- when we do replay during crash-recovery, we will check consistency at LSN_A and thus PANIC inXLogCheckMissingDirs()
commit
We should add a XLogFlush() in tblspc_redo(). This brings several other questions to my minds also.
1. Should we call XLogFlush() in dbase_redo() for XLOG_DBASE_DROP also?
It calls both XLogDropDatabase() and XLogForgetMissingDir, which seem to have this issue also?
2. xact_redo_abort() calls DropRelationFiles() also. Why do not we call XLogFlush() there?
- appendStringInfo(buf, "copy dir %u/%u to %u/%u",
- xlrec->src_tablespace_id, xlrec->src_db_id,
- xlrec->tablespace_id, xlrec->db_id);
+ dbpath1 = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
+ dbpath2 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
+ appendStringInfo(buf, "copy dir %s to %s", dbpath1, dbpath2);
+ pfree(dbpath2);
+ pfree(dbpath1);
If the patch is for the bug fix and would be back-ported, the above change
would lead to change pg_waldump's output for CREATE/DROP DATABASE between
minor versions. IMO it's better to avoid such change and separate the above
as a separate patch only for master.
I know we do not want wal format between minor releases, but does wal description string change
between minor releases affect users? Anyone I’ll extract this part into a separate patch in the series
since this change is actually independent of the other changes..
- appendStringInfo(buf, " %u/%u",
- xlrec->tablespace_ids[i], xlrec->db_id);
+ {
+ dbpath1 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_ids[i]);
+ appendStringInfo(buf, "%s", dbpath1);
+ pfree(dbpath1);
+ }
Same as above.
BTW, the above "%s" should be " %s", i.e., a space character needs to be
appended to the head of "%s”.
OK
+ get_parent_directory(parent_path);
+ if (!(stat(parent_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
+ XLogLogMissingDir(xlrec->tablespace_id, InvalidOid, dst_path);
The third argument of XLogLogMissingDir() should be parent_path instead of
dst_path?
The argument is for debug message printing so both should be fine, but admittedly we are
logging for the tablespace directory so parent_path might be better.
+ if (hash_search(missing_dir_tab, &key, HASH_REMOVE, NULL) == NULL)
+ elog(DEBUG2, "dir %s tablespace %d database %d is not missing",
+ path, spcNode, dbNode);
I think that this elog() is useless and rather confusing.
OK. Modified.
+ XLogForgetMissingDir(xlrec->ts_id, InvalidOid, "");
The third argument should be set to the actual path instead of an empty
string. Otherwise XLogForgetMissingDir() may emit a confusing DEBUG2
message. Or the third argument of XLogForgetMissingDir() should be removed
and the path in the DEBUG2 message should be calculated from the spcNode
and dbNode in the hash entry in XLogForgetMissingDir().
I’m now removing the third argument. Use GetDatabasePath() to get the path if database did I snot InvalidOid.
+#include "common/file_perm.h"
This seems not necessary.
Right.
Attachments:
[application/octet-stream] v10-0001-Support-node-initialization-from-backup-with-tab.patch (4.4K, ../../[email protected]/3-v10-0001-Support-node-initialization-from-backup-with-tab.patch)
download | inline diff:
From b29747de93fb70d3a7a6f843e232d3dec747451e Mon Sep 17 00:00:00 2001
From: Asim R P <[email protected]>
Date: Fri, 20 Sep 2019 17:31:25 +0530
Subject: [PATCH v10 1/4] Support node initialization from backup with
tablespaces
User defined tablespaces appear as symlinks in in the backup. This
commit tweaks recursive copy subroutine to allow for symlinks specific
to tablespaces.
Authored by Kyotaro Horiguchi
---
src/test/perl/PostgresNode.pm | 29 ++++++++++++++++++++++++++++-
src/test/perl/RecursiveCopy.pm | 40 +++++++++++++++++++++++++++++++++-------
2 files changed, 61 insertions(+), 8 deletions(-)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 1407359aef6..e1ddb36ff1d 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -593,6 +593,32 @@ sub backup_fs_cold
return;
}
+sub _srcsymlink
+{
+ my ($srcpath, $destpath) = @_;
+
+ croak "Cannot operate on symlink \"$srcpath\""
+ if ($srcpath !~ qr{/(pg_tblspc/[0-9]+)$});
+
+ # We have mapped tablespaces. Copy them individually
+ my $tmpdir = TestLib::tempdir;
+ my $dstrealdir = TestLib::perl2host($tmpdir);
+ my $srcrealdir = readlink($srcpath);
+
+ opendir(my $dh, $srcrealdir);
+ while (my $entry = (readdir $dh))
+ {
+ next if ($entry eq '.' or $entry eq '..');
+ my $spath = "$srcrealdir/$entry";
+ my $dpath = "$dstrealdir/$entry";
+ RecursiveCopy::copypath($spath, $dpath);
+ }
+ closedir $dh;
+
+ symlink $dstrealdir, $destpath;
+
+ return 1;
+}
# Common sub of backup_fs_hot and backup_fs_cold
sub _backup_fs
@@ -684,7 +710,8 @@ sub init_from_backup
my $data_path = $self->data_dir;
rmdir($data_path);
- RecursiveCopy::copypath($backup_path, $data_path);
+ RecursiveCopy::copypath($backup_path, $data_path,
+ srcsymlinkfn => \&_srcsymlink);
chmod(0700, $data_path);
# Base configuration for this node
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
index baf5d0ac63b..715edcdedd0 100644
--- a/src/test/perl/RecursiveCopy.pm
+++ b/src/test/perl/RecursiveCopy.pm
@@ -66,6 +66,7 @@ sub copypath
{
my ($base_src_dir, $base_dest_dir, %params) = @_;
my $filterfn;
+ my $srcsymlinkfn;
if (defined $params{filterfn})
{
@@ -80,31 +81,55 @@ sub copypath
$filterfn = sub { return 1; };
}
+ if (defined $params{srcsymlinkfn})
+ {
+ croak "if specified, srcsymlinkfn must be a subroutine reference"
+ unless defined(ref $params{srcsymlinkfn})
+ and (ref $params{srcsymlinkfn} eq 'CODE');
+
+ $srcsymlinkfn = $params{srcsymlinkfn};
+ }
+ else
+ {
+ $srcsymlinkfn = undef;
+ }
+
# Complain if original path is bogus, because _copypath_recurse won't.
croak "\"$base_src_dir\" does not exist" if !-e $base_src_dir;
# Start recursive copy from current directory
- return _copypath_recurse($base_src_dir, $base_dest_dir, "", $filterfn);
+ return _copypath_recurse($base_src_dir, $base_dest_dir, "", $filterfn, $srcsymlinkfn);
}
# Recursive private guts of copypath
sub _copypath_recurse
{
- my ($base_src_dir, $base_dest_dir, $curr_path, $filterfn) = @_;
+ my ($base_src_dir, $base_dest_dir, $curr_path, $filterfn,
+ $srcsymlinkfn) = @_;
my $srcpath = "$base_src_dir/$curr_path";
my $destpath = "$base_dest_dir/$curr_path";
# invoke the filter and skip all further operation if it returns false
return 1 unless &$filterfn($curr_path);
- # Check for symlink -- needed only on source dir
- # (note: this will fall through quietly if file is already gone)
- croak "Cannot operate on symlink \"$srcpath\"" if -l $srcpath;
-
# Abort if destination path already exists. Should we allow directories
# to exist already?
croak "Destination path \"$destpath\" already exists" if -e $destpath;
+ # Check for symlink -- needed only on source dir
+ # If caller provided us with a callback, call it; otherwise we're out.
+ if (-l $srcpath)
+ {
+ if (defined $srcsymlinkfn)
+ {
+ return &$srcsymlinkfn($srcpath, $destpath);
+ }
+ else
+ {
+ croak "Cannot operate on symlink \"$srcpath\"";
+ }
+ }
+
# If this source path is a file, simply copy it to destination with the
# same name and we're done.
if (-f $srcpath)
@@ -137,7 +162,8 @@ sub _copypath_recurse
{
next if ($entry eq '.' or $entry eq '..');
_copypath_recurse($base_src_dir, $base_dest_dir,
- $curr_path eq '' ? $entry : "$curr_path/$entry", $filterfn)
+ $curr_path eq '' ? $entry : "$curr_path/$entry", $filterfn,
+ $srcsymlinkfn)
or die "copypath $srcpath/$entry -> $destpath/$entry failed";
}
--
2.14.3
[application/octet-stream] v10-0002-Tests-to-replay-create-database-operation-on-sta.patch (10.4K, ../../[email protected]/4-v10-0002-Tests-to-replay-create-database-operation-on-sta.patch)
download | inline diff:
From 00a3a6e028afeca2495e55634cba53f0c813b241 Mon Sep 17 00:00:00 2001
From: Asim R P <[email protected]>
Date: Fri, 20 Sep 2019 17:34:19 +0530
Subject: [PATCH v10 2/4] Tests to replay create database operation on standby
The tests demonstrate that standby fails to replay a create database
WAL record during crash recovery, if one or more of underlying
directories are missing from the file system. This can happen if a
drop tablespace or drop database WAL record has been replayed in
archive recovery, before a crash. And then the create database record
happens to be replayed again during crash recovery. The failures
indicate bugs that need to be fixed.
The first test, TEST 4, performs several DDL operations resulting in a
database directory being removed, along with a few create database
operations. It expects crash recovery to succeed because for each
missing directory encountered during create database replay, a matching
drop tablespace or drop database WAL record is found later.
Second test, TEST 5, validates that a standby rightfully aborts replay
during archive recovery, if a missing directory is encountered when
replaying create database WAL record.
These tests have been proposed and implemented in various ways by
Alexandra Wang, Anastasia Lubennikova, Kyotaro Horiguchi, Paul Guo and me.
---
src/test/perl/PostgresNode.pm | 34 +++++--
src/test/recovery/t/011_crash_recovery.pl | 162 +++++++++++++++++++++++++++++-
2 files changed, 188 insertions(+), 8 deletions(-)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index e1ddb36ff1d..81538bbbebe 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -546,13 +546,22 @@ target server since it isn't done by default.
sub backup
{
- my ($self, $backup_name) = @_;
+ my ($self, $backup_name, %params) = @_;
my $backup_path = $self->backup_dir . '/' . $backup_name;
my $name = $self->name;
+ my @rest = ();
+
+ if (defined $params{tablespace_mappings})
+ {
+ my @ts_mappings = split(/,/, $params{tablespace_mappings});
+ foreach my $elem (@ts_mappings) {
+ push(@rest, '--tablespace-mapping='.$elem);
+ }
+ }
print "# Taking pg_basebackup $backup_name from node \"$name\"\n";
TestLib::system_or_bail('pg_basebackup', '-D', $backup_path, '-h',
- $self->host, '-p', $self->port, '--no-sync');
+ $self->host, '-p', $self->port, '--no-sync', @rest);
print "# Backup finished\n";
return;
}
@@ -1666,13 +1675,24 @@ Returns 1 if successful, 0 if timed out.
sub poll_query_until
{
- my ($self, $dbname, $query, $expected) = @_;
+ my ($self, $dbname, $query, $params) = @_;
+ my $expected;
+
+ # Be backwards-compatible
+ if (defined $params and ref $params eq '')
+ {
+ $params = {
+ expected => $params,
+ timeout => 180
+ };
+ }
- $expected = 't' unless defined($expected); # default value
+ $params->{expected} = 't' unless defined($params->{expected});
+ $params->{timeout} = 180 unless defined($params->{timeout});
my $cmd = [ 'psql', '-XAt', '-c', $query, '-d', $self->connstr($dbname) ];
my ($stdout, $stderr);
- my $max_attempts = 180 * 10;
+ my $max_attempts = $params->{timeout} * 10;
my $attempts = 0;
while ($attempts < $max_attempts)
@@ -1682,7 +1702,7 @@ sub poll_query_until
chomp($stdout);
$stdout =~ s/\r//g if $TestLib::windows_os;
- if ($stdout eq $expected)
+ if ($stdout eq $params->{expected})
{
return 1;
}
@@ -1700,7 +1720,7 @@ sub poll_query_until
diag qq(poll_query_until timed out executing this query:
$query
expecting this output:
-$expected
+$params->{expected}
last actual query output:
$stdout
with stderr:
diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl
index ca6e92b50df..55f96519fa0 100644
--- a/src/test/recovery/t/011_crash_recovery.pl
+++ b/src/test/recovery/t/011_crash_recovery.pl
@@ -6,6 +6,7 @@ use warnings;
use PostgresNode;
use TestLib;
use Test::More;
+use File::Path qw(rmtree);
use Config;
if ($Config{osname} eq 'MSWin32')
{
@@ -15,7 +16,7 @@ if ($Config{osname} eq 'MSWin32')
}
else
{
- plan tests => 3;
+ plan tests => 5;
}
my $node = get_new_node('master');
@@ -66,3 +67,162 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]),
'aborted', 'xid is aborted after crash');
$tx->kill_kill;
+
+# TEST 4
+#
+# Ensure that a missing tablespace directory during crash recovery on
+# a standby is handled correctly. The standby should finish crash
+# recovery successfully because a matching drop database record is
+# found in the WAL. The following scnearios are covered:
+#
+# 1. Create a database against a user-defined tablespace then drop the
+# database.
+#
+# 2. Create a database against a user-defined tablespace then drop the
+# database and the tablespace.
+#
+# 3. Move a database from source tablespace to target tablespace then
+# drop the source tablespace.
+#
+# 4. Create a database from another database as template then drop the
+# template database.
+#
+#
+
+my $node_master = get_new_node('master2');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Create tablespace
+my $dropme_ts_master1 = TestLib::tempdir;
+$dropme_ts_master1 = TestLib::perl2host($dropme_ts_master1);
+my $dropme_ts_master2 = TestLib::tempdir;
+$dropme_ts_master2 = TestLib::perl2host($dropme_ts_master2);
+my $source_ts_master = TestLib::tempdir;
+$source_ts_master = TestLib::perl2host($source_ts_master);
+my $target_ts_master = TestLib::tempdir;
+$target_ts_master = TestLib::perl2host($target_ts_master);
+
+$node_master->safe_psql('postgres',
+ qq[CREATE TABLESPACE dropme_ts1 location '$dropme_ts_master1';
+ CREATE TABLESPACE dropme_ts2 location '$dropme_ts_master2';
+ CREATE TABLESPACE source_ts location '$source_ts_master';
+ CREATE TABLESPACE target_ts location '$target_ts_master';
+ CREATE DATABASE template_db IS_TEMPLATE = true;]);
+
+my $dropme_ts_standby1 = TestLib::tempdir;
+$dropme_ts_standby1 = TestLib::perl2host($dropme_ts_standby1);
+my $dropme_ts_standby2 = TestLib::tempdir;
+$dropme_ts_standby2 = TestLib::perl2host($dropme_ts_standby2);
+my $source_ts_standby = TestLib::tempdir;
+$source_ts_standby = TestLib::perl2host($source_ts_standby);
+my $target_ts_standby = TestLib::tempdir;
+$target_ts_standby = TestLib::perl2host($target_ts_standby);
+
+# Take backup
+my $backup_name = 'my_backup';
+my $ts_mapping = "$dropme_ts_master1=$dropme_ts_standby1," .
+ "$dropme_ts_master2=$dropme_ts_standby2," .
+ "$source_ts_master=$source_ts_standby," .
+ "$target_ts_master=$target_ts_standby";
+$node_master->backup($backup_name, tablespace_mappings => $ts_mapping);
+
+my $node_standby = get_new_node('standby2');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->start;
+
+# Make sure connection is made
+$node_master->poll_query_until(
+ 'postgres', 'SELECT count(*) = 1 FROM pg_stat_replication');
+
+# Make sure to perform restartpoint after tablespace creation
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->safe_psql('postgres', 'CHECKPOINT');
+
+# Do immediate shutdown just after a sequence of CREAT DATABASE / DROP
+# DATABASE / DROP TABLESPACE. This causes CREATE DATBASE WAL records
+# to be applied to already-removed directories.
+$node_master->safe_psql('postgres',
+ q[CREATE DATABASE dropme_db1 WITH TABLESPACE dropme_ts1;
+ CREATE DATABASE dropme_db2 WITH TABLESPACE dropme_ts2;
+ CREATE DATABASE moveme_db TABLESPACE source_ts;
+ ALTER DATABASE moveme_db SET TABLESPACE target_ts;
+ CREATE DATABASE newdb TEMPLATE template_db;
+ ALTER DATABASE template_db IS_TEMPLATE = false;
+ DROP DATABASE dropme_db1;
+ DROP DATABASE dropme_db2; DROP TABLESPACE dropme_ts2;
+ DROP TABLESPACE source_ts;
+ DROP DATABASE template_db;]);
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->stop('immediate');
+
+# Should restart ignoring directory creation error.
+is($node_standby->start(fail_ok => 1), 1);
+
+# TEST 5
+#
+# Ensure that a missing tablespace directory during create database
+# replay immediately causes panic if the standby has already reached
+# consistent state (archive recovery is in progress).
+
+$node_master = get_new_node('master3');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Create tablespace
+my $ts_master = TestLib::tempdir;
+$ts_master = TestLib::perl2host($ts_master);
+$node_master->safe_psql('postgres', "CREATE TABLESPACE ts1 LOCATION '$ts_master'");
+$node_master->safe_psql('postgres', "CREATE DATABASE db1 TABLESPACE ts1");
+
+my $ts_standby = TestLib::tempdir("standby");
+$ts_standby = TestLib::perl2host($ts_standby);
+
+# Take backup
+$backup_name = 'my_backup';
+$node_master->backup($backup_name,
+ tablespace_mappings =>
+ "$ts_master=$ts_standby");
+$node_standby = get_new_node('standby3');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->start;
+
+# Make sure standby reached consistency and starts accepting connections
+$node_standby->poll_query_until('postgres', 'SELECT 1', '1');
+
+# Remove standby tablespace directory so it will be missing when
+# replay resumes.
+#
+# The tablespace mapping is lost when the standby node is initialized
+# from basebackup because RecursiveCopy::copypath creates a new temp
+# directory for each tablspace symlink found in backup. We must
+# obtain the correct tablespace directory by querying standby.
+$ts_standby = $node_standby->safe_psql(
+ 'postgres',
+ "select pg_tablespace_location(oid) from pg_tablespace where spcname = 'ts1'");
+rmtree($ts_standby);
+
+# Create a database in the tablespace and a table in default tablespace
+$node_master->safe_psql('postgres',
+ q[CREATE TABLE should_not_replay_insertion(a int);
+ CREATE DATABASE db2 WITH TABLESPACE ts1;
+ INSERT INTO should_not_replay_insertion VALUES (1);]);
+
+# Standby should fail and should not silently skip replaying the wal
+if ($node_master->poll_query_until(
+ 'postgres',
+ 'SELECT count(*) = 0 FROM pg_stat_replication',
+ 't') == 1)
+{
+ pass('standby failed as expected');
+ # We know that the standby has failed. Setting its pid to
+ # undefined avoids error when PostgreNode module tries to stop the
+ # standby node as part of tear_down sequence.
+ $node_standby->{_pid} = undef;
+}
+else
+{
+ fail('standby did not fail within 5 seconds');
+}
--
2.14.3
[application/octet-stream] v10-0003-Fix-replay-of-create-database-records-on-standby.patch (10.8K, ../../[email protected]/5-v10-0003-Fix-replay-of-create-database-records-on-standby.patch)
download | inline diff:
From 26b385bdcc42fe25c933a17ed68d636f57271138 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <[email protected]>
Date: Thu, 9 Jan 2020 17:54:40 -0300
Subject: [PATCH v10 3/4] Fix replay of create database records on standby
Crash recovery on standby may encounter missing directories when
replaying create database WAL records. Prior to this patch, the
standby would fail to recover in such a case. However, the
directories could be legitimately missing. Consider a sequence of WAL
records as follows:
CREATE DATABASE
DROP DATABASE
DROP TABLESPACE
If, after replaying the last WAL record and removing the tablespace
directory, the standby crashes and has to replay the create database
record again, the crash recovery must be able to move on.
This patch adds mechanism similar to invalid page hash table, to track
missing directories during crash recovery. If all the missing
directory references are matched with corresponding drop records at
the end of crash recovery, the standby can safely enter archive
recovery.
Bug identified by Paul Guo.
Authored by Paul Guo, Kyotaro Horiguchi and Asim R P.
---
src/backend/access/transam/xlog.c | 6 ++
src/backend/access/transam/xlogutils.c | 155 +++++++++++++++++++++++++++++++++
src/backend/commands/dbcommands.c | 53 +++++++++++
src/backend/commands/tablespace.c | 5 ++
src/include/access/xlogutils.h | 4 +
5 files changed, 223 insertions(+)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index fd93bcfaeba..af4dd19a71d 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8032,6 +8032,12 @@ CheckRecoveryConsistency(void)
*/
XLogCheckInvalidPages();
+ /*
+ * Check if the XLOG sequence contained any unresolved references to
+ * missing directories.
+ */
+ XLogCheckMissingDirs();
+
reachedConsistency = true;
ereport(LOG,
(errmsg("consistent recovery state reached at %X/%X",
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index 322b0e8ff5b..bd98e42c895 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -59,6 +59,161 @@ typedef struct xl_invalid_page
static HTAB *invalid_page_tab = NULL;
+/*
+ * If a create database WAL record is being replayed more than once during
+ * crash recovery on a standby, it is possible that either the tablespace
+ * directory or the template database directory is missing. This happens when
+ * the directories are removed by replay of subsequent drop records. Note
+ * that this problem happens only on standby and not on master. On master, a
+ * checkpoint is created at the end of create database operation. On standby,
+ * however, such a strategy (creating restart points during replay) is not
+ * viable because it will slow down WAL replay.
+ *
+ * The alternative is to track references to each missing directory
+ * encountered when performing crash recovery in the following hash table.
+ * Similar to invalid page table above, the expectation is that each missing
+ * directory entry should be matched with a drop database or drop tablespace
+ * WAL record by the end of crash recovery.
+ */
+typedef struct xl_missing_dir_key
+{
+ Oid spcNode;
+ Oid dbNode;
+} xl_missing_dir_key;
+
+typedef struct xl_missing_dir
+{
+ xl_missing_dir_key key;
+ char path[MAXPGPATH];
+} xl_missing_dir;
+
+static HTAB *missing_dir_tab = NULL;
+
+void
+XLogLogMissingDir(Oid spcNode, Oid dbNode, char *path)
+{
+ xl_missing_dir_key key;
+ bool found;
+ xl_missing_dir *entry;
+
+ /*
+ * Database OID may be invalid but tablespace OID must be valid. If
+ * dbNode is InvalidOid, we are logging a missing tablespace directory,
+ * otherwise we are logging a missing database directory.
+ */
+ Assert(OidIsValid(spcNode));
+
+ if (reachedConsistency)
+ {
+ if (dbNode == InvalidOid)
+ elog(PANIC, "cannot find directory %s (tablespace %d)",
+ path, spcNode);
+ else
+ elog(PANIC, "cannot find directory %s (tablespace %d database %d)",
+ path, spcNode, dbNode);
+ }
+
+ if (missing_dir_tab == NULL)
+ {
+ /* create hash table when first needed */
+ HASHCTL ctl;
+
+ memset(&ctl, 0, sizeof(ctl));
+ ctl.keysize = sizeof(xl_missing_dir_key);
+ ctl.entrysize = sizeof(xl_missing_dir);
+
+ missing_dir_tab = hash_create("XLOG missing directory table",
+ 100,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS);
+ }
+
+ key.spcNode = spcNode;
+ key.dbNode = dbNode;
+
+ entry = hash_search(missing_dir_tab, &key, HASH_ENTER, &found);
+
+ if (found)
+ {
+ if (dbNode == InvalidOid)
+ elog(DEBUG2, "missing directory %s (tablespace %d) already exists: %s",
+ path, spcNode, entry->path);
+ else
+ elog(DEBUG2, "missing directory %s (tablespace %d database %d) already exists: %s",
+ path, spcNode, dbNode, entry->path);
+ }
+ else
+ {
+ strlcpy(entry->path, path, sizeof(entry->path));
+ if (dbNode == InvalidOid)
+ elog(DEBUG2, "logged missing dir %s (tablespace %d)",
+ path, spcNode);
+ else
+ elog(DEBUG2, "logged missing dir %s (tablespace %d database %d)",
+ path, spcNode, dbNode);
+ }
+}
+
+void
+XLogForgetMissingDir(Oid spcNode, Oid dbNode)
+{
+ xl_missing_dir_key key;
+
+ key.spcNode = spcNode;
+ key.dbNode = dbNode;
+
+ /* Database OID may be invalid but tablespace OID must be valid. */
+ Assert(OidIsValid(spcNode));
+
+ if (missing_dir_tab == NULL)
+ return;
+
+ if (hash_search(missing_dir_tab, &key, HASH_REMOVE, NULL) != NULL)
+ {
+ if (dbNode == InvalidOid)
+ {
+ elog(DEBUG2, "forgot missing dir (tablespace %d)", spcNode);
+ }
+ else
+ {
+ char *path = GetDatabasePath(dbNode, spcNode);
+
+ elog(DEBUG2, "forgot missing dir %s (tablespace %d database %d)",
+ path, spcNode, dbNode);
+ pfree(path);
+ }
+ }
+}
+
+/*
+ * This is called at the end of crash recovery, before entering archive
+ * recovery on a standby. PANIC if the hash table is not empty.
+ */
+void
+XLogCheckMissingDirs(void)
+{
+ HASH_SEQ_STATUS status;
+ xl_missing_dir *hentry;
+ bool foundone = false;
+
+ if (missing_dir_tab == NULL)
+ return; /* nothing to do */
+
+ hash_seq_init(&status, missing_dir_tab);
+
+ while ((hentry = (xl_missing_dir *) hash_seq_search(&status)) != NULL)
+ {
+ elog(WARNING, "missing directory \"%s\" tablespace %d database %d",
+ hentry->path, hentry->key.spcNode, hentry->key.dbNode);
+ foundone = true;
+ }
+
+ if (foundone)
+ elog(PANIC, "WAL contains references to missing directories");
+
+ hash_destroy(missing_dir_tab);
+ missing_dir_tab = NULL;
+}
/* Report a reference to an invalid page */
static void
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index f27c3fe8c1c..4a3adc7c6fc 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -2185,7 +2185,9 @@ dbase_redo(XLogReaderState *record)
xl_dbase_create_rec *xlrec = (xl_dbase_create_rec *) XLogRecGetData(record);
char *src_path;
char *dst_path;
+ char *parent_path;
struct stat st;
+ bool skip = false;
src_path = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
dst_path = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
@@ -2203,6 +2205,54 @@ dbase_redo(XLogReaderState *record)
(errmsg("some useless files may be left behind in old database directory \"%s\"",
dst_path)));
}
+ else
+ {
+ /*
+ * It is possible that drop tablespace record appearing later in
+ * the WAL as already been replayed. That means we are replaying
+ * the create database record second time, as part of crash
+ * recovery. In that case, the tablespace directory has already
+ * been removed and the create database operation cannot be
+ * replayed. We should skip the replay but remember the missing
+ * tablespace directory, to be matched with a drop tablespace
+ * record later.
+ */
+ parent_path = pstrdup(dst_path);
+ get_parent_directory(parent_path);
+ if (!(stat(parent_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
+ XLogLogMissingDir(xlrec->tablespace_id, InvalidOid, parent_path);
+ skip = true;
+ ereport(WARNING,
+ (errmsg("skipping create database WAL record"),
+ errdetail("Target tablespace \"%s\" not found. We "
+ "expect to encounter a WAL record that "
+ "removes this directory before reaching "
+ "consistent state.", parent_path)));
+ }
+ pfree(parent_path);
+ }
+
+ /*
+ * Source directory may be missing. E.g. the template database used
+ * for creating this database may have been dropped, due to reasons
+ * noted above. Moving a database from one tablespace may also be a
+ * partner in the crime.
+ */
+ if (!(stat(src_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
+ XLogLogMissingDir(xlrec->src_tablespace_id, xlrec->src_db_id, src_path);
+ skip = true;
+ ereport(WARNING,
+ (errmsg("skipping create database WAL record"),
+ errdetail("Source database \"%s\" not found. We expect "
+ "to encounter a WAL record that removes this "
+ "directory before reaching consistent state.",
+ src_path)));
+ }
+
+ if (skip)
+ return;
/*
* Force dirty buffers out to disk, to ensure source database is
@@ -2260,6 +2310,9 @@ dbase_redo(XLogReaderState *record)
ereport(WARNING,
(errmsg("some useless files may be left behind in old database directory \"%s\"",
dst_path)));
+
+ XLogForgetMissingDir(xlrec->tablespace_ids[i], xlrec->db_id);
+
pfree(dst_path);
}
diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c
index 051478057f6..5fd36f93197 100644
--- a/src/backend/commands/tablespace.c
+++ b/src/backend/commands/tablespace.c
@@ -58,6 +58,7 @@
#include "access/xact.h"
#include "access/xlog.h"
#include "access/xloginsert.h"
+#include "access/xlogutils.h"
#include "catalog/catalog.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
@@ -1516,6 +1517,10 @@ tblspc_redo(XLogReaderState *record)
{
xl_tblspc_drop_rec *xlrec = (xl_tblspc_drop_rec *) XLogRecGetData(record);
+ XLogForgetMissingDir(xlrec->ts_id, InvalidOid);
+
+ XLogFlush(record->EndRecPtr);
+
/*
* If we issued a WAL record for a drop tablespace it implies that
* there were no files in it at all when the DROP was done. That means
diff --git a/src/include/access/xlogutils.h b/src/include/access/xlogutils.h
index e59b6cf3a9f..34eecfab791 100644
--- a/src/include/access/xlogutils.h
+++ b/src/include/access/xlogutils.h
@@ -23,6 +23,10 @@ extern void XLogDropDatabase(Oid dbid);
extern void XLogTruncateRelation(RelFileNode rnode, ForkNumber forkNum,
BlockNumber nblocks);
+extern void XLogLogMissingDir(Oid spcNode, Oid dbNode, char *path);
+extern void XLogForgetMissingDir(Oid spcNode, Oid dbNode);
+extern void XLogCheckMissingDirs(void);
+
/* Result codes for XLogReadBufferForRedo[Extended] */
typedef enum
{
--
2.14.3
[application/octet-stream] v10-0004-Fix-database-create-drop-wal-description.patch (1.8K, ../../[email protected]/6-v10-0004-Fix-database-create-drop-wal-description.patch)
download | inline diff:
From 7a002cd379e13794edae53aa926898e33445475d Mon Sep 17 00:00:00 2001
From: Paul Guo <[email protected]>
Date: Mon, 6 Jul 2020 21:20:15 +0800
Subject: [PATCH v10 4/4] Fix database create/drop wal description.
Previously the description messages are wrong since the database path is not
simply tablespce_oid/database_oid. Now we call GetDatabasePath() to get the
correct database path.
---
src/backend/access/rmgrdesc/dbasedesc.c | 16 +++++++++++-----
1 file changed, 11 insertions(+), 5 deletions(-)
diff --git a/src/backend/access/rmgrdesc/dbasedesc.c b/src/backend/access/rmgrdesc/dbasedesc.c
index d82484b9db4..8312ef8bd36 100644
--- a/src/backend/access/rmgrdesc/dbasedesc.c
+++ b/src/backend/access/rmgrdesc/dbasedesc.c
@@ -23,14 +23,17 @@ dbase_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ char *dbpath1, *dbpath2;
if (info == XLOG_DBASE_CREATE)
{
xl_dbase_create_rec *xlrec = (xl_dbase_create_rec *) rec;
- appendStringInfo(buf, "copy dir %u/%u to %u/%u",
- xlrec->src_tablespace_id, xlrec->src_db_id,
- xlrec->tablespace_id, xlrec->db_id);
+ dbpath1 = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
+ dbpath2 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
+ appendStringInfo(buf, "copy dir %s to %s", dbpath1, dbpath2);
+ pfree(dbpath2);
+ pfree(dbpath1);
}
else if (info == XLOG_DBASE_DROP)
{
@@ -39,8 +42,11 @@ dbase_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "dir");
for (i = 0; i < xlrec->ntablespaces; i++)
- appendStringInfo(buf, " %u/%u",
- xlrec->tablespace_ids[i], xlrec->db_id);
+ {
+ dbpath1 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_ids[i]);
+ appendStringInfo(buf, " %s", dbpath1);
+ pfree(dbpath1);
+ }
}
}
--
2.14.3
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-07-08 12:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
@ 2021-01-05 01:07 ` Kyotaro Horiguchi <[email protected]>
2021-01-27 08:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Kyotaro Horiguchi @ 2021-01-05 01:07 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
At Wed, 8 Jul 2020 12:56:44 +0000, Paul Guo <[email protected]> wrote in
> On 2020/01/15 19:18, Paul Guo wrote:
> I further fixed the last test failure (due to a small bug in the test, not in code). Attached are the new patch series. Let's see the CI pipeline result.
>
> Thanks for updating the patches!
>
> I started reading the 0003 patch.
>
> The approach that the 0003 patch uses is not the perfect solution.
> If the standby crashes after tblspc_redo() removes the directory and before
> its subsequent COMMIT record is replayed, PANIC error would occur since
> there can be some unresolved missing directory entries when we reach the
> consistent state. The problem would very rarely happen, though...
> Just idea; calling XLogFlush() to update the minimum recovery point just
> before tblspc_redo() performs destroy_tablespace_directories() may be
> safe and helpful for the problem?
It seems to me that what the current patch does is too complex. What
we need to do here is to remember every invalid operation then forget
it when the prerequisite object is dropped.
When a table space is dropped before consistency is established, we
don't need to care what has been performed inside the tablespace. In
this perspective, it is enough to remember tablespace ids when failed
to do something inside it due to the absence of the tablespace and
then forget it when we remove it. We could remember individual
database id to show them in error messages, but I'm not sure it's
useful. The reason log_invalid_page records block numbers is to allow
the machinery handle partial table truncations, but this is not the
case since dropping tablespace cannot leave some of containing
databases.
As the result, we won't see an unresolved invalid operations in a
dropped tablespace.
Am I missing something?
dbase_redo:
+ if (!(stat(parent_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
+ XLogRecordMissingDir(xlrec->tablespace_id, InvalidOid, parent_path);
This means "record the belonging table space directory if it is not
found OR it is not a directory". The former can be valid but the
latter is unconditionally can not (I don't think we bother considering
symlinks there).
+ /*
+ * Source directory may be missing. E.g. the template database used
+ * for creating this database may have been dropped, due to reasons
+ * noted above. Moving a database from one tablespace may also be a
+ * partner in the crime.
+ */
+ if (!(stat(src_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
+ XLogLogMissingDir(xlrec->src_tablespace_id, xlrec->src_db_id, src_path);
This is a part of *creation* of the target directory. Lack of the
source directory cannot be valid even if the source directory is
dropped afterwards in the WAL stream and we can allow that if the
*target* tablespace is dropped afterwards. As the result, as I
mentioned above, we don't need to record about the database directory.
By the way the name XLogLogMiss.. is somewhat confusing. How about
XLogReportMissingDir (named after report_invalid_page).
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-07-08 12:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-01-05 01:07 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
@ 2021-01-27 08:36 ` Paul Guo <[email protected]>
2021-03-27 14:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Paul Guo @ 2021-01-27 08:36 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: Fujii Masao <[email protected]>; Alvaro Herrera <[email protected]>; Anastasia Lubennikova <[email protected]>; Asim Praveen (Pivotal) <[email protected]>; Lei Wang (Pivotal) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
Thanks for the review, please see the replies below.
> On Jan 5, 2021, at 9:07 AM, Kyotaro Horiguchi <[email protected]> wrote:
>
> At Wed, 8 Jul 2020 12:56:44 +0000, Paul Guo <[email protected]> wrote in
>> On 2020/01/15 19:18, Paul Guo wrote:
>> I further fixed the last test failure (due to a small bug in the test, not in code). Attached are the new patch series. Let's see the CI pipeline result.
>>
>> Thanks for updating the patches!
>>
>> I started reading the 0003 patch.
>>
>> The approach that the 0003 patch uses is not the perfect solution.
>> If the standby crashes after tblspc_redo() removes the directory and before
>> its subsequent COMMIT record is replayed, PANIC error would occur since
>> there can be some unresolved missing directory entries when we reach the
>> consistent state. The problem would very rarely happen, though...
>> Just idea; calling XLogFlush() to update the minimum recovery point just
>> before tblspc_redo() performs destroy_tablespace_directories() may be
>> safe and helpful for the problem?
>
> It seems to me that what the current patch does is too complex. What
> we need to do here is to remember every invalid operation then forget
> it when the prerequisite object is dropped.
>
> When a table space is dropped before consistency is established, we
> don't need to care what has been performed inside the tablespace. In
> this perspective, it is enough to remember tablespace ids when failed
> to do something inside it due to the absence of the tablespace and
> then forget it when we remove it. We could remember individual
> database id to show them in error messages, but I'm not sure it's
> useful. The reason log_invalid_page records block numbers is to allow
> the machinery handle partial table truncations, but this is not the
> case since dropping tablespace cannot leave some of containing
> databases.
>
> As the result, we won't see an unresolved invalid operations in a
> dropped tablespace.
>
> Am I missing something?
Yes, removing the database id from the hash key in the log/forget code should
be usually fine, but the previous code does stricter/safer checking.
Consider the scenario:
CREATE DATABASE newdb1 TEMPLATE template_db1;
CREATE DATABASE newdb2 TEMPLATE template_db2; <- in case the template_db2 database directory is missing abnormally somehow.
DROP DATABASE template_db1;
The previous code could detect this but if we remove the database id in the code,
this bad scenario is skipped.
>
>
> dbase_redo:
> + if (!(stat(parent_path, &st) == 0 && S_ISDIR(st.st_mode)))
> + {
> + XLogRecordMissingDir(xlrec->tablespace_id, InvalidOid, parent_path);
>
> This means "record the belonging table space directory if it is not
> found OR it is not a directory". The former can be valid but the
> latter is unconditionally can not (I don't think we bother considering
> symlinks there).
Again this is a safer check, in the case the parent_path is a file for example somehow,
we should panic finally for the case and let the user checks and then does recovery again.
>
> + /*
> + * Source directory may be missing. E.g. the template database used
> + * for creating this database may have been dropped, due to reasons
> + * noted above. Moving a database from one tablespace may also be a
> + * partner in the crime.
> + */
> + if (!(stat(src_path, &st) == 0 && S_ISDIR(st.st_mode)))
> + {
> + XLogLogMissingDir(xlrec->src_tablespace_id, xlrec->src_db_id, src_path);
>
> This is a part of *creation* of the target directory. Lack of the
> source directory cannot be valid even if the source directory is
> dropped afterwards in the WAL stream and we can allow that if the
> *target* tablespace is dropped afterwards. As the result, as I
> mentioned above, we don't need to record about the database directory.
>
> By the way the name XLogLogMiss.. is somewhat confusing. How about
> XLogReportMissingDir (named after report_invalid_page).
Agree with you.
Also your words remind me that we should skip the checking if the consistency point
is reached.
Here is a git diff against the previous patch. I’ll send out the new rebased patches after
the consensus is reached.
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index 7ade385965..c8fe3fe228 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -90,7 +90,7 @@ typedef struct xl_missing_dir
static HTAB *missing_dir_tab = NULL;
void
-XLogLogMissingDir(Oid spcNode, Oid dbNode, char *path)
+XLogReportMissingDir(Oid spcNode, Oid dbNode, char *path)
{
xl_missing_dir_key key;
bool found;
@@ -103,16 +103,6 @@ XLogLogMissingDir(Oid spcNode, Oid dbNode, char *path)
*/
Assert(OidIsValid(spcNode));
- if (reachedConsistency)
- {
- if (dbNode == InvalidOid)
- elog(PANIC, "cannot find directory %s (tablespace %d)",
- path, spcNode);
- else
- elog(PANIC, "cannot find directory %s (tablespace %d database %d)",
- path, spcNode, dbNode);
- }
-
if (missing_dir_tab == NULL)
{
/* create hash table when first needed */
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index fbff422c3b..7bd6d4efd9 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -2205,7 +2205,7 @@ dbase_redo(XLogReaderState *record)
(errmsg("some useless files may be left behind in old database directory \"%s\"",
dst_path)));
}
- else
+ else if (!reachedConsistency)
{
/*
* It is possible that drop tablespace record appearing later in
@@ -2221,7 +2221,7 @@ dbase_redo(XLogReaderState *record)
get_parent_directory(parent_path);
if (!(stat(parent_path, &st) == 0 && S_ISDIR(st.st_mode)))
{
- XLogLogMissingDir(xlrec->tablespace_id, InvalidOid, parent_path);
+ XLogReportMissingDir(xlrec->tablespace_id, InvalidOid, parent_path);
skip = true;
ereport(WARNING,
(errmsg("skipping create database WAL record"),
@@ -2239,9 +2239,10 @@ dbase_redo(XLogReaderState *record)
* noted above. Moving a database from one tablespace may also be a
* partner in the crime.
*/
- if (!(stat(src_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ if (!(stat(src_path, &st) == 0 && S_ISDIR(st.st_mode)) &&
+ !reachedConsistency)
{
- XLogLogMissingDir(xlrec->src_tablespace_id, xlrec->src_db_id, src_path);
+ XLogReportMissingDir(xlrec->src_tablespace_id, xlrec->src_db_id, src_path);
skip = true;
ereport(WARNING,
(errmsg("skipping create database WAL record"),
@@ -2311,7 +2312,8 @@ dbase_redo(XLogReaderState *record)
(errmsg("some useless files may be left behind in old database directory \"%s\"",
dst_path)));
- XLogForgetMissingDir(xlrec->tablespace_ids[i], xlrec->db_id);
+ if (!reachedConsistency)
+ XLogForgetMissingDir(xlrec->tablespace_ids[i], xlrec->db_id);
pfree(dst_path);
}
diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c
index 294c9676b4..15eaa757cc 100644
--- a/src/backend/commands/tablespace.c
+++ b/src/backend/commands/tablespace.c
@@ -1534,7 +1534,8 @@ tblspc_redo(XLogReaderState *record)
{
xl_tblspc_drop_rec *xlrec = (xl_tblspc_drop_rec *) XLogRecGetData(record);
- XLogForgetMissingDir(xlrec->ts_id, InvalidOid);
+ if (!reachedConsistency)
+ XLogForgetMissingDir(xlrec->ts_id, InvalidOid);
XLogFlush(record->EndRecPtr);
diff --git a/src/include/access/xlogutils.h b/src/include/access/xlogutils.h
index da561af5ab..6561d9cebe 100644
--- a/src/include/access/xlogutils.h
+++ b/src/include/access/xlogutils.h
@@ -23,7 +23,7 @@ extern void XLogDropDatabase(Oid dbid);
extern void XLogTruncateRelation(RelFileNode rnode, ForkNumber forkNum,
BlockNumber nblocks);
-extern void XLogLogMissingDir(Oid spcNode, Oid dbNode, char *path);
+extern void XLogReportMissingDir(Oid spcNode, Oid dbNode, char *path);
extern void XLogForgetMissingDir(Oid spcNode, Oid dbNode);
extern void XLogCheckMissingDirs(void);
diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl
index 748200ebb5..95eb6d26cc 100644
--- a/src/test/recovery/t/011_crash_recovery.pl
+++ b/src/test/recovery/t/011_crash_recovery.pl
@@ -141,7 +141,7 @@ $node_master->wait_for_catchup($node_standby, 'replay',
$node_standby->safe_psql('postgres', 'CHECKPOINT');
# Do immediate shutdown just after a sequence of CREAT DATABASE / DROP
-# DATABASE / DROP TABLESPACE. This causes CREATE DATBASE WAL records
+# DATABASE / DROP TABLESPACE. This causes CREATE DATABASE WAL records
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-07-08 12:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-01-05 01:07 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-01-27 08:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
@ 2021-03-27 14:23 ` Alvaro Herrera <[email protected]>
2021-03-30 07:12 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Alvaro Herrera @ 2021-03-27 14:23 UTC (permalink / raw)
To: Paul Guo <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; Fujii Masao <[email protected]>; Anastasia Lubennikova <[email protected]>; Asim Praveen (Pivotal) <[email protected]>; Lei Wang (Pivotal) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On 2021-Jan-27, Paul Guo wrote:
> Here is a git diff against the previous patch. I’ll send out the new
> rebased patches after the consensus is reached.
Hmm, can you post a rebased set, where the points under discussion
are marked in XXX comments explaining what the issue is? This thread is
long and old ago that it's pretty hard to navigate the whole thing in
order to find out exactly what is being questioned.
I think 0004 can be pushed without further ado, since it's a clear and
simple fix. 0001 needs a comment about the new parameter in
RecursiveCopy's POD documentation.
As I understand, this is a backpatchable bug-fix.
--
Álvaro Herrera 39°49'30"S 73°17'W
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-07-08 12:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-01-05 01:07 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-01-27 08:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-03-27 14:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2021-03-30 07:12 ` Paul Guo <[email protected]>
2021-07-09 19:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Ibrar Ahmed <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Paul Guo @ 2021-03-30 07:12 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; Fujii Masao <[email protected]>; Anastasia Lubennikova <[email protected]>; Asim Praveen (Pivotal) <[email protected]>; Lei Wang (Pivotal) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On 2021/3/27, 10:23 PM, "Alvaro Herrera" <[email protected]> wrote:
> Hmm, can you post a rebased set, where the points under discussion
> are marked in XXX comments explaining what the issue is? This thread is
> long and old ago that it's pretty hard to navigate the whole thing in
> order to find out exactly what is being questioned.
OK. Attached are the rebased version that includes the change I discussed
in my previous reply. Also added POD documentation change for RecursiveCopy,
and modified the patch to use the backup_options introduced in
081876d75ea15c3bd2ee5ba64a794fd8ea46d794 for tablespace mapping.
> I think 0004 can be pushed without further ado, since it's a clear and
> simple fix. 0001 needs a comment about the new parameter in
> RecursiveCopy's POD documentation.
Yeah, 0004 is no any risky. One concern seemed to be the compatibility of some
WAL dump/analysis tools(?). I have no idea about this. But if we do not backport
0004 we do not seem to need to worry about this.
> As I understand, this is a backpatchable bug-fix.
Yes.
Thanks.
Attachments:
[application/octet-stream] v11-0001-Support-node-initialization-from-backup-with-tab.patch (4.9K, ../../[email protected]/2-v11-0001-Support-node-initialization-from-backup-with-tab.patch)
download | inline diff:
From 4075252c30fd5728913ef78b99f6c6cc70875ecd Mon Sep 17 00:00:00 2001
From: Asim R P <[email protected]>
Date: Fri, 20 Sep 2019 17:31:25 +0530
Subject: [PATCH v11 1/4] Support node initialization from backup with
tablespaces
User defined tablespaces appear as symlinks in in the backup. This
commit tweaks recursive copy subroutine to allow for symlinks specific
to tablespaces.
Authored by Kyotaro Horiguchi
---
src/test/perl/PostgresNode.pm | 29 ++++++++++++++++++++++++++-
src/test/perl/RecursiveCopy.pm | 45 +++++++++++++++++++++++++++++++++++-------
2 files changed, 66 insertions(+), 8 deletions(-)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 1e96357d7e..e27120e975 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -607,6 +607,32 @@ sub backup_fs_cold
return;
}
+sub _srcsymlink
+{
+ my ($srcpath, $destpath) = @_;
+
+ croak "Cannot operate on symlink \"$srcpath\""
+ if ($srcpath !~ qr{/(pg_tblspc/[0-9]+)$});
+
+ # We have mapped tablespaces. Copy them individually
+ my $tmpdir = TestLib::tempdir;
+ my $dstrealdir = TestLib::perl2host($tmpdir);
+ my $srcrealdir = readlink($srcpath);
+
+ opendir(my $dh, $srcrealdir);
+ while (my $entry = (readdir $dh))
+ {
+ next if ($entry eq '.' or $entry eq '..');
+ my $spath = "$srcrealdir/$entry";
+ my $dpath = "$dstrealdir/$entry";
+ RecursiveCopy::copypath($spath, $dpath);
+ }
+ closedir $dh;
+
+ symlink $dstrealdir, $destpath;
+
+ return 1;
+}
# Common sub of backup_fs_hot and backup_fs_cold
sub _backup_fs
@@ -715,7 +741,8 @@ sub init_from_backup
else
{
rmdir($data_path);
- RecursiveCopy::copypath($backup_path, $data_path);
+ RecursiveCopy::copypath($backup_path, $data_path,
+ srcsymlinkfn => \&_srcsymlink);
}
chmod(0700, $data_path);
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
index baf5d0ac63..aaa3ce8a1b 100644
--- a/src/test/perl/RecursiveCopy.pm
+++ b/src/test/perl/RecursiveCopy.pm
@@ -47,6 +47,11 @@ This subroutine will be called for each entry in the source directory with its
relative path as only parameter; if the subroutine returns true the entry is
copied, otherwise the file is skipped.
+If the B<srcsymlinkfn> parameter is given, it must be a subroutine reference.
+This subroutine will be called when the source directory is a symlink. It
+determines the mechanism that copies files from the source directory to the
+destination directory.
+
On failure the target directory may be in some incomplete state; no cleanup is
attempted.
@@ -66,6 +71,7 @@ sub copypath
{
my ($base_src_dir, $base_dest_dir, %params) = @_;
my $filterfn;
+ my $srcsymlinkfn;
if (defined $params{filterfn})
{
@@ -80,31 +86,55 @@ sub copypath
$filterfn = sub { return 1; };
}
+ if (defined $params{srcsymlinkfn})
+ {
+ croak "if specified, srcsymlinkfn must be a subroutine reference"
+ unless defined(ref $params{srcsymlinkfn})
+ and (ref $params{srcsymlinkfn} eq 'CODE');
+
+ $srcsymlinkfn = $params{srcsymlinkfn};
+ }
+ else
+ {
+ $srcsymlinkfn = undef;
+ }
+
# Complain if original path is bogus, because _copypath_recurse won't.
croak "\"$base_src_dir\" does not exist" if !-e $base_src_dir;
# Start recursive copy from current directory
- return _copypath_recurse($base_src_dir, $base_dest_dir, "", $filterfn);
+ return _copypath_recurse($base_src_dir, $base_dest_dir, "", $filterfn, $srcsymlinkfn);
}
# Recursive private guts of copypath
sub _copypath_recurse
{
- my ($base_src_dir, $base_dest_dir, $curr_path, $filterfn) = @_;
+ my ($base_src_dir, $base_dest_dir, $curr_path, $filterfn,
+ $srcsymlinkfn) = @_;
my $srcpath = "$base_src_dir/$curr_path";
my $destpath = "$base_dest_dir/$curr_path";
# invoke the filter and skip all further operation if it returns false
return 1 unless &$filterfn($curr_path);
- # Check for symlink -- needed only on source dir
- # (note: this will fall through quietly if file is already gone)
- croak "Cannot operate on symlink \"$srcpath\"" if -l $srcpath;
-
# Abort if destination path already exists. Should we allow directories
# to exist already?
croak "Destination path \"$destpath\" already exists" if -e $destpath;
+ # Check for symlink -- needed only on source dir
+ # If caller provided us with a callback, call it; otherwise we're out.
+ if (-l $srcpath)
+ {
+ if (defined $srcsymlinkfn)
+ {
+ return &$srcsymlinkfn($srcpath, $destpath);
+ }
+ else
+ {
+ croak "Cannot operate on symlink \"$srcpath\"";
+ }
+ }
+
# If this source path is a file, simply copy it to destination with the
# same name and we're done.
if (-f $srcpath)
@@ -137,7 +167,8 @@ sub _copypath_recurse
{
next if ($entry eq '.' or $entry eq '..');
_copypath_recurse($base_src_dir, $base_dest_dir,
- $curr_path eq '' ? $entry : "$curr_path/$entry", $filterfn)
+ $curr_path eq '' ? $entry : "$curr_path/$entry", $filterfn,
+ $srcsymlinkfn)
or die "copypath $srcpath/$entry -> $destpath/$entry failed";
}
--
2.14.3
[application/octet-stream] v11-0002-Tests-to-replay-create-database-operation-on-sta.patch (9.7K, ../../[email protected]/3-v11-0002-Tests-to-replay-create-database-operation-on-sta.patch)
download | inline diff:
From 154497b47de2de423b1ecf420d45abf2af0e1139 Mon Sep 17 00:00:00 2001
From: Asim R P <[email protected]>
Date: Fri, 20 Sep 2019 17:34:19 +0530
Subject: [PATCH v11 2/4] Tests to replay create database operation on standby
The tests demonstrate that standby fails to replay a create database
WAL record during crash recovery, if one or more of underlying
directories are missing from the file system. This can happen if a
drop tablespace or drop database WAL record has been replayed in
archive recovery, before a crash. And then the create database record
happens to be replayed again during crash recovery. The failures
indicate bugs that need to be fixed.
The first test, TEST 4, performs several DDL operations resulting in a
database directory being removed, along with a few create database
operations. It expects crash recovery to succeed because for each
missing directory encountered during create database replay, a matching
drop tablespace or drop database WAL record is found later.
Second test, TEST 5, validates that a standby rightfully aborts replay
during archive recovery, if a missing directory is encountered when
replaying create database WAL record.
These tests have been proposed and implemented in various ways by
Alexandra Wang, Anastasia Lubennikova, Kyotaro Horiguchi, Paul Guo and me.
---
src/test/perl/PostgresNode.pm | 21 +++-
src/test/recovery/t/011_crash_recovery.pl | 162 +++++++++++++++++++++++++++++-
2 files changed, 177 insertions(+), 6 deletions(-)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index e27120e975..d4e17644b4 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -1888,15 +1888,26 @@ Returns 1 if successful, 0 if timed out.
sub poll_query_until
{
- my ($self, $dbname, $query, $expected) = @_;
+ my ($self, $dbname, $query, $params) = @_;
+ my $expected;
+
+ # Be backwards-compatible
+ if (defined $params and ref $params eq '')
+ {
+ $params = {
+ expected => $params,
+ timeout => 180
+ };
+ }
local %ENV = $self->_get_env();
- $expected = 't' unless defined($expected); # default value
+ $params->{expected} = 't' unless defined($params->{expected});
+ $params->{timeout} = 180 unless defined($params->{timeout});
my $cmd = [ 'psql', '-XAt', '-c', $query, '-d', $self->connstr($dbname) ];
my ($stdout, $stderr);
- my $max_attempts = 180 * 10;
+ my $max_attempts = $params->{timeout} * 10;
my $attempts = 0;
while ($attempts < $max_attempts)
@@ -1906,7 +1917,7 @@ sub poll_query_until
$stdout =~ s/\r\n/\n/g if $Config{osname} eq 'msys';
chomp($stdout);
- if ($stdout eq $expected)
+ if ($stdout eq $params->{expected})
{
return 1;
}
@@ -1924,7 +1935,7 @@ sub poll_query_until
diag qq(poll_query_until timed out executing this query:
$query
expecting this output:
-$expected
+$params->{expected}
last actual query output:
$stdout
with stderr:
diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl
index 10cd98f70a..faae3fb5e7 100644
--- a/src/test/recovery/t/011_crash_recovery.pl
+++ b/src/test/recovery/t/011_crash_recovery.pl
@@ -6,9 +6,10 @@ use warnings;
use PostgresNode;
use TestLib;
use Test::More;
+use File::Path qw(rmtree);
use Config;
-plan tests => 3;
+plan tests => 5;
my $node = get_new_node('primary');
$node->init(allows_streaming => 1);
@@ -59,3 +60,162 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]),
$stdin .= "\\q\n";
$tx->finish; # wait for psql to quit gracefully
+
+# TEST 4
+#
+# Ensure that a missing tablespace directory during crash recovery on
+# a standby is handled correctly. The standby should finish crash
+# recovery successfully because a matching drop database record is
+# found in the WAL. The following scnearios are covered:
+#
+# 1. Create a database against a user-defined tablespace then drop the
+# database.
+#
+# 2. Create a database against a user-defined tablespace then drop the
+# database and the tablespace.
+#
+# 3. Move a database from source tablespace to target tablespace then
+# drop the source tablespace.
+#
+# 4. Create a database from another database as template then drop the
+# template database.
+#
+#
+
+my $node_master = get_new_node('master2');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Create tablespace
+my $dropme_ts_master1 = TestLib::tempdir;
+$dropme_ts_master1 = TestLib::perl2host($dropme_ts_master1);
+my $dropme_ts_master2 = TestLib::tempdir;
+$dropme_ts_master2 = TestLib::perl2host($dropme_ts_master2);
+my $source_ts_master = TestLib::tempdir;
+$source_ts_master = TestLib::perl2host($source_ts_master);
+my $target_ts_master = TestLib::tempdir;
+$target_ts_master = TestLib::perl2host($target_ts_master);
+
+$node_master->safe_psql('postgres',
+ qq[CREATE TABLESPACE dropme_ts1 location '$dropme_ts_master1';
+ CREATE TABLESPACE dropme_ts2 location '$dropme_ts_master2';
+ CREATE TABLESPACE source_ts location '$source_ts_master';
+ CREATE TABLESPACE target_ts location '$target_ts_master';
+ CREATE DATABASE template_db IS_TEMPLATE = true;]);
+
+my $dropme_ts_standby1 = TestLib::tempdir;
+$dropme_ts_standby1 = TestLib::perl2host($dropme_ts_standby1);
+my $dropme_ts_standby2 = TestLib::tempdir;
+$dropme_ts_standby2 = TestLib::perl2host($dropme_ts_standby2);
+my $source_ts_standby = TestLib::tempdir;
+$source_ts_standby = TestLib::perl2host($source_ts_standby);
+my $target_ts_standby = TestLib::tempdir;
+$target_ts_standby = TestLib::perl2host($target_ts_standby);
+
+# Take backup
+my $backup_name = 'my_backup';
+my $ts_mapping = [ "--tablespace-mapping=$dropme_ts_master1=$dropme_ts_standby1",
+ "--tablespace-mapping=$dropme_ts_master2=$dropme_ts_standby2",
+ "--tablespace-mapping=$source_ts_master=$source_ts_standby",
+ "--tablespace-mapping=$target_ts_master=$target_ts_standby" ];
+$node_master->backup($backup_name, backup_options => $ts_mapping);
+
+my $node_standby = get_new_node('standby2');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->start;
+
+# Make sure connection is made
+$node_master->poll_query_until(
+ 'postgres', 'SELECT count(*) = 1 FROM pg_stat_replication');
+
+# Make sure to perform restartpoint after tablespace creation
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->safe_psql('postgres', 'CHECKPOINT');
+
+# Do immediate shutdown just after a sequence of CREAT DATABASE / DROP
+# DATABASE / DROP TABLESPACE. This causes CREATE DATABASE WAL records
+# to be applied to already-removed directories.
+$node_master->safe_psql('postgres',
+ q[CREATE DATABASE dropme_db1 WITH TABLESPACE dropme_ts1;
+ CREATE DATABASE dropme_db2 WITH TABLESPACE dropme_ts2;
+ CREATE DATABASE moveme_db TABLESPACE source_ts;
+ ALTER DATABASE moveme_db SET TABLESPACE target_ts;
+ CREATE DATABASE newdb TEMPLATE template_db;
+ ALTER DATABASE template_db IS_TEMPLATE = false;
+ DROP DATABASE dropme_db1;
+ DROP DATABASE dropme_db2; DROP TABLESPACE dropme_ts2;
+ DROP TABLESPACE source_ts;
+ DROP DATABASE template_db;]);
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->stop('immediate');
+
+# Should restart ignoring directory creation error.
+is($node_standby->start(fail_ok => 1), 1);
+
+# TEST 5
+#
+# Ensure that a missing tablespace directory during create database
+# replay immediately causes panic if the standby has already reached
+# consistent state (archive recovery is in progress).
+
+$node_master = get_new_node('master3');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Create tablespace
+my $ts_master = TestLib::tempdir;
+$ts_master = TestLib::perl2host($ts_master);
+$node_master->safe_psql('postgres', "CREATE TABLESPACE ts1 LOCATION '$ts_master'");
+$node_master->safe_psql('postgres', "CREATE DATABASE db1 TABLESPACE ts1");
+
+my $ts_standby = TestLib::tempdir("standby");
+$ts_standby = TestLib::perl2host($ts_standby);
+
+# Take backup
+$backup_name = 'my_backup';
+$node_master->backup($backup_name,
+ backup_options =>
+ [ "--tablespace-mapping=$ts_master=$ts_standby" ]);
+$node_standby = get_new_node('standby3');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->start;
+
+# Make sure standby reached consistency and starts accepting connections
+$node_standby->poll_query_until('postgres', 'SELECT 1', '1');
+
+# Remove standby tablespace directory so it will be missing when
+# replay resumes.
+#
+# The tablespace mapping is lost when the standby node is initialized
+# from basebackup because RecursiveCopy::copypath creates a new temp
+# directory for each tablspace symlink found in backup. We must
+# obtain the correct tablespace directory by querying standby.
+$ts_standby = $node_standby->safe_psql(
+ 'postgres',
+ "select pg_tablespace_location(oid) from pg_tablespace where spcname = 'ts1'");
+rmtree($ts_standby);
+
+# Create a database in the tablespace and a table in default tablespace
+$node_master->safe_psql('postgres',
+ q[CREATE TABLE should_not_replay_insertion(a int);
+ CREATE DATABASE db2 WITH TABLESPACE ts1;
+ INSERT INTO should_not_replay_insertion VALUES (1);]);
+
+# Standby should fail and should not silently skip replaying the wal
+if ($node_master->poll_query_until(
+ 'postgres',
+ 'SELECT count(*) = 0 FROM pg_stat_replication',
+ 't') == 1)
+{
+ pass('standby failed as expected');
+ # We know that the standby has failed. Setting its pid to
+ # undefined avoids error when PostgreNode module tries to stop the
+ # standby node as part of tear_down sequence.
+ $node_standby->{_pid} = undef;
+}
+else
+{
+ fail('standby did not fail within 5 seconds');
+}
--
2.14.3
[application/octet-stream] v11-0003-Fix-replay-of-create-database-records-on-standby.patch (10.6K, ../../[email protected]/4-v11-0003-Fix-replay-of-create-database-records-on-standby.patch)
download | inline diff:
From 16bf763dda617d6aaf454f8aff3c4d08e09b198b Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <[email protected]>
Date: Thu, 9 Jan 2020 17:54:40 -0300
Subject: [PATCH v11 3/4] Fix replay of create database records on standby
Crash recovery on standby may encounter missing directories when
replaying create database WAL records. Prior to this patch, the
standby would fail to recover in such a case. However, the
directories could be legitimately missing. Consider a sequence of WAL
records as follows:
CREATE DATABASE
DROP DATABASE
DROP TABLESPACE
If, after replaying the last WAL record and removing the tablespace
directory, the standby crashes and has to replay the create database
record again, the crash recovery must be able to move on.
This patch adds mechanism similar to invalid page hash table, to track
missing directories during crash recovery. If all the missing
directory references are matched with corresponding drop records at
the end of crash recovery, the standby can safely enter archive
recovery.
Bug identified by Paul Guo.
Authored by Paul Guo, Kyotaro Horiguchi and Asim R P.
---
src/backend/access/transam/xlog.c | 6 ++
src/backend/access/transam/xlogutils.c | 145 +++++++++++++++++++++++++++++++++
src/backend/commands/dbcommands.c | 55 +++++++++++++
src/backend/commands/tablespace.c | 6 ++
src/include/access/xlogutils.h | 4 +
5 files changed, 216 insertions(+)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 6f8810e149..b525679e8b 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8175,6 +8175,12 @@ CheckRecoveryConsistency(void)
*/
XLogCheckInvalidPages();
+ /*
+ * Check if the XLOG sequence contained any unresolved references to
+ * missing directories.
+ */
+ XLogCheckMissingDirs();
+
reachedConsistency = true;
ereport(LOG,
(errmsg("consistent recovery state reached at %X/%X",
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index d17d660f46..8341b7a242 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -59,6 +59,151 @@ typedef struct xl_invalid_page
static HTAB *invalid_page_tab = NULL;
+/*
+ * If a create database WAL record is being replayed more than once during
+ * crash recovery on a standby, it is possible that either the tablespace
+ * directory or the template database directory is missing. This happens when
+ * the directories are removed by replay of subsequent drop records. Note
+ * that this problem happens only on standby and not on master. On master, a
+ * checkpoint is created at the end of create database operation. On standby,
+ * however, such a strategy (creating restart points during replay) is not
+ * viable because it will slow down WAL replay.
+ *
+ * The alternative is to track references to each missing directory
+ * encountered when performing crash recovery in the following hash table.
+ * Similar to invalid page table above, the expectation is that each missing
+ * directory entry should be matched with a drop database or drop tablespace
+ * WAL record by the end of crash recovery.
+ */
+typedef struct xl_missing_dir_key
+{
+ Oid spcNode;
+ Oid dbNode;
+} xl_missing_dir_key;
+
+typedef struct xl_missing_dir
+{
+ xl_missing_dir_key key;
+ char path[MAXPGPATH];
+} xl_missing_dir;
+
+static HTAB *missing_dir_tab = NULL;
+
+void
+XLogReportMissingDir(Oid spcNode, Oid dbNode, char *path)
+{
+ xl_missing_dir_key key;
+ bool found;
+ xl_missing_dir *entry;
+
+ /*
+ * Database OID may be invalid but tablespace OID must be valid. If
+ * dbNode is InvalidOid, we are logging a missing tablespace directory,
+ * otherwise we are logging a missing database directory.
+ */
+ Assert(OidIsValid(spcNode));
+
+ if (missing_dir_tab == NULL)
+ {
+ /* create hash table when first needed */
+ HASHCTL ctl;
+
+ memset(&ctl, 0, sizeof(ctl));
+ ctl.keysize = sizeof(xl_missing_dir_key);
+ ctl.entrysize = sizeof(xl_missing_dir);
+
+ missing_dir_tab = hash_create("XLOG missing directory table",
+ 100,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS);
+ }
+
+ key.spcNode = spcNode;
+ key.dbNode = dbNode;
+
+ entry = hash_search(missing_dir_tab, &key, HASH_ENTER, &found);
+
+ if (found)
+ {
+ if (dbNode == InvalidOid)
+ elog(DEBUG2, "missing directory %s (tablespace %d) already exists: %s",
+ path, spcNode, entry->path);
+ else
+ elog(DEBUG2, "missing directory %s (tablespace %d database %d) already exists: %s",
+ path, spcNode, dbNode, entry->path);
+ }
+ else
+ {
+ strlcpy(entry->path, path, sizeof(entry->path));
+ if (dbNode == InvalidOid)
+ elog(DEBUG2, "logged missing dir %s (tablespace %d)",
+ path, spcNode);
+ else
+ elog(DEBUG2, "logged missing dir %s (tablespace %d database %d)",
+ path, spcNode, dbNode);
+ }
+}
+
+void
+XLogForgetMissingDir(Oid spcNode, Oid dbNode)
+{
+ xl_missing_dir_key key;
+
+ key.spcNode = spcNode;
+ key.dbNode = dbNode;
+
+ /* Database OID may be invalid but tablespace OID must be valid. */
+ Assert(OidIsValid(spcNode));
+
+ if (missing_dir_tab == NULL)
+ return;
+
+ if (hash_search(missing_dir_tab, &key, HASH_REMOVE, NULL) != NULL)
+ {
+ if (dbNode == InvalidOid)
+ {
+ elog(DEBUG2, "forgot missing dir (tablespace %d)", spcNode);
+ }
+ else
+ {
+ char *path = GetDatabasePath(dbNode, spcNode);
+
+ elog(DEBUG2, "forgot missing dir %s (tablespace %d database %d)",
+ path, spcNode, dbNode);
+ pfree(path);
+ }
+ }
+}
+
+/*
+ * This is called at the end of crash recovery, before entering archive
+ * recovery on a standby. PANIC if the hash table is not empty.
+ */
+void
+XLogCheckMissingDirs(void)
+{
+ HASH_SEQ_STATUS status;
+ xl_missing_dir *hentry;
+ bool foundone = false;
+
+ if (missing_dir_tab == NULL)
+ return; /* nothing to do */
+
+ hash_seq_init(&status, missing_dir_tab);
+
+ while ((hentry = (xl_missing_dir *) hash_seq_search(&status)) != NULL)
+ {
+ elog(WARNING, "missing directory \"%s\" tablespace %d database %d",
+ hentry->path, hentry->key.spcNode, hentry->key.dbNode);
+ foundone = true;
+ }
+
+ if (foundone)
+ elog(PANIC, "WAL contains references to missing directories");
+
+ hash_destroy(missing_dir_tab);
+ missing_dir_tab = NULL;
+}
/* Report a reference to an invalid page */
static void
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index 2b159b60eb..7bd6d4efd9 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -2185,7 +2185,9 @@ dbase_redo(XLogReaderState *record)
xl_dbase_create_rec *xlrec = (xl_dbase_create_rec *) XLogRecGetData(record);
char *src_path;
char *dst_path;
+ char *parent_path;
struct stat st;
+ bool skip = false;
src_path = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
dst_path = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
@@ -2203,6 +2205,55 @@ dbase_redo(XLogReaderState *record)
(errmsg("some useless files may be left behind in old database directory \"%s\"",
dst_path)));
}
+ else if (!reachedConsistency)
+ {
+ /*
+ * It is possible that drop tablespace record appearing later in
+ * the WAL as already been replayed. That means we are replaying
+ * the create database record second time, as part of crash
+ * recovery. In that case, the tablespace directory has already
+ * been removed and the create database operation cannot be
+ * replayed. We should skip the replay but remember the missing
+ * tablespace directory, to be matched with a drop tablespace
+ * record later.
+ */
+ parent_path = pstrdup(dst_path);
+ get_parent_directory(parent_path);
+ if (!(stat(parent_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
+ XLogReportMissingDir(xlrec->tablespace_id, InvalidOid, parent_path);
+ skip = true;
+ ereport(WARNING,
+ (errmsg("skipping create database WAL record"),
+ errdetail("Target tablespace \"%s\" not found. We "
+ "expect to encounter a WAL record that "
+ "removes this directory before reaching "
+ "consistent state.", parent_path)));
+ }
+ pfree(parent_path);
+ }
+
+ /*
+ * Source directory may be missing. E.g. the template database used
+ * for creating this database may have been dropped, due to reasons
+ * noted above. Moving a database from one tablespace may also be a
+ * partner in the crime.
+ */
+ if (!(stat(src_path, &st) == 0 && S_ISDIR(st.st_mode)) &&
+ !reachedConsistency)
+ {
+ XLogReportMissingDir(xlrec->src_tablespace_id, xlrec->src_db_id, src_path);
+ skip = true;
+ ereport(WARNING,
+ (errmsg("skipping create database WAL record"),
+ errdetail("Source database \"%s\" not found. We expect "
+ "to encounter a WAL record that removes this "
+ "directory before reaching consistent state.",
+ src_path)));
+ }
+
+ if (skip)
+ return;
/*
* Force dirty buffers out to disk, to ensure source database is
@@ -2260,6 +2311,10 @@ dbase_redo(XLogReaderState *record)
ereport(WARNING,
(errmsg("some useless files may be left behind in old database directory \"%s\"",
dst_path)));
+
+ if (!reachedConsistency)
+ XLogForgetMissingDir(xlrec->tablespace_ids[i], xlrec->db_id);
+
pfree(dst_path);
}
diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c
index 69ea155d50..15eaa757cc 100644
--- a/src/backend/commands/tablespace.c
+++ b/src/backend/commands/tablespace.c
@@ -58,6 +58,7 @@
#include "access/xact.h"
#include "access/xlog.h"
#include "access/xloginsert.h"
+#include "access/xlogutils.h"
#include "catalog/catalog.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
@@ -1533,6 +1534,11 @@ tblspc_redo(XLogReaderState *record)
{
xl_tblspc_drop_rec *xlrec = (xl_tblspc_drop_rec *) XLogRecGetData(record);
+ if (!reachedConsistency)
+ XLogForgetMissingDir(xlrec->ts_id, InvalidOid);
+
+ XLogFlush(record->EndRecPtr);
+
/*
* If we issued a WAL record for a drop tablespace it implies that
* there were no files in it at all when the DROP was done. That means
diff --git a/src/include/access/xlogutils.h b/src/include/access/xlogutils.h
index 9ac602b674..6561d9cebe 100644
--- a/src/include/access/xlogutils.h
+++ b/src/include/access/xlogutils.h
@@ -23,6 +23,10 @@ extern void XLogDropDatabase(Oid dbid);
extern void XLogTruncateRelation(RelFileNode rnode, ForkNumber forkNum,
BlockNumber nblocks);
+extern void XLogReportMissingDir(Oid spcNode, Oid dbNode, char *path);
+extern void XLogForgetMissingDir(Oid spcNode, Oid dbNode);
+extern void XLogCheckMissingDirs(void);
+
/* Result codes for XLogReadBufferForRedo[Extended] */
typedef enum
{
--
2.14.3
[application/octet-stream] v11-0004-Fix-database-create-drop-wal-description.patch (1.9K, ../../[email protected]/5-v11-0004-Fix-database-create-drop-wal-description.patch)
download | inline diff:
From 1939f82fc67d218b6990b8a694709f0a7cb59430 Mon Sep 17 00:00:00 2001
From: Paul Guo <[email protected]>
Date: Mon, 6 Jul 2020 21:20:15 +0800
Subject: [PATCH v11 4/4] Fix database create/drop wal description.
Previously the description messages are wrong since the database path is not
simply tablespce_oid/database_oid. Now we call GetDatabasePath() to get the
correct database path.
Authored by Paul Guo
---
src/backend/access/rmgrdesc/dbasedesc.c | 16 +++++++++++-----
1 file changed, 11 insertions(+), 5 deletions(-)
diff --git a/src/backend/access/rmgrdesc/dbasedesc.c b/src/backend/access/rmgrdesc/dbasedesc.c
index 26609845aa..873737161c 100644
--- a/src/backend/access/rmgrdesc/dbasedesc.c
+++ b/src/backend/access/rmgrdesc/dbasedesc.c
@@ -23,14 +23,17 @@ dbase_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ char *dbpath1, *dbpath2;
if (info == XLOG_DBASE_CREATE)
{
xl_dbase_create_rec *xlrec = (xl_dbase_create_rec *) rec;
- appendStringInfo(buf, "copy dir %u/%u to %u/%u",
- xlrec->src_tablespace_id, xlrec->src_db_id,
- xlrec->tablespace_id, xlrec->db_id);
+ dbpath1 = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
+ dbpath2 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
+ appendStringInfo(buf, "copy dir %s to %s", dbpath1, dbpath2);
+ pfree(dbpath2);
+ pfree(dbpath1);
}
else if (info == XLOG_DBASE_DROP)
{
@@ -39,8 +42,11 @@ dbase_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, "dir");
for (i = 0; i < xlrec->ntablespaces; i++)
- appendStringInfo(buf, " %u/%u",
- xlrec->tablespace_ids[i], xlrec->db_id);
+ {
+ dbpath1 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_ids[i]);
+ appendStringInfo(buf, " %s", dbpath1);
+ pfree(dbpath1);
+ }
}
}
--
2.14.3
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-07-08 12:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-01-05 01:07 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-01-27 08:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-03-27 14:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2021-03-30 07:12 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
@ 2021-07-09 19:37 ` Ibrar Ahmed <[email protected]>
2021-08-05 10:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Ibrar Ahmed @ 2021-07-09 19:37 UTC (permalink / raw)
To: Paul Guo <[email protected]>; Kyotaro Horiguchi <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Fujii Masao <[email protected]>; Anastasia Lubennikova <[email protected]>; Asim Praveen (Pivotal) <[email protected]>; Lei Wang (Pivotal) <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Tue, Mar 30, 2021 at 12:12 PM Paul Guo <[email protected]> wrote:
> On 2021/3/27, 10:23 PM, "Alvaro Herrera" <[email protected]> wrote:
>
> > Hmm, can you post a rebased set, where the points under discussion
> > are marked in XXX comments explaining what the issue is? This thread
> is
> > long and old ago that it's pretty hard to navigate the whole thing in
> > order to find out exactly what is being questioned.
>
> OK. Attached are the rebased version that includes the change I discussed
> in my previous reply. Also added POD documentation change for
> RecursiveCopy,
> and modified the patch to use the backup_options introduced in
> 081876d75ea15c3bd2ee5ba64a794fd8ea46d794 for tablespace mapping.
>
> > I think 0004 can be pushed without further ado, since it's a clear and
> > simple fix. 0001 needs a comment about the new parameter in
> > RecursiveCopy's POD documentation.
>
> Yeah, 0004 is no any risky. One concern seemed to be the compatibility of
> some
> WAL dump/analysis tools(?). I have no idea about this. But if we do not
> backport
> 0004 we do not seem to need to worry about this.
>
> > As I understand, this is a backpatchable bug-fix.
>
> Yes.
>
> Thanks.
>
> Patch does not apply successfully,
http://cfbot.cputube.org/patch_33_2161.log
Can you please rebase the patch.
--
Ibrar Ahmed
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-07-08 12:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-01-05 01:07 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-01-27 08:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-03-27 14:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2021-03-30 07:12 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-07-09 19:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Ibrar Ahmed <[email protected]>
@ 2021-08-05 10:20 ` Paul Guo <[email protected]>
2021-08-10 20:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Paul Guo @ 2021-08-05 10:20 UTC (permalink / raw)
To: Ibrar Ahmed <[email protected]>; Kyotaro Horiguchi <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Fujii Masao <[email protected]>; Anastasia Lubennikova <[email protected]>; Asim Praveen <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
Rebased.
Attachments:
[application/octet-stream] v12-0002-Tests-to-replay-create-database-operation-on-sta.patch (9.8K, ../../SA1PR05MB85821DEC48A6841277BDBF57C3F29@SA1PR05MB8582.namprd05.prod.outlook.com/3-v12-0002-Tests-to-replay-create-database-operation-on-sta.patch)
download | inline diff:
From 6096936d13526f121db4d5f684053cdfb4881ebe Mon Sep 17 00:00:00 2001
From: Asim R P <[email protected]>
Date: Fri, 20 Sep 2019 17:34:19 +0530
Subject: [PATCH v12 2/4] Tests to replay create database operation on standby
The tests demonstrate that standby fails to replay a create database
WAL record during crash recovery, if one or more of underlying
directories are missing from the file system. This can happen if a
drop tablespace or drop database WAL record has been replayed in
archive recovery, before a crash. And then the create database record
happens to be replayed again during crash recovery. The failures
indicate bugs that need to be fixed.
The first test, TEST 4, performs several DDL operations resulting in a
database directory being removed, along with a few create database
operations. It expects crash recovery to succeed because for each
missing directory encountered during create database replay, a matching
drop tablespace or drop database WAL record is found later.
Second test, TEST 5, validates that a standby rightfully aborts replay
during archive recovery, if a missing directory is encountered when
replaying create database WAL record.
These tests have been proposed and implemented in various ways by
Alexandra Wang, Anastasia Lubennikova, Kyotaro Horiguchi, Paul Guo and me.
---
src/test/perl/PostgresNode.pm | 21 +++-
src/test/recovery/t/011_crash_recovery.pl | 162 +++++++++++++++++++++++++++++-
2 files changed, 177 insertions(+), 6 deletions(-)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index b71e98f254..86e5b2f7be 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -2183,18 +2183,29 @@ Returns 1 if successful, 0 if timed out.
sub poll_query_until
{
- my ($self, $dbname, $query, $expected) = @_;
+ my ($self, $dbname, $query, $params) = @_;
+ my $expected;
+
+ # Be backwards-compatible
+ if (defined $params and ref $params eq '')
+ {
+ $params = {
+ expected => $params,
+ timeout => 180
+ };
+ }
local %ENV = $self->_get_env();
- $expected = 't' unless defined($expected); # default value
+ $params->{expected} = 't' unless defined($params->{expected});
+ $params->{timeout} = 180 unless defined($params->{timeout});
my $cmd = [
$self->installed_command('psql'), '-XAt',
'-d', $self->connstr($dbname)
];
my ($stdout, $stderr);
- my $max_attempts = 180 * 10;
+ my $max_attempts = $params->{timeout} * 10;
my $attempts = 0;
while ($attempts < $max_attempts)
@@ -2207,7 +2218,7 @@ sub poll_query_until
$stderr =~ s/\r\n/\n/g if $Config{osname} eq 'msys';
chomp($stderr);
- if ($stdout eq $expected && $stderr eq '')
+ if ($stdout eq $params->{expected} && $stderr eq '')
{
return 1;
}
@@ -2223,7 +2234,7 @@ sub poll_query_until
diag qq(poll_query_until timed out executing this query:
$query
expecting this output:
-$expected
+$params->{expected}
last actual query output:
$stdout
with stderr:
diff --git a/src/test/recovery/t/011_crash_recovery.pl b/src/test/recovery/t/011_crash_recovery.pl
index 72fc603e6d..fec330e808 100644
--- a/src/test/recovery/t/011_crash_recovery.pl
+++ b/src/test/recovery/t/011_crash_recovery.pl
@@ -9,9 +9,10 @@ use warnings;
use PostgresNode;
use TestLib;
use Test::More;
+use File::Path qw(rmtree);
use Config;
-plan tests => 3;
+plan tests => 5;
my $node = PostgresNode->new('primary');
$node->init(allows_streaming => 1);
@@ -62,3 +63,162 @@ is($node->safe_psql('postgres', qq[SELECT pg_xact_status('$xid');]),
$stdin .= "\\q\n";
$tx->finish; # wait for psql to quit gracefully
+
+# TEST 4
+#
+# Ensure that a missing tablespace directory during crash recovery on
+# a standby is handled correctly. The standby should finish crash
+# recovery successfully because a matching drop database record is
+# found in the WAL. The following scnearios are covered:
+#
+# 1. Create a database against a user-defined tablespace then drop the
+# database.
+#
+# 2. Create a database against a user-defined tablespace then drop the
+# database and the tablespace.
+#
+# 3. Move a database from source tablespace to target tablespace then
+# drop the source tablespace.
+#
+# 4. Create a database from another database as template then drop the
+# template database.
+#
+#
+
+my $node_master = PostgresNode->new('master2');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Create tablespace
+my $dropme_ts_master1 = TestLib::tempdir;
+$dropme_ts_master1 = TestLib::perl2host($dropme_ts_master1);
+my $dropme_ts_master2 = TestLib::tempdir;
+$dropme_ts_master2 = TestLib::perl2host($dropme_ts_master2);
+my $source_ts_master = TestLib::tempdir;
+$source_ts_master = TestLib::perl2host($source_ts_master);
+my $target_ts_master = TestLib::tempdir;
+$target_ts_master = TestLib::perl2host($target_ts_master);
+
+$node_master->safe_psql('postgres',
+ qq[CREATE TABLESPACE dropme_ts1 location '$dropme_ts_master1';
+ CREATE TABLESPACE dropme_ts2 location '$dropme_ts_master2';
+ CREATE TABLESPACE source_ts location '$source_ts_master';
+ CREATE TABLESPACE target_ts location '$target_ts_master';
+ CREATE DATABASE template_db IS_TEMPLATE = true;]);
+
+my $dropme_ts_standby1 = TestLib::tempdir;
+$dropme_ts_standby1 = TestLib::perl2host($dropme_ts_standby1);
+my $dropme_ts_standby2 = TestLib::tempdir;
+$dropme_ts_standby2 = TestLib::perl2host($dropme_ts_standby2);
+my $source_ts_standby = TestLib::tempdir;
+$source_ts_standby = TestLib::perl2host($source_ts_standby);
+my $target_ts_standby = TestLib::tempdir;
+$target_ts_standby = TestLib::perl2host($target_ts_standby);
+
+# Take backup
+my $backup_name = 'my_backup';
+my $ts_mapping = [ "--tablespace-mapping=$dropme_ts_master1=$dropme_ts_standby1",
+ "--tablespace-mapping=$dropme_ts_master2=$dropme_ts_standby2",
+ "--tablespace-mapping=$source_ts_master=$source_ts_standby",
+ "--tablespace-mapping=$target_ts_master=$target_ts_standby" ];
+$node_master->backup($backup_name, backup_options => $ts_mapping);
+
+my $node_standby = PostgresNode->new('standby2');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->start;
+
+# Make sure connection is made
+$node_master->poll_query_until(
+ 'postgres', 'SELECT count(*) = 1 FROM pg_stat_replication');
+
+# Make sure to perform restartpoint after tablespace creation
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->safe_psql('postgres', 'CHECKPOINT');
+
+# Do immediate shutdown just after a sequence of CREAT DATABASE / DROP
+# DATABASE / DROP TABLESPACE. This causes CREATE DATABASE WAL records
+# to be applied to already-removed directories.
+$node_master->safe_psql('postgres',
+ q[CREATE DATABASE dropme_db1 WITH TABLESPACE dropme_ts1;
+ CREATE DATABASE dropme_db2 WITH TABLESPACE dropme_ts2;
+ CREATE DATABASE moveme_db TABLESPACE source_ts;
+ ALTER DATABASE moveme_db SET TABLESPACE target_ts;
+ CREATE DATABASE newdb TEMPLATE template_db;
+ ALTER DATABASE template_db IS_TEMPLATE = false;
+ DROP DATABASE dropme_db1;
+ DROP DATABASE dropme_db2; DROP TABLESPACE dropme_ts2;
+ DROP TABLESPACE source_ts;
+ DROP DATABASE template_db;]);
+$node_master->wait_for_catchup($node_standby, 'replay',
+ $node_master->lsn('replay'));
+$node_standby->stop('immediate');
+
+# Should restart ignoring directory creation error.
+is($node_standby->start(fail_ok => 1), 1);
+
+# TEST 5
+#
+# Ensure that a missing tablespace directory during create database
+# replay immediately causes panic if the standby has already reached
+# consistent state (archive recovery is in progress).
+
+$node_master = PostgresNode->new('master3');
+$node_master->init(allows_streaming => 1);
+$node_master->start;
+
+# Create tablespace
+my $ts_master = TestLib::tempdir;
+$ts_master = TestLib::perl2host($ts_master);
+$node_master->safe_psql('postgres', "CREATE TABLESPACE ts1 LOCATION '$ts_master'");
+$node_master->safe_psql('postgres', "CREATE DATABASE db1 TABLESPACE ts1");
+
+my $ts_standby = TestLib::tempdir("standby");
+$ts_standby = TestLib::perl2host($ts_standby);
+
+# Take backup
+$backup_name = 'my_backup';
+$node_master->backup($backup_name,
+ backup_options =>
+ [ "--tablespace-mapping=$ts_master=$ts_standby" ]);
+$node_standby = PostgresNode->new('standby3');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->start;
+
+# Make sure standby reached consistency and starts accepting connections
+$node_standby->poll_query_until('postgres', 'SELECT 1', '1');
+
+# Remove standby tablespace directory so it will be missing when
+# replay resumes.
+#
+# The tablespace mapping is lost when the standby node is initialized
+# from basebackup because RecursiveCopy::copypath creates a new temp
+# directory for each tablspace symlink found in backup. We must
+# obtain the correct tablespace directory by querying standby.
+$ts_standby = $node_standby->safe_psql(
+ 'postgres',
+ "select pg_tablespace_location(oid) from pg_tablespace where spcname = 'ts1'");
+rmtree($ts_standby);
+
+# Create a database in the tablespace and a table in default tablespace
+$node_master->safe_psql('postgres',
+ q[CREATE TABLE should_not_replay_insertion(a int);
+ CREATE DATABASE db2 WITH TABLESPACE ts1;
+ INSERT INTO should_not_replay_insertion VALUES (1);]);
+
+# Standby should fail and should not silently skip replaying the wal
+if ($node_master->poll_query_until(
+ 'postgres',
+ 'SELECT count(*) = 0 FROM pg_stat_replication',
+ 't') == 1)
+{
+ pass('standby failed as expected');
+ # We know that the standby has failed. Setting its pid to
+ # undefined avoids error when PostgreNode module tries to stop the
+ # standby node as part of tear_down sequence.
+ $node_standby->{_pid} = undef;
+}
+else
+{
+ fail('standby did not fail within 5 seconds');
+}
--
2.14.3
[application/octet-stream] v12-0001-Support-node-initialization-from-backup-with-tab.patch (4.9K, ../../SA1PR05MB85821DEC48A6841277BDBF57C3F29@SA1PR05MB8582.namprd05.prod.outlook.com/4-v12-0001-Support-node-initialization-from-backup-with-tab.patch)
download | inline diff:
From 5cd04cebf8361cfd7941063cab055a2608ff546f Mon Sep 17 00:00:00 2001
From: Asim R P <[email protected]>
Date: Fri, 20 Sep 2019 17:31:25 +0530
Subject: [PATCH v12 1/4] Support node initialization from backup with
tablespaces
User defined tablespaces appear as symlinks in in the backup. This
commit tweaks recursive copy subroutine to allow for symlinks specific
to tablespaces.
Authored by Kyotaro Horiguchi
---
src/test/perl/PostgresNode.pm | 29 ++++++++++++++++++++++++++-
src/test/perl/RecursiveCopy.pm | 45 +++++++++++++++++++++++++++++++++++-------
2 files changed, 66 insertions(+), 8 deletions(-)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 8158ea5b2f..b71e98f254 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -634,6 +634,32 @@ sub backup_fs_cold
return;
}
+sub _srcsymlink
+{
+ my ($srcpath, $destpath) = @_;
+
+ croak "Cannot operate on symlink \"$srcpath\""
+ if ($srcpath !~ qr{/(pg_tblspc/[0-9]+)$});
+
+ # We have mapped tablespaces. Copy them individually
+ my $tmpdir = TestLib::tempdir;
+ my $dstrealdir = TestLib::perl2host($tmpdir);
+ my $srcrealdir = readlink($srcpath);
+
+ opendir(my $dh, $srcrealdir);
+ while (my $entry = (readdir $dh))
+ {
+ next if ($entry eq '.' or $entry eq '..');
+ my $spath = "$srcrealdir/$entry";
+ my $dpath = "$dstrealdir/$entry";
+ RecursiveCopy::copypath($spath, $dpath);
+ }
+ closedir $dh;
+
+ symlink $dstrealdir, $destpath;
+
+ return 1;
+}
# Common sub of backup_fs_hot and backup_fs_cold
sub _backup_fs
@@ -743,7 +769,8 @@ sub init_from_backup
else
{
rmdir($data_path);
- RecursiveCopy::copypath($backup_path, $data_path);
+ RecursiveCopy::copypath($backup_path, $data_path,
+ srcsymlinkfn => \&_srcsymlink);
}
chmod(0700, $data_path);
diff --git a/src/test/perl/RecursiveCopy.pm b/src/test/perl/RecursiveCopy.pm
index 8a9cc722b5..cbd7a874d7 100644
--- a/src/test/perl/RecursiveCopy.pm
+++ b/src/test/perl/RecursiveCopy.pm
@@ -49,6 +49,11 @@ This subroutine will be called for each entry in the source directory with its
relative path as only parameter; if the subroutine returns true the entry is
copied, otherwise the file is skipped.
+If the B<srcsymlinkfn> parameter is given, it must be a subroutine reference.
+This subroutine will be called when the source directory is a symlink. It
+determines the mechanism that copies files from the source directory to the
+destination directory.
+
On failure the target directory may be in some incomplete state; no cleanup is
attempted.
@@ -68,6 +73,7 @@ sub copypath
{
my ($base_src_dir, $base_dest_dir, %params) = @_;
my $filterfn;
+ my $srcsymlinkfn;
if (defined $params{filterfn})
{
@@ -82,31 +88,55 @@ sub copypath
$filterfn = sub { return 1; };
}
+ if (defined $params{srcsymlinkfn})
+ {
+ croak "if specified, srcsymlinkfn must be a subroutine reference"
+ unless defined(ref $params{srcsymlinkfn})
+ and (ref $params{srcsymlinkfn} eq 'CODE');
+
+ $srcsymlinkfn = $params{srcsymlinkfn};
+ }
+ else
+ {
+ $srcsymlinkfn = undef;
+ }
+
# Complain if original path is bogus, because _copypath_recurse won't.
croak "\"$base_src_dir\" does not exist" if !-e $base_src_dir;
# Start recursive copy from current directory
- return _copypath_recurse($base_src_dir, $base_dest_dir, "", $filterfn);
+ return _copypath_recurse($base_src_dir, $base_dest_dir, "", $filterfn, $srcsymlinkfn);
}
# Recursive private guts of copypath
sub _copypath_recurse
{
- my ($base_src_dir, $base_dest_dir, $curr_path, $filterfn) = @_;
+ my ($base_src_dir, $base_dest_dir, $curr_path, $filterfn,
+ $srcsymlinkfn) = @_;
my $srcpath = "$base_src_dir/$curr_path";
my $destpath = "$base_dest_dir/$curr_path";
# invoke the filter and skip all further operation if it returns false
return 1 unless &$filterfn($curr_path);
- # Check for symlink -- needed only on source dir
- # (note: this will fall through quietly if file is already gone)
- croak "Cannot operate on symlink \"$srcpath\"" if -l $srcpath;
-
# Abort if destination path already exists. Should we allow directories
# to exist already?
croak "Destination path \"$destpath\" already exists" if -e $destpath;
+ # Check for symlink -- needed only on source dir
+ # If caller provided us with a callback, call it; otherwise we're out.
+ if (-l $srcpath)
+ {
+ if (defined $srcsymlinkfn)
+ {
+ return &$srcsymlinkfn($srcpath, $destpath);
+ }
+ else
+ {
+ croak "Cannot operate on symlink \"$srcpath\"";
+ }
+ }
+
# If this source path is a file, simply copy it to destination with the
# same name and we're done.
if (-f $srcpath)
@@ -139,7 +169,8 @@ sub _copypath_recurse
{
next if ($entry eq '.' or $entry eq '..');
_copypath_recurse($base_src_dir, $base_dest_dir,
- $curr_path eq '' ? $entry : "$curr_path/$entry", $filterfn)
+ $curr_path eq '' ? $entry : "$curr_path/$entry", $filterfn,
+ $srcsymlinkfn)
or die "copypath $srcpath/$entry -> $destpath/$entry failed";
}
--
2.14.3
[application/octet-stream] v12-0003-Fix-replay-of-create-database-records-on-standby.patch (10.4K, ../../SA1PR05MB85821DEC48A6841277BDBF57C3F29@SA1PR05MB8582.namprd05.prod.outlook.com/5-v12-0003-Fix-replay-of-create-database-records-on-standby.patch)
download | inline diff:
From 42eda96adda2815ac91f4323964d8cbf7863b23c Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <[email protected]>
Date: Thu, 9 Jan 2020 17:54:40 -0300
Subject: [PATCH v12 3/4] Fix replay of create database records on standby
Crash recovery on standby may encounter missing directories when
replaying create database WAL records. Prior to this patch, the
standby would fail to recover in such a case. However, the
directories could be legitimately missing. Consider a sequence of WAL
records as follows:
CREATE DATABASE
DROP DATABASE
DROP TABLESPACE
If, after replaying the last WAL record and removing the tablespace
directory, the standby crashes and has to replay the create database
record again, the crash recovery must be able to move on.
This patch adds mechanism similar to invalid page hash table, to track
missing directories during crash recovery. If all the missing
directory references are matched with corresponding drop records at
the end of crash recovery, the standby can safely enter archive
recovery.
Bug identified by Paul Guo.
Authored by Paul Guo, Kyotaro Horiguchi and Asim R P.
---
src/backend/access/transam/xlog.c | 6 ++
src/backend/access/transam/xlogutils.c | 145 +++++++++++++++++++++++++++++++++
src/backend/commands/dbcommands.c | 55 +++++++++++++
src/backend/commands/tablespace.c | 5 ++
src/include/access/xlogutils.h | 4 +
5 files changed, 215 insertions(+)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 8b39a2fdaa..d48716b506 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8172,6 +8172,12 @@ CheckRecoveryConsistency(void)
*/
XLogCheckInvalidPages();
+ /*
+ * Check if the XLOG sequence contained any unresolved references to
+ * missing directories.
+ */
+ XLogCheckMissingDirs();
+
reachedConsistency = true;
ereport(LOG,
(errmsg("consistent recovery state reached at %X/%X",
diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c
index b1702bc6be..13d0f1c101 100644
--- a/src/backend/access/transam/xlogutils.c
+++ b/src/backend/access/transam/xlogutils.c
@@ -79,6 +79,151 @@ typedef struct xl_invalid_page
static HTAB *invalid_page_tab = NULL;
+/*
+ * If a create database WAL record is being replayed more than once during
+ * crash recovery on a standby, it is possible that either the tablespace
+ * directory or the template database directory is missing. This happens when
+ * the directories are removed by replay of subsequent drop records. Note
+ * that this problem happens only on standby and not on master. On master, a
+ * checkpoint is created at the end of create database operation. On standby,
+ * however, such a strategy (creating restart points during replay) is not
+ * viable because it will slow down WAL replay.
+ *
+ * The alternative is to track references to each missing directory
+ * encountered when performing crash recovery in the following hash table.
+ * Similar to invalid page table above, the expectation is that each missing
+ * directory entry should be matched with a drop database or drop tablespace
+ * WAL record by the end of crash recovery.
+ */
+typedef struct xl_missing_dir_key
+{
+ Oid spcNode;
+ Oid dbNode;
+} xl_missing_dir_key;
+
+typedef struct xl_missing_dir
+{
+ xl_missing_dir_key key;
+ char path[MAXPGPATH];
+} xl_missing_dir;
+
+static HTAB *missing_dir_tab = NULL;
+
+void
+XLogReportMissingDir(Oid spcNode, Oid dbNode, char *path)
+{
+ xl_missing_dir_key key;
+ bool found;
+ xl_missing_dir *entry;
+
+ /*
+ * Database OID may be invalid but tablespace OID must be valid. If
+ * dbNode is InvalidOid, we are logging a missing tablespace directory,
+ * otherwise we are logging a missing database directory.
+ */
+ Assert(OidIsValid(spcNode));
+
+ if (missing_dir_tab == NULL)
+ {
+ /* create hash table when first needed */
+ HASHCTL ctl;
+
+ memset(&ctl, 0, sizeof(ctl));
+ ctl.keysize = sizeof(xl_missing_dir_key);
+ ctl.entrysize = sizeof(xl_missing_dir);
+
+ missing_dir_tab = hash_create("XLOG missing directory table",
+ 100,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS);
+ }
+
+ key.spcNode = spcNode;
+ key.dbNode = dbNode;
+
+ entry = hash_search(missing_dir_tab, &key, HASH_ENTER, &found);
+
+ if (found)
+ {
+ if (dbNode == InvalidOid)
+ elog(DEBUG2, "missing directory %s (tablespace %d) already exists: %s",
+ path, spcNode, entry->path);
+ else
+ elog(DEBUG2, "missing directory %s (tablespace %d database %d) already exists: %s",
+ path, spcNode, dbNode, entry->path);
+ }
+ else
+ {
+ strlcpy(entry->path, path, sizeof(entry->path));
+ if (dbNode == InvalidOid)
+ elog(DEBUG2, "logged missing dir %s (tablespace %d)",
+ path, spcNode);
+ else
+ elog(DEBUG2, "logged missing dir %s (tablespace %d database %d)",
+ path, spcNode, dbNode);
+ }
+}
+
+void
+XLogForgetMissingDir(Oid spcNode, Oid dbNode)
+{
+ xl_missing_dir_key key;
+
+ key.spcNode = spcNode;
+ key.dbNode = dbNode;
+
+ /* Database OID may be invalid but tablespace OID must be valid. */
+ Assert(OidIsValid(spcNode));
+
+ if (missing_dir_tab == NULL)
+ return;
+
+ if (hash_search(missing_dir_tab, &key, HASH_REMOVE, NULL) != NULL)
+ {
+ if (dbNode == InvalidOid)
+ {
+ elog(DEBUG2, "forgot missing dir (tablespace %d)", spcNode);
+ }
+ else
+ {
+ char *path = GetDatabasePath(dbNode, spcNode);
+
+ elog(DEBUG2, "forgot missing dir %s (tablespace %d database %d)",
+ path, spcNode, dbNode);
+ pfree(path);
+ }
+ }
+}
+
+/*
+ * This is called at the end of crash recovery, before entering archive
+ * recovery on a standby. PANIC if the hash table is not empty.
+ */
+void
+XLogCheckMissingDirs(void)
+{
+ HASH_SEQ_STATUS status;
+ xl_missing_dir *hentry;
+ bool foundone = false;
+
+ if (missing_dir_tab == NULL)
+ return; /* nothing to do */
+
+ hash_seq_init(&status, missing_dir_tab);
+
+ while ((hentry = (xl_missing_dir *) hash_seq_search(&status)) != NULL)
+ {
+ elog(WARNING, "missing directory \"%s\" tablespace %d database %d",
+ hentry->path, hentry->key.spcNode, hentry->key.dbNode);
+ foundone = true;
+ }
+
+ if (foundone)
+ elog(PANIC, "WAL contains references to missing directories");
+
+ hash_destroy(missing_dir_tab);
+ missing_dir_tab = NULL;
+}
/* Report a reference to an invalid page */
static void
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index 029fab48df..0f483edb71 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -2143,7 +2143,9 @@ dbase_redo(XLogReaderState *record)
xl_dbase_create_rec *xlrec = (xl_dbase_create_rec *) XLogRecGetData(record);
char *src_path;
char *dst_path;
+ char *parent_path;
struct stat st;
+ bool skip = false;
src_path = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
dst_path = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
@@ -2161,6 +2163,55 @@ dbase_redo(XLogReaderState *record)
(errmsg("some useless files may be left behind in old database directory \"%s\"",
dst_path)));
}
+ else if (!reachedConsistency)
+ {
+ /*
+ * It is possible that drop tablespace record appearing later in
+ * the WAL as already been replayed. That means we are replaying
+ * the create database record second time, as part of crash
+ * recovery. In that case, the tablespace directory has already
+ * been removed and the create database operation cannot be
+ * replayed. We should skip the replay but remember the missing
+ * tablespace directory, to be matched with a drop tablespace
+ * record later.
+ */
+ parent_path = pstrdup(dst_path);
+ get_parent_directory(parent_path);
+ if (!(stat(parent_path, &st) == 0 && S_ISDIR(st.st_mode)))
+ {
+ XLogReportMissingDir(xlrec->tablespace_id, InvalidOid, parent_path);
+ skip = true;
+ ereport(WARNING,
+ (errmsg("skipping create database WAL record"),
+ errdetail("Target tablespace \"%s\" not found. We "
+ "expect to encounter a WAL record that "
+ "removes this directory before reaching "
+ "consistent state.", parent_path)));
+ }
+ pfree(parent_path);
+ }
+
+ /*
+ * Source directory may be missing. E.g. the template database used
+ * for creating this database may have been dropped, due to reasons
+ * noted above. Moving a database from one tablespace may also be a
+ * partner in the crime.
+ */
+ if (!(stat(src_path, &st) == 0 && S_ISDIR(st.st_mode)) &&
+ !reachedConsistency)
+ {
+ XLogReportMissingDir(xlrec->src_tablespace_id, xlrec->src_db_id, src_path);
+ skip = true;
+ ereport(WARNING,
+ (errmsg("skipping create database WAL record"),
+ errdetail("Source database \"%s\" not found. We expect "
+ "to encounter a WAL record that removes this "
+ "directory before reaching consistent state.",
+ src_path)));
+ }
+
+ if (skip)
+ return;
/*
* Force dirty buffers out to disk, to ensure source database is
@@ -2218,6 +2269,10 @@ dbase_redo(XLogReaderState *record)
ereport(WARNING,
(errmsg("some useless files may be left behind in old database directory \"%s\"",
dst_path)));
+
+ if (!reachedConsistency)
+ XLogForgetMissingDir(xlrec->tablespace_ids[i], xlrec->db_id);
+
pfree(dst_path);
}
diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c
index a54239a8b3..2a4d14e29f 100644
--- a/src/backend/commands/tablespace.c
+++ b/src/backend/commands/tablespace.c
@@ -1531,6 +1531,11 @@ tblspc_redo(XLogReaderState *record)
{
xl_tblspc_drop_rec *xlrec = (xl_tblspc_drop_rec *) XLogRecGetData(record);
+ if (!reachedConsistency)
+ XLogForgetMissingDir(xlrec->ts_id, InvalidOid);
+
+ XLogFlush(record->EndRecPtr);
+
/*
* If we issued a WAL record for a drop tablespace it implies that
* there were no files in it at all when the DROP was done. That means
diff --git a/src/include/access/xlogutils.h b/src/include/access/xlogutils.h
index a5cb3d322c..2ee3bd378d 100644
--- a/src/include/access/xlogutils.h
+++ b/src/include/access/xlogutils.h
@@ -65,6 +65,10 @@ extern void XLogDropDatabase(Oid dbid);
extern void XLogTruncateRelation(RelFileNode rnode, ForkNumber forkNum,
BlockNumber nblocks);
+extern void XLogReportMissingDir(Oid spcNode, Oid dbNode, char *path);
+extern void XLogForgetMissingDir(Oid spcNode, Oid dbNode);
+extern void XLogCheckMissingDirs(void);
+
/* Result codes for XLogReadBufferForRedo[Extended] */
typedef enum
{
--
2.14.3
[application/octet-stream] v12-0004-Fix-database-create-drop-wal-description.patch (1.9K, ../../SA1PR05MB85821DEC48A6841277BDBF57C3F29@SA1PR05MB8582.namprd05.prod.outlook.com/6-v12-0004-Fix-database-create-drop-wal-description.patch)
download | inline diff:
From 7c03826e7fb9872ea293f95d8270b3e64433949a Mon Sep 17 00:00:00 2001
From: Paul Guo <[email protected]>
Date: Mon, 6 Jul 2020 21:20:15 +0800
Subject: [PATCH v12 4/4] Fix database create/drop wal description.
Previously the description messages are wrong since the database path is not
simply tablespce_oid/database_oid. Now we call GetDatabasePath() to get the
correct database path.
Authored by Paul Guo
---
src/backend/access/rmgrdesc/dbasedesc.c | 16 +++++++++++-----
1 file changed, 11 insertions(+), 5 deletions(-)
diff --git a/src/backend/access/rmgrdesc/dbasedesc.c b/src/backend/access/rmgrdesc/dbasedesc.c
index 26609845aa..873737161c 100644
--- a/src/backend/access/rmgrdesc/dbasedesc.c
+++ b/src/backend/access/rmgrdesc/dbasedesc.c
@@ -23,14 +23,17 @@ dbase_desc(StringInfo buf, XLogReaderState *record)
{
char *rec = XLogRecGetData(record);
uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
+ char *dbpath1, *dbpath2;
if (info == XLOG_DBASE_CREATE)
{
xl_dbase_create_rec *xlrec = (xl_dbase_create_rec *) rec;
- appendStringInfo(buf, "copy dir %u/%u to %u/%u",
- xlrec->src_tablespace_id, xlrec->src_db_id,
- xlrec->tablespace_id, xlrec->db_id);
+ dbpath1 = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id);
+ dbpath2 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id);
+ appendStringInfo(buf, "copy dir %s to %s", dbpath1, dbpath2);
+ pfree(dbpath2);
+ pfree(dbpath1);
}
else if (info == XLOG_DBASE_DROP)
{
@@ -39,8 +42,11 @@ dbase_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, "dir");
for (i = 0; i < xlrec->ntablespaces; i++)
- appendStringInfo(buf, " %u/%u",
- xlrec->tablespace_ids[i], xlrec->db_id);
+ {
+ dbpath1 = GetDatabasePath(xlrec->db_id, xlrec->tablespace_ids[i]);
+ appendStringInfo(buf, " %s", dbpath1);
+ pfree(dbpath1);
+ }
}
}
--
2.14.3
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-07-08 12:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-01-05 01:07 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-01-27 08:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-03-27 14:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2021-03-30 07:12 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-07-09 19:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Ibrar Ahmed <[email protected]>
2021-08-05 10:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
@ 2021-08-10 20:56 ` Robert Haas <[email protected]>
2021-08-11 07:58 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-09-24 18:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
0 siblings, 2 replies; 109+ messages in thread
From: Robert Haas @ 2021-08-10 20:56 UTC (permalink / raw)
To: Paul Guo <[email protected]>; +Cc: Ibrar Ahmed <[email protected]>; Kyotaro Horiguchi <[email protected]>; Alvaro Herrera <[email protected]>; Fujii Masao <[email protected]>; Anastasia Lubennikova <[email protected]>; Asim Praveen <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Thu, Aug 5, 2021 at 6:20 AM Paul Guo <[email protected]> wrote:
> Rebased.
The commit message for 0001 is not clear enough for me to understand
what problem it's supposed to be fixing. The code comments aren't
really either. They make it sound like there's some problem with
copying symlinks but mostly they just talk about callbacks, which
doesn't really help me understand what problem we'd have if we just
didn't commit this (or reverted it later).
I am not really convinced by Álvaro's claim that 0004 is a "fix"; I
think I'd call it an improvement. But either way I agree that could
just be committed.
I haven't analyzed 0002 and 0003 yet.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-07-08 12:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-01-05 01:07 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-01-27 08:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-03-27 14:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2021-03-30 07:12 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-07-09 19:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Ibrar Ahmed <[email protected]>
2021-08-05 10:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-08-10 20:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
@ 2021-08-11 07:58 ` Paul Guo <[email protected]>
2021-08-11 12:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
1 sibling, 1 reply; 109+ messages in thread
From: Paul Guo @ 2021-08-11 07:58 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Paul Guo <[email protected]>; Ibrar Ahmed <[email protected]>; Kyotaro Horiguchi <[email protected]>; Alvaro Herrera <[email protected]>; Fujii Masao <[email protected]>; Anastasia Lubennikova <[email protected]>; Asim Praveen <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Wed, Aug 11, 2021 at 4:56 AM Robert Haas <[email protected]> wrote:
>
> On Thu, Aug 5, 2021 at 6:20 AM Paul Guo <[email protected]> wrote:
> > Rebased.
>
> The commit message for 0001 is not clear enough for me to understand
> what problem it's supposed to be fixing. The code comments aren't
> really either. They make it sound like there's some problem with
> copying symlinks but mostly they just talk about callbacks, which
> doesn't really help me understand what problem we'd have if we just
> didn't commit this (or reverted it later).
Thanks for reviewing. Let me explain a bit. The patch series includes
four patches.
0001 and 0002 are test changes for the fix (0003).
- 0001 is the test framework change that's needed by 0002.
- 0002 is the test for the code fix (0003).
0003 is the code change and the commit message explains the issue in detail.
0004 as said is a small enhancement which is a bit independent of the
previous patches.
Basically the issue is that without the fix crash recovery might fail
relevant to tablespace.
Here is the log after I run the tests in 0001/0002 without the 0003 fix.
2021-08-04 10:00:42.231 CST [875] FATAL: could not create directory
"pg_tblspc/16385/PG_15_202107261/16390": No such file or directory
2021-08-04 10:00:42.231 CST [875] CONTEXT: WAL redo at 0/3001320 for
Database/CREATE: copy dir base/1 to
pg_tblspc/16385/PG_15_202107261/16390
>
> I am not really convinced by Álvaro's claim that 0004 is a "fix"; I
> think I'd call it an improvement. But either way I agree that could
> just be committed.
>
> I haven't analyzed 0002 and 0003 yet.
>
> --
> Robert Haas
> EDB: http://www.enterprisedb.com
>
>
--
Paul Guo (Vmware)
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-07-08 12:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-01-05 01:07 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-01-27 08:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-03-27 14:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2021-03-30 07:12 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-07-09 19:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Ibrar Ahmed <[email protected]>
2021-08-05 10:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-08-10 20:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
2021-08-11 07:58 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
@ 2021-08-11 12:59 ` Robert Haas <[email protected]>
0 siblings, 0 replies; 109+ messages in thread
From: Robert Haas @ 2021-08-11 12:59 UTC (permalink / raw)
To: Paul Guo <[email protected]>; +Cc: Paul Guo <[email protected]>; Ibrar Ahmed <[email protected]>; Kyotaro Horiguchi <[email protected]>; Alvaro Herrera <[email protected]>; Fujii Masao <[email protected]>; Anastasia Lubennikova <[email protected]>; Asim Praveen <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Wed, Aug 11, 2021 at 3:59 AM Paul Guo <[email protected]> wrote:
> Thanks for reviewing. Let me explain a bit. The patch series includes
> four patches.
>
> 0001 and 0002 are test changes for the fix (0003).
> - 0001 is the test framework change that's needed by 0002.
> - 0002 is the test for the code fix (0003).
> 0003 is the code change and the commit message explains the issue in detail.
> 0004 as said is a small enhancement which is a bit independent of the
> previous patches.
>
> Basically the issue is that without the fix crash recovery might fail
> relevant to tablespace.
> Here is the log after I run the tests in 0001/0002 without the 0003 fix.
I do understand all of this, but I (or whoever might commit this)
needs to also be able to understand specifically what each patch is
doing.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-07-08 12:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-01-05 01:07 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-01-27 08:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-03-27 14:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2021-03-30 07:12 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-07-09 19:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Ibrar Ahmed <[email protected]>
2021-08-05 10:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-08-10 20:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
@ 2021-09-24 18:14 ` Tom Lane <[email protected]>
2021-11-04 12:34 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Daniel Gustafsson <[email protected]>
1 sibling, 1 reply; 109+ messages in thread
From: Tom Lane @ 2021-09-24 18:14 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Paul Guo <[email protected]>; Ibrar Ahmed <[email protected]>; Kyotaro Horiguchi <[email protected]>; Alvaro Herrera <[email protected]>; Fujii Masao <[email protected]>; Anastasia Lubennikova <[email protected]>; Asim Praveen <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
Robert Haas <[email protected]> writes:
> The commit message for 0001 is not clear enough for me to understand
> what problem it's supposed to be fixing. The code comments aren't
> really either. They make it sound like there's some problem with
> copying symlinks but mostly they just talk about callbacks, which
> doesn't really help me understand what problem we'd have if we just
> didn't commit this (or reverted it later).
> I am not really convinced by Álvaro's claim that 0004 is a "fix"; I
> think I'd call it an improvement. But either way I agree that could
> just be committed.
> I haven't analyzed 0002 and 0003 yet.
I took a quick look through this:
* I don't like 0001 either, though it seems like the issue is mostly
documentation. sub _srcsymlink should have a comment explaining
what it's doing and why. The documentation of copypath's new parameter
seems like gobbledegook too --- I suppose it should read more like
"By default, copypath fails if a source item is a symlink. But if
B<srcsymlinkfn> is provided, that subroutine is called to process any
symlink."
* I'm allergic to 0002's completely undocumented changes to
poll_query_until, especially since I don't see anything in the
patch that actually uses them. Can't we just drop these diffs
in PostgresNode.pm? BTW, the last error message in the patch,
talking about a 5-second timeout, seems wrong. With or without
these changes, poll_query_until's default timeout is 180 sec.
The actual test case might be okay other than that nit and a
comment typo or two.
* 0003 might actually be okay. I've not read it line-by-line,
but it seems like it's implementing a sane solution and it's
adequately commented.
* I'm inclined to reject 0004 out of hand, because I don't
agree with what it's doing. The purpose of the rmgrdesc
functions is to show you what is in the WAL records, and
everywhere else we interpret that as "show the verbatim,
numeric field contents". heapdesc.c, for example, doesn't
attempt to look up the name of the table being operated on.
0004 isn't adhering to that style, and aside from being
inconsistent I'm afraid that it's adding failure modes
we don't want.
regards, tom lane
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-07-08 12:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-01-05 01:07 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-01-27 08:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-03-27 14:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2021-03-30 07:12 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-07-09 19:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Ibrar Ahmed <[email protected]>
2021-08-05 10:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-08-10 20:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
2021-09-24 18:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
@ 2021-11-04 12:34 ` Daniel Gustafsson <[email protected]>
2021-11-08 08:55 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Daniel Gustafsson @ 2021-11-04 12:34 UTC (permalink / raw)
To: Paul Guo <[email protected]>; +Cc: Tom Lane <[email protected]>; Robert Haas <[email protected]>; Ibrar Ahmed <[email protected]>; Kyotaro Horiguchi <[email protected]>; Alvaro Herrera <[email protected]>; Fujii Masao <[email protected]>; Anastasia Lubennikova <[email protected]>; Asim Praveen <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
> On 24 Sep 2021, at 20:14, Tom Lane <[email protected]> wrote:
>
> Robert Haas <[email protected]> writes:
>> The commit message for 0001 is not clear enough for me to understand
>> what problem it's supposed to be fixing. The code comments aren't
>> really either. They make it sound like there's some problem with
>> copying symlinks but mostly they just talk about callbacks, which
>> doesn't really help me understand what problem we'd have if we just
>> didn't commit this (or reverted it later).
>
>> I am not really convinced by Álvaro's claim that 0004 is a "fix"; I
>> think I'd call it an improvement. But either way I agree that could
>> just be committed.
>
>> I haven't analyzed 0002 and 0003 yet.
>
> I took a quick look through this:
>
> * I don't like 0001 either, though it seems like the issue is mostly
> documentation. sub _srcsymlink should have a comment explaining
> what it's doing and why. The documentation of copypath's new parameter
> seems like gobbledegook too --- I suppose it should read more like
> "By default, copypath fails if a source item is a symlink. But if
> B<srcsymlinkfn> is provided, that subroutine is called to process any
> symlink."
>
> * I'm allergic to 0002's completely undocumented changes to
> poll_query_until, especially since I don't see anything in the
> patch that actually uses them. Can't we just drop these diffs
> in PostgresNode.pm? BTW, the last error message in the patch,
> talking about a 5-second timeout, seems wrong. With or without
> these changes, poll_query_until's default timeout is 180 sec.
> The actual test case might be okay other than that nit and a
> comment typo or two.
>
> * 0003 might actually be okay. I've not read it line-by-line,
> but it seems like it's implementing a sane solution and it's
> adequately commented.
>
> * I'm inclined to reject 0004 out of hand, because I don't
> agree with what it's doing. The purpose of the rmgrdesc
> functions is to show you what is in the WAL records, and
> everywhere else we interpret that as "show the verbatim,
> numeric field contents". heapdesc.c, for example, doesn't
> attempt to look up the name of the table being operated on.
> 0004 isn't adhering to that style, and aside from being
> inconsistent I'm afraid that it's adding failure modes
> we don't want.
This patch again fails to apply (seemingly from the Perl namespace work on the
testcode), and needs a few updates as per the above review.
--
Daniel Gustafsson https://vmware.com/
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-07-08 12:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-01-05 01:07 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-01-27 08:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-03-27 14:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2021-03-30 07:12 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-07-09 19:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Ibrar Ahmed <[email protected]>
2021-08-05 10:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-08-10 20:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
2021-09-24 18:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2021-11-04 12:34 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Daniel Gustafsson <[email protected]>
@ 2021-11-08 08:55 ` Kyotaro Horiguchi <[email protected]>
2021-11-09 03:51 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Michael Paquier <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Kyotaro Horiguchi @ 2021-11-08 08:55 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
At Thu, 4 Nov 2021 13:34:33 +0100, Daniel Gustafsson <[email protected]> wrote in
> This patch again fails to apply (seemingly from the Perl namespace work on the
> testcode), and needs a few updates as per the above review.
Rebased the latest patch removing some of the chages.
0001: (I don't remember about this, though) I don't see how to make it
work on Windows. Anyway the next step would be to write comments.
0002: I didin't see it in details and didn't check if it finds the
issue but it actually scceeds with the fix. The change to
poll_query_until is removed since it doesn't seem actually used.
0003: The fix. I didn't touch this.
0004: Removed at all. I agree to Tom. (And I faintly remember that I
said something like that.)
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-07-08 12:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-01-05 01:07 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-01-27 08:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-03-27 14:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2021-03-30 07:12 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-07-09 19:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Ibrar Ahmed <[email protected]>
2021-08-05 10:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-08-10 20:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
2021-09-24 18:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2021-11-04 12:34 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Daniel Gustafsson <[email protected]>
2021-11-08 08:55 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
@ 2021-11-09 03:51 ` Michael Paquier <[email protected]>
2021-11-09 08:05 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Michael Paquier @ 2021-11-09 03:51 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
On Mon, Nov 08, 2021 at 05:55:16PM +0900, Kyotaro Horiguchi wrote:
I have quickly looked at the patch set.
> 0001: (I don't remember about this, though) I don't see how to make it
> work on Windows. Anyway the next step would be to write comments.
Look at Utils.pm where we have dir_symlink, then. symlink() does not
work on WIN32, so we have a wrapper that uses junction points. FWIW,
I don't like much the behavior you are enforcing in init_from_backup
when coldly copying a source path, but I have not looked enough at the
patch set to have a strong opinion about this part, either.
> 0002: I didn't see it in details and didn't check if it finds the
> issue but it actually scceeds with the fix. The change to
> poll_query_until is removed since it doesn't seem actually used.
+# Create tablespace
+my $dropme_ts_master1 = PostgreSQL::Test::Utils::tempdir();
+$dropme_ts_master1 =
PostgreSQL::Test::Utils::perl2host($dropme_ts_master1);
+my $dropme_ts_master2 = PostgreSQL::Test::Utils::tempdir();
+$dropme_ts_master2 =
PostgreSQL::Test::Utils::perl2host($dropme_ts_master2);
+my $source_ts_master = PostgreSQL::Test::Utils::tempdir();
+$source_ts_master =
PostgreSQL::Test::Utils::perl2host($source_ts_master);
+my $target_ts_master = PostgreSQL::Test::Utils::tempdir();
+$target_ts_master =
PostgreSQL::Test::Utils::perl2host($target_ts_master);
Rather than creating N temporary directories, it would be simpler to
create only one, and have subdirs in it for the rest? It seems to me
that it would make debugging much easier. The uses of perl2host()
seem sufficient.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-07-08 12:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-01-05 01:07 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-01-27 08:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-03-27 14:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2021-03-30 07:12 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-07-09 19:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Ibrar Ahmed <[email protected]>
2021-08-05 10:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-08-10 20:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
2021-09-24 18:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2021-11-04 12:34 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Daniel Gustafsson <[email protected]>
2021-11-08 08:55 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-11-09 03:51 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Michael Paquier <[email protected]>
@ 2021-11-09 08:05 ` Kyotaro Horiguchi <[email protected]>
2021-11-10 08:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Kyotaro Horiguchi @ 2021-11-09 08:05 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
At Tue, 9 Nov 2021 12:51:15 +0900, Michael Paquier <[email protected]> wrote in
> On Mon, Nov 08, 2021 at 05:55:16PM +0900, Kyotaro Horiguchi wrote:
>
> I have quickly looked at the patch set.
>
> > 0001: (I don't remember about this, though) I don't see how to make it
> > work on Windows. Anyway the next step would be to write comments.
>
> Look at Utils.pm where we have dir_symlink, then. symlink() does not
> work on WIN32, so we have a wrapper that uses junction points. FWIW,
> I don't like much the behavior you are enforcing in init_from_backup
> when coldly copying a source path, but I have not looked enough at the
> patch set to have a strong opinion about this part, either.
Thanks for the info. If we can handle symlink on Windows, we don't
need to have a cold copy.
> > 0002: I didn't see it in details and didn't check if it finds the
> > issue but it actually scceeds with the fix. The change to
> > poll_query_until is removed since it doesn't seem actually used.
>
> +# Create tablespace
> +my $dropme_ts_master1 = PostgreSQL::Test::Utils::tempdir();
> +$dropme_ts_master1 =
> PostgreSQL::Test::Utils::perl2host($dropme_ts_master1);
> +my $dropme_ts_master2 = PostgreSQL::Test::Utils::tempdir();
> +$dropme_ts_master2 =
> PostgreSQL::Test::Utils::perl2host($dropme_ts_master2);
> +my $source_ts_master = PostgreSQL::Test::Utils::tempdir();
> +$source_ts_master =
> PostgreSQL::Test::Utils::perl2host($source_ts_master);
> +my $target_ts_master = PostgreSQL::Test::Utils::tempdir();
> +$target_ts_master =
> PostgreSQL::Test::Utils::perl2host($target_ts_master);
>
> Rather than creating N temporary directories, it would be simpler to
> create only one, and have subdirs in it for the rest? It seems to me
> that it would make debugging much easier. The uses of perl2host()
> seem sufficient.
Thanks for the suggestion. My eyeballs got hopping around looking
that part so I gave up looking there in more detail:p I agree to that.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-07-08 12:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-01-05 01:07 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-01-27 08:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-03-27 14:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2021-03-30 07:12 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-07-09 19:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Ibrar Ahmed <[email protected]>
2021-08-05 10:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-08-10 20:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
2021-09-24 18:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2021-11-04 12:34 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Daniel Gustafsson <[email protected]>
2021-11-08 08:55 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-11-09 03:51 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Michael Paquier <[email protected]>
2021-11-09 08:05 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
@ 2021-11-10 08:14 ` Kyotaro Horiguchi <[email protected]>
2021-11-10 12:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Kyotaro Horiguchi @ 2021-11-10 08:14 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
At Tue, 09 Nov 2021 17:05:49 +0900 (JST), Kyotaro Horiguchi <[email protected]> wrote in
> At Tue, 9 Nov 2021 12:51:15 +0900, Michael Paquier <[email protected]> wrote in
> > Look at Utils.pm where we have dir_symlink, then. symlink() does not
> > work on WIN32, so we have a wrapper that uses junction points. FWIW,
> > I don't like much the behavior you are enforcing in init_from_backup
> > when coldly copying a source path, but I have not looked enough at the
> > patch set to have a strong opinion about this part, either.
>
> Thanks for the info. If we can handle symlink on Windows, we don't
> need to have a cold copy.
I bumped into the good-old 100-byte limit of the (v7?) tar format on
which pg_basebackup is depending. It is unlikely in the real world but
I think it is quite common in developping environment. The tablespace
directory path in my dev environment was 110 chacters-long. As small
as 10 bytes but it's quite annoying to chip off that number of bytes
from the path..
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-07-08 12:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-01-05 01:07 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-01-27 08:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-03-27 14:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2021-03-30 07:12 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-07-09 19:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Ibrar Ahmed <[email protected]>
2021-08-05 10:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-08-10 20:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
2021-09-24 18:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2021-11-04 12:34 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Daniel Gustafsson <[email protected]>
2021-11-08 08:55 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-11-09 03:51 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Michael Paquier <[email protected]>
2021-11-09 08:05 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-11-10 08:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
@ 2021-11-10 12:14 ` Alvaro Herrera <[email protected]>
2021-11-11 02:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Alvaro Herrera @ 2021-11-10 12:14 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
On 2021-Nov-10, Kyotaro Horiguchi wrote:
> I bumped into the good-old 100-byte limit of the (v7?) tar format on
> which pg_basebackup is depending. It is unlikely in the real world but
> I think it is quite common in developping environment. The tablespace
> directory path in my dev environment was 110 chacters-long. As small
> as 10 bytes but it's quite annoying to chip off that number of bytes
> from the path..
Can you use PostgreSQL::Test::Utils::tempdir_short() for those
tablespaces?
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-07-08 12:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-01-05 01:07 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-01-27 08:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-03-27 14:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2021-03-30 07:12 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-07-09 19:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Ibrar Ahmed <[email protected]>
2021-08-05 10:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-08-10 20:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
2021-09-24 18:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2021-11-04 12:34 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Daniel Gustafsson <[email protected]>
2021-11-08 08:55 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-11-09 03:51 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Michael Paquier <[email protected]>
2021-11-09 08:05 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-11-10 08:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-11-10 12:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2021-11-11 02:13 ` Kyotaro Horiguchi <[email protected]>
2021-11-12 07:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Kyotaro Horiguchi @ 2021-11-11 02:13 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
At Wed, 10 Nov 2021 09:14:30 -0300, Alvaro Herrera <[email protected]> wrote in
> Can you use PostgreSQL::Test::Utils::tempdir_short() for those
> tablespaces?
Thanks for the suggestion!
It works for a live cluster. But doesn't work for backups, since I
find no way to relate a tablespace directory with a backup directory
not using a symlink. One way would be taking a backup with tentative
tablespace directory in the short-named temporary directory then move
it into the backup direcotry. I'm going that way for now.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-07-08 12:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-01-05 01:07 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-01-27 08:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-03-27 14:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2021-03-30 07:12 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-07-09 19:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Ibrar Ahmed <[email protected]>
2021-08-05 10:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-08-10 20:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
2021-09-24 18:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2021-11-04 12:34 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Daniel Gustafsson <[email protected]>
2021-11-08 08:55 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-11-09 03:51 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Michael Paquier <[email protected]>
2021-11-09 08:05 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-11-10 08:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-11-10 12:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2021-11-11 02:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
@ 2021-11-12 07:43 ` Kyotaro Horiguchi <[email protected]>
2021-12-24 10:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Kyotaro Horiguchi @ 2021-11-12 07:43 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
At Thu, 11 Nov 2021 11:13:52 +0900 (JST), Kyotaro Horiguchi <[email protected]> wrote in
> At Wed, 10 Nov 2021 09:14:30 -0300, Alvaro Herrera <[email protected]> wrote in
> > Can you use PostgreSQL::Test::Utils::tempdir_short() for those
> > tablespaces?
>
> Thanks for the suggestion!
>
> It works for a live cluster. But doesn't work for backups, since I
> find no way to relate a tablespace directory with a backup directory
> not using a symlink. One way would be taking a backup with tentative
> tablespace directory in the short-named temporary directory then move
> it into the backup direcotry. I'm going that way for now.
This is that.
0001 adds several routines to handle tablespace directories, and adds
tablespace support to backup/_backup_fs.
We don't know an oid corresponding to a tablespace directory before
actually assigning the oid to the tablespace. So we cannot name a
tablespace directory after the oid. On the other hand, after defining
the tablespace, cold data files don't tell the real directory name of
the tablespace directory for an oid or a tablespace name, unless we
have readlink.
The function dir_readlink added to Utils.pm is that. Honestly I don't
like the way function works. It uses "cmd /c "dir /A:L $dir"" to
collect information of junctions. I'm not sure that the type label
"<JUNCTION>" is immutable among locales but at least it is shown as
"<JUNCTION>" on Japanese (CP-932) environment. I didn't actually
tested it on Windows and msys environment ...yet.
Premising the availability of the function, we can name tablespace
directories from meaningful words.
The directory to store tablespace directories can be a temporary
directory, but with that way it is needed to create a symlink to find
those directories from a backup. I chose to place tablespace
directories directly under backup directory.
The attached first file is a revised (or remade) version of tablespace
support for TAP test.
The second is the version adapted to the revised framework. (I
confirmed that the test actually detects the error.)
The third is not changed at all.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-04-22 04:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-22 07:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-23 07:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-30 06:33 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-07 06:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-05-14 03:06 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-06-19 07:21 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-07-08 03:15 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-07-15 10:52 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-08-01 22:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2019-08-22 13:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-09-19 11:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-09-20 12:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Asim R P <[email protected]>
2019-12-26 14:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-15 10:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2020-01-27 15:24 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Fujii Masao <[email protected]>
2020-07-08 12:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-01-05 01:07 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-01-27 08:36 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-03-27 14:23 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2021-03-30 07:12 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-07-09 19:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Ibrar Ahmed <[email protected]>
2021-08-05 10:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2021-08-10 20:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
2021-09-24 18:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2021-11-04 12:34 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Daniel Gustafsson <[email protected]>
2021-11-08 08:55 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-11-09 03:51 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Michael Paquier <[email protected]>
2021-11-09 08:05 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-11-10 08:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-11-10 12:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2021-11-11 02:13 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2021-11-12 07:43 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
@ 2021-12-24 10:21 ` Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 109+ messages in thread
From: Kyotaro Horiguchi @ 2021-12-24 10:21 UTC (permalink / raw)
To: pgsql-hackers
Just a complaint..
At Fri, 12 Nov 2021 16:43:27 +0900 (JST), Kyotaro Horiguchi <[email protected]> wrote in
> "<JUNCTION>" on Japanese (CP-932) environment. I didn't actually
> tested it on Windows and msys environment ...yet.
Active perl cannot be installed because of (perhaps) a powershell
version issue... Annoying..
https://community.activestate.com/t/please-update-your-powershell-install-scripts/7897
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
@ 2022-03-25 12:26 Alvaro Herrera <[email protected]>
2022-03-28 01:01 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-03-28 14:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
0 siblings, 2 replies; 109+ messages in thread
From: Alvaro Herrera @ 2022-03-25 12:26 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; pgsql-hackers
On 2022-Mar-21, Alvaro Herrera wrote:
> I had a look at this latest version of the patch, and found some things
> to tweak. Attached is v21 with three main changes from Kyotaro's v20:
Pushed this, backpatching to 14 and 13. It would have been good to
backpatch further, but there's an (textually trivial) merge conflict
related to commit e6d8069522c8. Because that commit conceptually
touches the same area that this bugfix is about, I'm not sure that
backpatching further without a lot more thought is wise -- particularly
so when there's no way to automate the test in branches older than
master.
This is quite annoying, considering that the bug was reported shortly
before 12 went into beta.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"If you have nothing to say, maybe you need just the right tool to help you
not say it." (New York Times, about Microsoft PowerPoint)
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-03-25 12:26 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2022-03-28 01:01 ` Kyotaro Horiguchi <[email protected]>
2022-03-28 01:34 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-03-28 08:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
1 sibling, 2 replies; 109+ messages in thread
From: Kyotaro Horiguchi @ 2022-03-28 01:01 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; pgsql-hackers
At Fri, 25 Mar 2022 13:26:05 +0100, Alvaro Herrera <[email protected]> wrote in
> On 2022-Mar-21, Alvaro Herrera wrote:
>
> > I had a look at this latest version of the patch, and found some things
> > to tweak. Attached is v21 with three main changes from Kyotaro's v20:
>
> Pushed this, backpatching to 14 and 13. It would have been good to
> backpatch further, but there's an (textually trivial) merge conflict
> related to commit e6d8069522c8. Because that commit conceptually
> touches the same area that this bugfix is about, I'm not sure that
> backpatching further without a lot more thought is wise -- particularly
> so when there's no way to automate the test in branches older than
> master.
Thaks for committing.
> This is quite annoying, considering that the bug was reported shortly
> before 12 went into beta.
Sure. I'm going to look into that.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-03-25 12:26 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-03-28 01:01 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
@ 2022-03-28 01:34 ` Thomas Munro <[email protected]>
2022-03-28 06:25 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-03-29 01:57 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Michael Paquier <[email protected]>
1 sibling, 2 replies; 109+ messages in thread
From: Thomas Munro @ 2022-03-28 01:34 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Michael Paquier <[email protected]>; Julien Rouhaud <[email protected]>; pgsql-hackers
On Mon, Mar 28, 2022 at 2:01 PM Kyotaro Horiguchi
<[email protected]> wrote:
> At Fri, 25 Mar 2022 13:26:05 +0100, Alvaro Herrera <[email protected]> wrote in
> > Pushed this, backpatching to 14 and 13. It would have been good to
> > backpatch further, but there's an (textually trivial) merge conflict
> > related to commit e6d8069522c8. Because that commit conceptually
> > touches the same area that this bugfix is about, I'm not sure that
> > backpatching further without a lot more thought is wise -- particularly
> > so when there's no way to automate the test in branches older than
> > master.
Just a thought: we could consider back-patching
allow_in_place_tablespaces, after a little while, if we're happy with
how that is working out, if it'd be useful for verifying bug fixes in
back branches. It's non-end-user-facing testing infrastructure.
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-03-25 12:26 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-03-28 01:01 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-03-28 01:34 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
@ 2022-03-28 06:25 ` Kyotaro Horiguchi <[email protected]>
1 sibling, 0 replies; 109+ messages in thread
From: Kyotaro Horiguchi @ 2022-03-28 06:25 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers
At Mon, 28 Mar 2022 14:34:44 +1300, Thomas Munro <[email protected]> wrote in
> On Mon, Mar 28, 2022 at 2:01 PM Kyotaro Horiguchi
> <[email protected]> wrote:
> > At Fri, 25 Mar 2022 13:26:05 +0100, Alvaro Herrera <[email protected]> wrote in
> > > Pushed this, backpatching to 14 and 13. It would have been good to
> > > backpatch further, but there's an (textually trivial) merge conflict
> > > related to commit e6d8069522c8. Because that commit conceptually
> > > touches the same area that this bugfix is about, I'm not sure that
> > > backpatching further without a lot more thought is wise -- particularly
> > > so when there's no way to automate the test in branches older than
> > > master.
>
> Just a thought: we could consider back-patching
> allow_in_place_tablespaces, after a little while, if we're happy with
> how that is working out, if it'd be useful for verifying bug fixes in
> back branches. It's non-end-user-facing testing infrastructure.
I appreciate if we accept that. The patch is simple. And it now has
the clear use-case for back-patching.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-03-25 12:26 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-03-28 01:01 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-03-28 01:34 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
@ 2022-03-29 01:57 ` Michael Paquier <[email protected]>
1 sibling, 0 replies; 109+ messages in thread
From: Michael Paquier @ 2022-03-29 01:57 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; Alvaro Herrera <[email protected]>; Julien Rouhaud <[email protected]>; pgsql-hackers
On Mon, Mar 28, 2022 at 02:34:44PM +1300, Thomas Munro wrote:
> Just a thought: we could consider back-patching
> allow_in_place_tablespaces, after a little while, if we're happy with
> how that is working out, if it'd be useful for verifying bug fixes in
> back branches. It's non-end-user-facing testing infrastructure.
+1 for a backpatch on that. That would be useful.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../YkJnliFnXODscRJ%[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-03-25 12:26 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-03-28 01:01 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
@ 2022-03-28 08:20 ` Kyotaro Horiguchi <[email protected]>
1 sibling, 0 replies; 109+ messages in thread
From: Kyotaro Horiguchi @ 2022-03-28 08:20 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; pgsql-hackers
At Mon, 28 Mar 2022 10:01:05 +0900 (JST), Kyotaro Horiguchi <[email protected]> wrote in
> At Fri, 25 Mar 2022 13:26:05 +0100, Alvaro Herrera <[email protected]> wrote in
> > Pushed this, backpatching to 14 and 13. It would have been good to
> > backpatch further, but there's an (textually trivial) merge conflict
> > related to commit e6d8069522c8. Because that commit conceptually
> > touches the same area that this bugfix is about, I'm not sure that
> > backpatching further without a lot more thought is wise -- particularly
> > so when there's no way to automate the test in branches older than
> > master.
>
> Thaks for committing.
>
> > This is quite annoying, considering that the bug was reported shortly
> > before 12 went into beta.
>
> Sure. I'm going to look into that.
This is a preparatory patch and tentative (yes, it's just tentative)
test. This is made for 12 but applies with some warnings to 10-11.
(Hope the attachments are attached as "attachment", not "inline".)
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-03-25 12:26 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2022-03-28 14:37 ` Robert Haas <[email protected]>
2022-03-29 01:34 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
1 sibling, 1 reply; 109+ messages in thread
From: Robert Haas @ 2022-03-28 14:37 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; Michael Paquier <[email protected]>; Julien Rouhaud <[email protected]>; pgsql-hackers
On Fri, Mar 25, 2022 at 8:26 AM Alvaro Herrera <[email protected]> wrote:
> On 2022-Mar-21, Alvaro Herrera wrote:
> > I had a look at this latest version of the patch, and found some things
> > to tweak. Attached is v21 with three main changes from Kyotaro's v20:
>
> Pushed this, backpatching to 14 and 13. It would have been good to
> backpatch further, but there's an (textually trivial) merge conflict
> related to commit e6d8069522c8. Because that commit conceptually
> touches the same area that this bugfix is about, I'm not sure that
> backpatching further without a lot more thought is wise -- particularly
> so when there's no way to automate the test in branches older than
> master.
>
> This is quite annoying, considering that the bug was reported shortly
> before 12 went into beta.
I think that the warnings this patch issues may cause some unnecessary
end-user alarm. It seems to me that they are basically warning about a
situation that is unusual but not scary. Isn't the appropriate level
for that DEBUG1, maybe without the errhint?
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-03-25 12:26 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-03-28 14:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
@ 2022-03-29 01:34 ` Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 109+ messages in thread
From: Kyotaro Horiguchi @ 2022-03-29 01:34 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers
At Mon, 28 Mar 2022 10:37:04 -0400, Robert Haas <[email protected]> wrote in
> On Fri, Mar 25, 2022 at 8:26 AM Alvaro Herrera <[email protected]> wrote:
> > On 2022-Mar-21, Alvaro Herrera wrote:
> > > I had a look at this latest version of the patch, and found some things
> > > to tweak. Attached is v21 with three main changes from Kyotaro's v20:
> >
> > Pushed this, backpatching to 14 and 13. It would have been good to
> > backpatch further, but there's an (textually trivial) merge conflict
> > related to commit e6d8069522c8. Because that commit conceptually
> > touches the same area that this bugfix is about, I'm not sure that
> > backpatching further without a lot more thought is wise -- particularly
> > so when there's no way to automate the test in branches older than
> > master.
> >
> > This is quite annoying, considering that the bug was reported shortly
> > before 12 went into beta.
>
> I think that the warnings this patch issues may cause some unnecessary
> end-user alarm. It seems to me that they are basically warning about a
> situation that is unusual but not scary. Isn't the appropriate level
> for that DEBUG1, maybe without the errhint?
log_invalid_page reports missing pages with DEBUG1 before reaching
consistency. And since missing directory is not an issue if all of
those reports are forgotten until reaching consistency, DEBUG1 sounds
reasonable. Maybe we lower the DEBUG1 messages to DEBUG2 in
XLogRememberMissingDir?
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
@ 2022-03-28 16:17 Robert Haas <[email protected]>
2022-03-29 04:55 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Robert Haas @ 2022-03-28 16:17 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; Michael Paquier <[email protected]>; Julien Rouhaud <[email protected]>; pgsql-hackers
On Mon, Mar 21, 2022 at 3:02 PM Alvaro Herrera <[email protected]> wrote:
> > 2. Why not instead change the code so that the operation can succeed,
> > by creating the prerequisite parent directories? Do we not have enough
> > information for that? I'm not saying that we definitely should do it
> > that way rather than this way, but I think we do take that approach in
> > some cases.
>
> It seems we can choose freely between these two implementations -- I
> mean I don't see any upsides or downsides to either one.
What got committed here feels inconsistent to me. Suppose we have a
checkpoint, and then a series of operations that touch a tablespace,
and then a drop database and drop tablespace. If the first operation
happens to be CREATE DATABASE, then this patch is going to fix it by
skipping the operation. However, if the first operation happens to be
almost anything else, the way it's going to reference the dropped
tablespace is via a block reference in a WAL record of a wide variety
of types. That's going to result in a call to
XLogReadBufferForRedoExtended() which will call
XLogReadBufferExtended() which will do smgrcreate(smgr, forknum, true)
which will in turn call TablespaceCreateDbspace() to fill in all the
missing directories.
I don't think that's very good. It would be reasonable to decide that
we're never going to create the missing directories and instead just
remember that they were not found so we can do a cross check. It's
also reasonable to just create the directories on the fly. But doing a
mix of those systems doesn't really seem like the right idea -
particularly because it means that the cross-check system is probably
not very effective at finding actual problems in the code.
Am I missing something here?
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-03-28 16:17 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
@ 2022-03-29 04:55 ` Kyotaro Horiguchi <[email protected]>
2022-03-29 11:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Kyotaro Horiguchi @ 2022-03-29 04:55 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers
At Mon, 28 Mar 2022 12:17:50 -0400, Robert Haas <[email protected]> wrote in
> On Mon, Mar 21, 2022 at 3:02 PM Alvaro Herrera <[email protected]> wrote:
> > > 2. Why not instead change the code so that the operation can succeed,
> > > by creating the prerequisite parent directories? Do we not have enough
> > > information for that? I'm not saying that we definitely should do it
> > > that way rather than this way, but I think we do take that approach in
> > > some cases.
> >
> > It seems we can choose freely between these two implementations -- I
> > mean I don't see any upsides or downsides to either one.
>
> What got committed here feels inconsistent to me. Suppose we have a
> checkpoint, and then a series of operations that touch a tablespace,
> and then a drop database and drop tablespace. If the first operation
> happens to be CREATE DATABASE, then this patch is going to fix it by
> skipping the operation. However, if the first operation happens to be
> almost anything else, the way it's going to reference the dropped
> tablespace is via a block reference in a WAL record of a wide variety
> of types. That's going to result in a call to
> XLogReadBufferForRedoExtended() which will call
> XLogReadBufferExtended() which will do smgrcreate(smgr, forknum, true)
> which will in turn call TablespaceCreateDbspace() to fill in all the
> missing directories.
Right. I thought that recovery avoids that but that's wrong. This
behavior creates a bare (non-linked) directly within pg_tblspc. The
directory would dissapear soon if recovery proceeds to the consistency
point, though.
> I don't think that's very good. It would be reasonable to decide that
> we're never going to create the missing directories and instead just
> remember that they were not found so we can do a cross check. It's
> also reasonable to just create the directories on the fly. But doing a
> mix of those systems doesn't really seem like the right idea -
> particularly because it means that the cross-check system is probably
> not very effective at finding actual problems in the code.
>
> Am I missing something here?
No. I agree that mixing them is not good. On the other hand we
already doing that by heapam. AFAICS sometimes it avoid creating a
new page but sometimes creates it. But I don't mean to use the fact
for justifying this patch to do that, or denying to do that.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-03-28 16:17 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
2022-03-29 04:55 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
@ 2022-03-29 11:37 ` Alvaro Herrera <[email protected]>
2022-03-29 12:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Alvaro Herrera @ 2022-03-29 11:37 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers
On 2022-Mar-29, Kyotaro Horiguchi wrote:
> > That's going to result in a call to
> > XLogReadBufferForRedoExtended() which will call
> > XLogReadBufferExtended() which will do smgrcreate(smgr, forknum, true)
> > which will in turn call TablespaceCreateDbspace() to fill in all the
> > missing directories.
>
> Right. I thought that recovery avoids that but that's wrong. This
> behavior creates a bare (non-linked) directly within pg_tblspc. The
> directory would dissapear soon if recovery proceeds to the consistency
> point, though.
Hmm, this is not good.
> No. I agree that mixing them is not good. On the other hand we
> already doing that by heapam. AFAICS sometimes it avoid creating a
> new page but sometimes creates it. But I don't mean to use the fact
> for justifying this patch to do that, or denying to do that.
I think we should revert this patch and do it again using the other
approach: create a stub directory during recovery that can be deleted
later.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"Porque francamente, si para saber manejarse a uno mismo hubiera que
rendir examen... ¿Quién es el machito que tendría carnet?" (Mafalda)
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-03-28 16:17 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
2022-03-29 04:55 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-03-29 11:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2022-03-29 12:45 ` Robert Haas <[email protected]>
2022-03-29 13:28 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Robert Haas @ 2022-03-29 12:45 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; Michael Paquier <[email protected]>; Julien Rouhaud <[email protected]>; pgsql-hackers
On Tue, Mar 29, 2022 at 7:37 AM Alvaro Herrera <[email protected]> wrote:
> I think we should revert this patch and do it again using the other
> approach: create a stub directory during recovery that can be deleted
> later.
I'm fine with that approach, but I'd like to ask that we proceed
expeditiously, because I have another patch that I want to commit that
touches this area. I can commit to helping with whatever we decide to
do here, but I don't want to keep that patch on ice while we figure it
out and then have it miss the release.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-03-28 16:17 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
2022-03-29 04:55 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-03-29 11:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-03-29 12:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
@ 2022-03-29 13:28 ` Alvaro Herrera <[email protected]>
2022-03-29 13:31 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Alvaro Herrera @ 2022-03-29 13:28 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; Michael Paquier <[email protected]>; Julien Rouhaud <[email protected]>; pgsql-hackers
On 2022-Mar-29, Robert Haas wrote:
> I'm fine with that approach, but I'd like to ask that we proceed
> expeditiously, because I have another patch that I want to commit that
> touches this area. I can commit to helping with whatever we decide to
> do here, but I don't want to keep that patch on ice while we figure it
> out and then have it miss the release.
OK, this is a bug that's been open for years. A fix can be committed
after the feature freeze anyway.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-03-28 16:17 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
2022-03-29 04:55 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-03-29 11:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-03-29 12:45 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
2022-03-29 13:28 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2022-03-29 13:31 ` Robert Haas <[email protected]>
0 siblings, 0 replies; 109+ messages in thread
From: Robert Haas @ 2022-03-29 13:31 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; Michael Paquier <[email protected]>; Julien Rouhaud <[email protected]>; pgsql-hackers
On Tue, Mar 29, 2022 at 9:28 AM Alvaro Herrera <[email protected]> wrote:
> OK, this is a bug that's been open for years. A fix can be committed
> after the feature freeze anyway.
+1
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
@ 2022-07-13 16:43 Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Alvaro Herrera @ 2022-07-13 16:43 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
Not a review, just a preparatory rebase across some trivially
conflicting changes. I also noticed that
src/test/recovery/t/031_recovery_conflict.pl, which was added two days
after v23 was sent, and which uses allow_in_place_tablespaces, bails out
because of the checks introduced by this patch, so I made the check
routine do nothing in that case.
Anyway, here's v24.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"La conclusión que podemos sacar de esos estudios es que
no podemos sacar ninguna conclusión de ellos" (Tanenbaum)
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2022-07-14 21:47 ` Alvaro Herrera <[email protected]>
2022-07-15 07:30 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
0 siblings, 2 replies; 109+ messages in thread
From: Alvaro Herrera @ 2022-07-14 21:47 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
Here's a couple of fixups. 0001 is the same as before. In 0002 I think
CheckTablespaceDirectory ends up easier to read if we split out the test
for validity of the link. Looking at that again, I think we don't need
to piggyback on ignore_invalid_pages, which is already a stretch, so
let's not -- instead we can use allow_in_place_tablespaces if users need
a workaround. So that's 0003 (this bit needs more than zero docs,
however.)
0004 is straightforward: let's check for bad directories before logging
about consistent state.
After all this, I'm not sure what to think of dbase_redo. At line 3102,
is the directory supposed to exist or not? I'm confused as to what is
the expected state at that point. I rewrote this, but now I think my
rewrite continues to be confusing, so I'll have to think more about it.
Another aspect are the tests. Robert described a scenario where the
previously committed version of this patch created trouble. Do we have
a test case to cover that problematic case? I think we should strive to
cover it, if possible.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"The eagle never lost so much time, as
when he submitted to learn of the crow." (William Blake)
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2022-07-15 07:30 ` Kyotaro Horiguchi <[email protected]>
2022-07-15 07:56 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-15 10:58 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
1 sibling, 2 replies; 109+ messages in thread
From: Kyotaro Horiguchi @ 2022-07-15 07:30 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
At Thu, 14 Jul 2022 23:47:40 +0200, Alvaro Herrera <[email protected]> wrote in
> Here's a couple of fixups. 0001 is the same as before. In 0002 I think
Thanks!
+ if (!S_ISLNK(st.st_mode))
+ #else
+ if (!pgwin32_is_junction(path))
+ #endif
+ elog(ignore_invalid_pages ? WARNING : PANIC,
+ "real directory found in pg_tblspc directory: %s", de->d_name);
A regular file with an oid-name also causes this error. Doesn't
something like "unexpected non-(sym)link entry..." work?
> CheckTablespaceDirectory ends up easier to read if we split out the test
> for validity of the link. Looking at that again, I think we don't need
> to piggyback on ignore_invalid_pages, which is already a stretch, so
> let's not -- instead we can use allow_in_place_tablespaces if users need
> a workaround. So that's 0003 (this bit needs more than zero docs,
> however.)
The result of 0003 looks good.
0002:
+is_path_tslink(const char *path)
What the "ts" of tslink stands for? If it stands for tablespace, the
function is not specific for table spaces. We already have
+ errmsg("could not stat file \"%s\": %m", path));
I'm not sure we need such correctness, but what is failing there is
lstat. I found similar codes in two places in backend and one place
in frontend. So couldn't it be moved to /common and have a more
generic name?
- dir = AllocateDir(tblspc_path);
- while ((de = ReadDir(dir, tblspc_path)) != NULL)
+ dir = AllocateDir("pg_tblspc");
+ while ((de = ReadDir(dir, "pg_tblspc")) != NULL)
xlog.c uses the macro XLOGDIR. Why don't we define TBLSPCDIR?
- for (p = de->d_name; *p && isdigit(*p); p++);
- if (*p)
+ if (strspn(de->d_name, "0123456789") != strlen(de->d_name))
continue;
The pattern "strspn != strlen" looks kind of remote, or somewhat
pedantic..
+ char path[MAXPGPATH + 10];
..
- snprintf(path, MAXPGPATH, "%s/%s", tblspc_path, de->d_name);
+ snprintf(path, sizeof(path), "pg_tblspc/%s", de->d_name);
I don't think we need the extra 10 bytes. A bit paranoic, but we can
check the return value to confirm the d_name is fully stored in the
buffer.
> 0004 is straightforward: let's check for bad directories before logging
> about consistent state.
I was about to write a comment to do this when looking 0001.
> After all this, I'm not sure what to think of dbase_redo. At line 3102,
> is the directory supposed to exist or not? I'm confused as to what is
> the expected state at that point. I rewrote this, but now I think my
> rewrite continues to be confusing, so I'll have to think more about it.
I'm not sure l3102 exactly points, but haven't we chosen to create
everything required to keep recovery going, whether it is supposed to
exist or not?
> Another aspect are the tests. Robert described a scenario where the
> previously committed version of this patch created trouble. Do we have
> a test case to cover that problematic case? I think we should strive to
> cover it, if possible.
I counldn't recall that clearly and failed to dig out from the thread,
but doesn't the "creating everything needed" strategy naturally save
that case? We could add that test, but it seems to me a little
cumbersome to confirm the test correctly detect that case..
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-15 07:30 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
@ 2022-07-15 07:56 ` Alvaro Herrera <[email protected]>
1 sibling, 0 replies; 109+ messages in thread
From: Alvaro Herrera @ 2022-07-15 07:56 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
On 2022-Jul-15, Kyotaro Horiguchi wrote:
> At Thu, 14 Jul 2022 23:47:40 +0200, Alvaro Herrera <[email protected]> wrote in
> > Here's a couple of fixups. 0001 is the same as before. In 0002 I think
>
> Thanks!
>
> + if (!S_ISLNK(st.st_mode))
> + #else
> + if (!pgwin32_is_junction(path))
> + #endif
> + elog(ignore_invalid_pages ? WARNING : PANIC,
> + "real directory found in pg_tblspc directory: %s", de->d_name);
>
> A regular file with an oid-name also causes this error. Doesn't
> something like "unexpected non-(sym)link entry..." work?
Hmm, good point. I also wonder if we need to cater for using the term
"junction point" rather than "symlink" when under Windows.
> > CheckTablespaceDirectory ends up easier to read if we split out the test
> > for validity of the link. Looking at that again, I think we don't need
> > to piggyback on ignore_invalid_pages, which is already a stretch, so
> > let's not -- instead we can use allow_in_place_tablespaces if users need
> > a workaround. So that's 0003 (this bit needs more than zero docs,
> > however.)
>
> The result of 0003 looks good.
Great, will merge.
> 0002:
> +is_path_tslink(const char *path)
>
> What the "ts" of tslink stands for? If it stands for tablespace, the
> function is not specific for table spaces.
Oh, of course.
> We already have
>
> + errmsg("could not stat file \"%s\": %m", path));
>
> I'm not sure we need such correctness, but what is failing there is
> lstat.
I'll have a look at what we use for lstat failures in other places.
> I found similar codes in two places in backend and one place
> in frontend. So couldn't it be moved to /common and have a more
> generic name?
I'll have a look at those. I had the same instinct initially ...
> - dir = AllocateDir(tblspc_path);
> - while ((de = ReadDir(dir, tblspc_path)) != NULL)
> + dir = AllocateDir("pg_tblspc");
> + while ((de = ReadDir(dir, "pg_tblspc")) != NULL)
>
> xlog.c uses the macro XLOGDIR. Why don't we define TBLSPCDIR?
Oh yes, let's do that. I'd even backpatch that, to avoid a future
backpatching gotcha.
> - for (p = de->d_name; *p && isdigit(*p); p++);
> - if (*p)
> + if (strspn(de->d_name, "0123456789") != strlen(de->d_name))
> continue;
>
> The pattern "strspn != strlen" looks kind of remote, or somewhat
> pedantic..
>
> + char path[MAXPGPATH + 10];
> ..
> - snprintf(path, MAXPGPATH, "%s/%s", tblspc_path, de->d_name);
> + snprintf(path, sizeof(path), "pg_tblspc/%s", de->d_name);
>
> I don't think we need the extra 10 bytes.
I forgot to mention this, but I just copied these bits from some other
place that processes pg_tblspc entries. It seemed to me that the
bodiless for loop was a bit too suspicious-looking.
> A bit paranoic, but we can check the return value to confirm the
> d_name is fully stored in the buffer.
Hmm ... I don't think we need to care about that in this patch. This
coding pattern is already being used in other places. If we want to
change that, let's do it everywhere, and not in an unrelated
backpatchable bug fix.
> > After all this, I'm not sure what to think of dbase_redo. At line 3102,
> > is the directory supposed to exist or not? I'm confused as to what is
> > the expected state at that point. I rewrote this, but now I think my
> > rewrite continues to be confusing, so I'll have to think more about it.
>
> I'm not sure l3102 exactly points, but haven't we chosen to create
> everything required to keep recovery going, whether it is supposed to
> exist or not?
I mean just after the two stat() calls for the target directory.
> > Another aspect are the tests. Robert described a scenario where the
> > previously committed version of this patch created trouble. Do we have
> > a test case to cover that problematic case? I think we should strive to
> > cover it, if possible.
>
> I counldn't recall that clearly and failed to dig out from the thread,
> but doesn't the "creating everything needed" strategy naturally save
> that case? We could add that test, but it seems to me a little
> cumbersome to confirm the test correctly detect that case..
Well, I *hope* it does ... but hope is no strategy, and I've frequently
been on the wrong side when trusting that untested code does what I
think it does.
Thanks for reviewing,
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"It takes less than 2 seconds to get to 78% complete; that's a good sign.
A few seconds later it's at 90%, but it seems to have stuck there. Did
somebody make percentages logarithmic while I wasn't looking?"
http://smylers.hates-software.com/2005/09/08/1995c749.html
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-15 07:30 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
@ 2022-07-15 10:58 ` Alvaro Herrera <[email protected]>
2022-07-15 12:03 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
1 sibling, 1 reply; 109+ messages in thread
From: Alvaro Herrera @ 2022-07-15 10:58 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
On 2022-Jul-15, Kyotaro Horiguchi wrote:
> 0002:
> +is_path_tslink(const char *path)
>
> What the "ts" of tslink stands for? If it stands for tablespace, the
> function is not specific for table spaces. We already have
>
> + errmsg("could not stat file \"%s\": %m", path));
>
> I'm not sure we need such correctness, but what is failing there is
> lstat. I found similar codes in two places in backend and one place
> in frontend. So couldn't it be moved to /common and have a more
> generic name?
I wondered whether it'd be better to check whether get_dirent_type
returns PGFILETYPE_LNK. However, that doesn't deal with junction points
at all, which seems pretty odd ... I mean, isn't it rather useful as an
abstraction if it doesn't abstract away the one platform-dependent point
we have in the area?
However, looking closer I noticed that on Windows we use our own
readdir() implementation, which AFAICT includes everything to handle
reparse points as symlinks correctly in get_dirent_type. Which means
that do_pg_start_backup is wasting its time with the "#ifdef WIN32" bits
to handle junction points separately. We could just do this
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index b809a2152c..4966213fde 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8302,13 +8302,8 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* we sometimes use allow_in_place_tablespaces to create
* directories directly under pg_tblspc, which would fail below.
*/
-#ifdef WIN32
- if (!pgwin32_is_junction(fullpath))
- continue;
-#else
if (get_dirent_type(fullpath, de, false, ERROR) != PGFILETYPE_LNK)
continue;
-#endif
#if defined(HAVE_READLINK) || defined(WIN32)
rllen = readlink(fullpath, linkpath, sizeof(linkpath));
And everything should continue to work.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-15 07:30 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-07-15 10:58 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2022-07-15 12:03 ` Alvaro Herrera <[email protected]>
0 siblings, 0 replies; 109+ messages in thread
From: Alvaro Herrera @ 2022-07-15 12:03 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
On 2022-Jul-15, Alvaro Herrera wrote:
> However, looking closer I noticed that on Windows we use our own
> readdir() implementation, which AFAICT includes everything to handle
> reparse points as symlinks correctly in get_dirent_type. Which means
> that do_pg_start_backup is wasting its time with the "#ifdef WIN32" bits
> to handle junction points separately. We could just do this
>
> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
> index b809a2152c..4966213fde 100644
> --- a/src/backend/access/transam/xlog.c
> +++ b/src/backend/access/transam/xlog.c
> @@ -8302,13 +8302,8 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
> * we sometimes use allow_in_place_tablespaces to create
> * directories directly under pg_tblspc, which would fail below.
> */
> -#ifdef WIN32
> - if (!pgwin32_is_junction(fullpath))
> - continue;
> -#else
> if (get_dirent_type(fullpath, de, false, ERROR) != PGFILETYPE_LNK)
> continue;
> -#endif
>
> #if defined(HAVE_READLINK) || defined(WIN32)
> rllen = readlink(fullpath, linkpath, sizeof(linkpath));
>
> And everything should continue to work.
Hmm, but it does not:
https://cirrus-ci.com/build/4824963784900608
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2022-07-20 10:50 ` Alvaro Herrera <[email protected]>
2022-07-20 15:25 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:11 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-07-27 18:54 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
1 sibling, 3 replies; 109+ messages in thread
From: Alvaro Herrera @ 2022-07-20 10:50 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
v26 here. I spent some time fighting the readdir() stuff for
Windows (so that get_dirent_type returns LNK for junction points)
but couldn't make it to work and was unable to figure out why.
So I ended up doing what do_pg_backup_start is already doing:
an #ifdef to call pgwin32_is_junction instead. I remove the
newly added path_is_symlink function, because I realized that
it would mean an extra syscall everywhere other than Windows.
So if somebody wants to fix get_dirent_type() so that it works properly
on Windows, we can change all these places together.
I also change the use of allow_invalid_pages to
allow_in_place_tablespaces. We could add a
separate GUC for this, but it seems overengineering.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Most hackers will be perfectly comfortable conceptualizing users as entropy
sources, so let's move on." (Nathaniel Smith)
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2022-07-20 15:25 ` Alvaro Herrera <[email protected]>
2022-07-20 16:34 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:01 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2 siblings, 2 replies; 109+ messages in thread
From: Alvaro Herrera @ 2022-07-20 15:25 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
On 2022-Jul-20, Alvaro Herrera wrote:
> I also change the use of allow_invalid_pages to
> allow_in_place_tablespaces. We could add a
> separate GUC for this, but it seems overengineering.
Oh, but allow_in_place_tablespaces doesn't exist in versions 14 and
older, so this strategy doesn't really work.
I see the following alternatives:
1. not backpatch this fix to 14 and older
2. use a different GUC; either allow_invalid_pages as previously
suggested, or create a new one just for this purpose
3. not provide any overriding mechanism in versions 14 and older
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Always assume the user will do much worse than the stupidest thing
you can imagine." (Julien PUYDT)
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 15:25 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2022-07-20 16:34 ` Alvaro Herrera <[email protected]>
1 sibling, 0 replies; 109+ messages in thread
From: Alvaro Herrera @ 2022-07-20 16:34 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
On 2022-Jul-20, Alvaro Herrera wrote:
> On 2022-Jul-20, Alvaro Herrera wrote:
>
> > I also change the use of allow_invalid_pages to
> > allow_in_place_tablespaces. We could add a
> > separate GUC for this, but it seems overengineering.
>
> Oh, but allow_in_place_tablespaces doesn't exist in versions 14 and
> older, so this strategy doesn't really work.
... and get_dirent_type is new in 14, so that'll be one more hurdle.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"Cuando no hay humildad las personas se degradan" (A. Christie)
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 15:25 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2022-07-21 11:01 ` Alvaro Herrera <[email protected]>
2022-07-21 11:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
1 sibling, 1 reply; 109+ messages in thread
From: Alvaro Herrera @ 2022-07-21 11:01 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
On 2022-Jul-20, Alvaro Herrera wrote:
> I see the following alternatives:
>
> 1. not backpatch this fix to 14 and older
> 2. use a different GUC; either allow_invalid_pages as previously
> suggested, or create a new one just for this purpose
> 3. not provide any overriding mechanism in versions 14 and older
I've got no opinions on this. I don't like either 1 or 3, so I'm going
to add and backpatch a new GUC allow_recovery_tablespaces as the
override mechanism.
If others disagree with this choice, please speak up.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 15:25 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:01 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2022-07-21 11:14 ` Thomas Munro <[email protected]>
2022-07-21 11:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-22 00:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
0 siblings, 2 replies; 109+ messages in thread
From: Thomas Munro @ 2022-07-21 11:14 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; Dilip Kumar <[email protected]>; Robert Haas <[email protected]>; Michael Paquier <[email protected]>; Julien Rouhaud <[email protected]>; pgsql-hackers
On Thu, Jul 21, 2022 at 11:01 PM Alvaro Herrera <[email protected]> wrote:
> On 2022-Jul-20, Alvaro Herrera wrote:
> > I see the following alternatives:
> >
> > 1. not backpatch this fix to 14 and older
> > 2. use a different GUC; either allow_invalid_pages as previously
> > suggested, or create a new one just for this purpose
> > 3. not provide any overriding mechanism in versions 14 and older
>
> I've got no opinions on this. I don't like either 1 or 3, so I'm going
> to add and backpatch a new GUC allow_recovery_tablespaces as the
> override mechanism.
>
> If others disagree with this choice, please speak up.
Would it help if we back-patched the allow_in_place_tablespaces stuff?
I'm not sure how hard/destabilising that would be, but I could take a
look tomorrow.
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 15:25 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:01 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
@ 2022-07-21 11:20 ` Alvaro Herrera <[email protected]>
2022-07-21 11:25 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
1 sibling, 1 reply; 109+ messages in thread
From: Alvaro Herrera @ 2022-07-21 11:20 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; Dilip Kumar <[email protected]>; Robert Haas <[email protected]>; Michael Paquier <[email protected]>; Julien Rouhaud <[email protected]>; pgsql-hackers
On 2022-Jul-21, Thomas Munro wrote:
> On Thu, Jul 21, 2022 at 11:01 PM Alvaro Herrera <[email protected]> wrote:
> > I've got no opinions on this. I don't like either 1 or 3, so I'm going
> > to add and backpatch a new GUC allow_recovery_tablespaces as the
> > override mechanism.
> >
> > If others disagree with this choice, please speak up.
>
> Would it help if we back-patched the allow_in_place_tablespaces stuff?
> I'm not sure how hard/destabilising that would be, but I could take a
> look tomorrow.
Yeah, I think that would reduce cruft. I'm not sure this is more
against backpatching policy or less, compared to adding a separate
GUC just for this bugfix.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"The problem with the facetime model is not just that it's demoralizing, but
that the people pretending to work interrupt the ones actually working."
(Paul Graham)
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 15:25 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:01 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-07-21 11:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2022-07-21 11:25 ` Alvaro Herrera <[email protected]>
2022-07-22 01:02 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Alvaro Herrera @ 2022-07-21 11:25 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; Dilip Kumar <[email protected]>; Robert Haas <[email protected]>; Michael Paquier <[email protected]>; Julien Rouhaud <[email protected]>; pgsql-hackers
On 2022-Jul-21, Alvaro Herrera wrote:
> Yeah, I think that would reduce cruft. I'm not sure this is more
> against backpatching policy or less, compared to adding a separate
> GUC just for this bugfix.
cruft:
{
{"allow_recovery_tablespaces", PG_POSTMASTER, WAL_RECOVERY,
gettext_noop("Continues recovery after finding invalid database directories."),
gettext_noop("It is possible for tablespace drop to interfere with database creation "
"so that WAL replay is forced to create fake database directories. "
"These should have been dropped by the time recovery ends; "
"but in case they aren't, this option lets recovery continue if they "
"are present. Note that these directories must be removed manually afterwards."),
GUC_NOT_IN_SAMPLE
},
&allow_recovery_tablespaces,
false,
NULL, NULL, NULL
},
This is not a very good explanation, but I don't know how to make it
better.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"I think my standards have lowered enough that now I think 'good design'
is when the page doesn't irritate the living f*ck out of me." (JWZ)
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 15:25 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:01 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-07-21 11:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:25 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2022-07-22 01:02 ` Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 109+ messages in thread
From: Kyotaro Horiguchi @ 2022-07-22 01:02 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
At Thu, 21 Jul 2022 13:25:05 +0200, Alvaro Herrera <[email protected]> wrote in
> On 2022-Jul-21, Alvaro Herrera wrote:
>
> > Yeah, I think that would reduce cruft. I'm not sure this is more
> > against backpatching policy or less, compared to adding a separate
> > GUC just for this bugfix.
>
> cruft:
>
> {
> {"allow_recovery_tablespaces", PG_POSTMASTER, WAL_RECOVERY,
> gettext_noop("Continues recovery after finding invalid database directories."),
> gettext_noop("It is possible for tablespace drop to interfere with database creation "
> "so that WAL replay is forced to create fake database directories. "
> "These should have been dropped by the time recovery ends; "
> "but in case they aren't, this option lets recovery continue if they "
> "are present. Note that these directories must be removed manually afterwards."),
> GUC_NOT_IN_SAMPLE
> },
> &allow_recovery_tablespaces,
> false,
> NULL, NULL, NULL
> },
>
> This is not a very good explanation, but I don't know how to make it
> better.
It looks a bit too detailed. I crafted the following..
Recovery can create tentative in-place tablespace directories under
pg_tblspc/. They are assumed to be removed until reaching recovery
consistency, but otherwise PostgreSQL raises a PANIC-level error,
aborting the recovery. Setting allow_recovery_tablespaces to true
causes the system to allow such directories during normal
operation. In case those directories are left after reaching
consistency, that implies data loss and metadata inconsistency and may
cause failure of future tablespace creation.
Though, after writing this, I became to think that piggy-backing on
allow_in_place_tablespaces might be a bit different..
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 15:25 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:01 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
@ 2022-07-22 00:20 ` Kyotaro Horiguchi <[email protected]>
2022-07-22 08:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
1 sibling, 1 reply; 109+ messages in thread
From: Kyotaro Horiguchi @ 2022-07-22 00:20 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
At Thu, 21 Jul 2022 23:14:57 +1200, Thomas Munro <[email protected]> wrote in
> On Thu, Jul 21, 2022 at 11:01 PM Alvaro Herrera <[email protected]> wrote:
> > On 2022-Jul-20, Alvaro Herrera wrote:
> > > I see the following alternatives:
> > >
> > > 1. not backpatch this fix to 14 and older
> > > 2. use a different GUC; either allow_invalid_pages as previously
> > > suggested, or create a new one just for this purpose
> > > 3. not provide any overriding mechanism in versions 14 and older
> >
> > I've got no opinions on this. I don't like either 1 or 3, so I'm going
> > to add and backpatch a new GUC allow_recovery_tablespaces as the
> > override mechanism.
> >
> > If others disagree with this choice, please speak up.
>
> Would it help if we back-patched the allow_in_place_tablespaces stuff?
> I'm not sure how hard/destabilising that would be, but I could take a
> look tomorrow.
+1. Addiotional reason for me is it is a developer option.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 15:25 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:01 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-07-22 00:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
@ 2022-07-22 08:18 ` Alvaro Herrera <[email protected]>
2022-07-22 08:49 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-07-22 08:53 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
0 siblings, 2 replies; 109+ messages in thread
From: Alvaro Herrera @ 2022-07-22 08:18 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
On 2022-Jul-22, Kyotaro Horiguchi wrote:
> At Thu, 21 Jul 2022 23:14:57 +1200, Thomas Munro <[email protected]> wrote in
> > Would it help if we back-patched the allow_in_place_tablespaces stuff?
> > I'm not sure how hard/destabilising that would be, but I could take a
> > look tomorrow.
>
> +1. Addiotional reason for me is it is a developer option.
OK, I'll wait for allow_in_place_tablespaces to be backpatched then.
I would like to get this fix pushed before the next set of minors, so if
you won't have time for the backpatches early enough, maybe I can work
on getting it done.
Which commits would we consider?
7170f2159fb2 Allow "in place" tablespaces.
f6f0db4d6240 Fix pg_tablespace_location() with in-place tablespaces
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"Most hackers will be perfectly comfortable conceptualizing users as entropy
sources, so let's move on." (Nathaniel Smith)
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 15:25 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:01 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-07-22 00:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-07-22 08:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2022-07-22 08:49 ` Kyotaro Horiguchi <[email protected]>
2022-07-27 06:07 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
1 sibling, 1 reply; 109+ messages in thread
From: Kyotaro Horiguchi @ 2022-07-22 08:49 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
At Fri, 22 Jul 2022 10:18:58 +0200, Alvaro Herrera <[email protected]> wrote in
> OK, I'll wait for allow_in_place_tablespaces to be backpatched then.
>
> I would like to get this fix pushed before the next set of minors, so if
> you won't have time for the backpatches early enough, maybe I can work
> on getting it done.
>
> Which commits would we consider?
>
> 7170f2159fb2 Allow "in place" tablespaces.
> f6f0db4d6240 Fix pg_tablespace_location() with in-place tablespaces
The second one is just to make the function work with in-place
tablespaces. Without it the function yeilds the following error.
> ERROR: could not read symbolic link "pg_tblspc/16407": Invalid argument
This looks actually odd but I think no need of back-patching because
there's no actual user of the feature is not seen in our test suite.
If we have a test that needs the feature in future, it would be enough
to back-patch it then.
So I think only the first one is needed for now.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 15:25 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:01 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-07-22 00:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-07-22 08:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-22 08:49 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
@ 2022-07-27 06:07 ` Alvaro Herrera <[email protected]>
0 siblings, 0 replies; 109+ messages in thread
From: Alvaro Herrera @ 2022-07-27 06:07 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
On 2022-Jul-22, Kyotaro Horiguchi wrote:
> At Fri, 22 Jul 2022 10:18:58 +0200, Alvaro Herrera <[email protected]> wrote in
> > Which commits would we consider?
> >
> > 7170f2159fb2 Allow "in place" tablespaces.
> > f6f0db4d6240 Fix pg_tablespace_location() with in-place tablespaces
>
> The second one is just to make the function work with in-place
> tablespaces. Without it the function yeilds the following error.
>
> > ERROR: could not read symbolic link "pg_tblspc/16407": Invalid argument
>
> This looks actually odd but I think no need of back-patching because
> there's no actual user of the feature is not seen in our test suite.
> If we have a test that needs the feature in future, it would be enough
> to back-patch it then.
Actually, I found that the new test added by the fix in this thread does
depend on this being fixed, so I included an even larger set, which I
think makes this more complete:
7170f2159fb2 Allow "in place" tablespaces.
c6f2f01611d4 Fix pg_basebackup with in-place tablespaces.
f6f0db4d6240 Fix pg_tablespace_location() with in-place tablespaces
7a7cd84893e0 doc: Remove mention to in-place tablespaces for pg_tablespace_location()
5344723755bd Remove unnecessary Windows-specific basebackup code.
I didn't include any of the test changes for now. I don't intend to do
so, unless we see another reason for that; I think the new tests that
are going to be added by the recovery bugfix should be sufficient
coverage.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"La fuerza no está en los medios físicos
sino que reside en una voluntad indomable" (Gandhi)
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 15:25 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:01 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:14 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-07-22 00:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-07-22 08:18 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2022-07-22 08:53 ` Thomas Munro <[email protected]>
1 sibling, 0 replies; 109+ messages in thread
From: Thomas Munro @ 2022-07-22 08:53 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; Dilip Kumar <[email protected]>; Robert Haas <[email protected]>; Michael Paquier <[email protected]>; Julien Rouhaud <[email protected]>; pgsql-hackers
On Fri, Jul 22, 2022 at 8:19 PM Alvaro Herrera <[email protected]> wrote:
> On 2022-Jul-22, Kyotaro Horiguchi wrote:
> > At Thu, 21 Jul 2022 23:14:57 +1200, Thomas Munro <[email protected]> wrote in
> > > Would it help if we back-patched the allow_in_place_tablespaces stuff?
> > > I'm not sure how hard/destabilising that would be, but I could take a
> > > look tomorrow.
> >
> > +1. Addiotional reason for me is it is a developer option.
>
> OK, I'll wait for allow_in_place_tablespaces to be backpatched then.
>
> I would like to get this fix pushed before the next set of minors, so if
> you won't have time for the backpatches early enough, maybe I can work
> on getting it done.
>
> Which commits would we consider?
I wonder how crazy it would be to back-patch
src/test/recovery/t/027_stream_regress.pl too.
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2022-07-21 11:11 ` Thomas Munro <[email protected]>
2022-07-21 11:17 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2 siblings, 1 reply; 109+ messages in thread
From: Thomas Munro @ 2022-07-21 11:11 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; Dilip Kumar <[email protected]>; Robert Haas <[email protected]>; Michael Paquier <[email protected]>; Julien Rouhaud <[email protected]>; pgsql-hackers
On Wed, Jul 20, 2022 at 10:51 PM Alvaro Herrera <[email protected]> wrote:
> v26 here. I spent some time fighting the readdir() stuff for
> Windows (so that get_dirent_type returns LNK for junction points)
> but couldn't make it to work and was unable to figure out why.
Was it because of this?
https://www.postgresql.org/message-id/CA%2BhUKGKv%2B736Pc8kSj3%3DDijDGd1eC79-uT3Vi16n7jYkcc_raw%40ma...
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-21 11:11 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
@ 2022-07-21 11:17 ` Alvaro Herrera <[email protected]>
0 siblings, 0 replies; 109+ messages in thread
From: Alvaro Herrera @ 2022-07-21 11:17 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; Dilip Kumar <[email protected]>; Robert Haas <[email protected]>; Michael Paquier <[email protected]>; Julien Rouhaud <[email protected]>; pgsql-hackers
On 2022-Jul-21, Thomas Munro wrote:
> On Wed, Jul 20, 2022 at 10:51 PM Alvaro Herrera <[email protected]> wrote:
> > v26 here. I spent some time fighting the readdir() stuff for
> > Windows (so that get_dirent_type returns LNK for junction points)
> > but couldn't make it to work and was unable to figure out why.
>
> Was it because of this?
>
> https://www.postgresql.org/message-id/CA%2BhUKGKv%2B736Pc8kSj3%3DDijDGd1eC79-uT3Vi16n7jYkcc_raw%40ma...
Oh, that sounds very likely, yeah. I didn't think of testing the
FILE_ATTRIBUTE_DIRECTORY bit for junction points.
I +1 pushing both of these patches to 14. Then this patch becomes a
couple of lines shorter.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Before you were born your parents weren't as boring as they are now. They
got that way paying your bills, cleaning up your room and listening to you
tell them how idealistic you are." -- Charles J. Sykes' advice to teenagers
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2022-07-27 18:54 ` Alvaro Herrera <[email protected]>
2022-07-28 18:04 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Matthias van de Meent <[email protected]>
2 siblings, 1 reply; 109+ messages in thread
From: Alvaro Herrera @ 2022-07-27 18:54 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers
Okay, I think I'm done with this. Here's v27 for the master branch,
where I fixed some comments as well as thinkos in the test script.
The ones on older branches aren't materially different, they just have
tonnes of conflicts resolved. I'll get this pushed tomorrow morning.
I have run it through CI and it seems ... not completely broken, at
least, but I have no working recipes for Windows on branches 14 and
older, so it doesn't really work fully. If anybody does, please share.
You can see mine here
https://github.com/alvherre/postgres/commits/REL_11_STABLE [etc]
https://cirrus-ci.com/build/5320904228995072
https://cirrus-ci.com/github/alvherre/postgres
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Every machine is a smoke machine if you operate it wrong enough."
https://twitter.com/libseybieda/status/1541673325781196801
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-27 18:54 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2022-07-28 18:04 ` Matthias van de Meent <[email protected]>
2022-07-28 21:57 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Matthias van de Meent @ 2022-07-28 18:04 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: pgsql-hackers
On Wed, 27 Jul 2022 at 20:55, Alvaro Herrera <[email protected]> wrote:
>
> Okay, I think I'm done with this. Here's v27 for the master branch,
> where I fixed some comments as well as thinkos in the test script.
> The ones on older branches aren't materially different, they just have
> tonnes of conflicts resolved. I'll get this pushed tomorrow morning.
>
> I have run it through CI and it seems ... not completely broken, at
> least, but I have no working recipes for Windows on branches 14 and
> older, so it doesn't really work fully. If anybody does, please share.
> You can see mine here
> https://github.com/alvherre/postgres/commits/REL_11_STABLE [etc]
> https://cirrus-ci.com/build/5320904228995072
> https://cirrus-ci.com/github/alvherre/postgres
I'd like to bring to your attention that the test that was introduced
with 9e4f914b seem to be flaky in FreeBSD 13 in the CFBot builds: it
sometimes times out while waiting for the secondary to catch up. Or,
at least I think it does, and I'm not too familiar with TAP failure
outputs: it returns with error code 29 and logs that I'd expect when
the timeout is reached.
See bottom for examples (all 3 builds for different patches).
Kind regards,
Matthias van de Meent.
[1] https://cirrus-ci.com/task/4960990331666432?logs=test_world#L2631-L2662
[2] https://cirrus-ci.com/task/5012678384025600?logs=test_world#L2631-L2662
[3] https://cirrus-ci.com/task/5147001137397760?logs=test_world#L2631-L2662
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-27 18:54 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-28 18:04 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Matthias van de Meent <[email protected]>
@ 2022-07-28 21:57 ` Tom Lane <[email protected]>
2022-07-28 23:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Tom Lane @ 2022-07-28 21:57 UTC (permalink / raw)
To: Matthias van de Meent <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; pgsql-hackers
Matthias van de Meent <[email protected]> writes:
> I'd like to bring to your attention that the test that was introduced
> with 9e4f914b seem to be flaky in FreeBSD 13 in the CFBot builds: it
> sometimes times out while waiting for the secondary to catch up. Or,
> at least I think it does, and I'm not too familiar with TAP failure
> outputs: it returns with error code 29 and logs that I'd expect when
> the timeout is reached.
It's also failing in the buildfarm, eg
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=conchuela&dt=2022-07-28%2020%3A57%3A50
Looks like only conchuela so far, reinforcing the idea that we're
only seeing it on FreeBSD. I'd tentatively bet on a timing problem
that requires some FreeBSD scheduling quirk to manifest; we've seen
such quirks before.
regards, tom lane
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-27 18:54 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-28 18:04 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Matthias van de Meent <[email protected]>
2022-07-28 21:57 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
@ 2022-07-28 23:27 ` Thomas Munro <[email protected]>
2022-07-29 05:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Thomas Munro @ 2022-07-28 23:27 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>; pgsql-hackers
On Fri, Jul 29, 2022 at 9:57 AM Tom Lane <[email protected]> wrote:
> Matthias van de Meent <[email protected]> writes:
> > I'd like to bring to your attention that the test that was introduced
> > with 9e4f914b seem to be flaky in FreeBSD 13 in the CFBot builds: it
> > sometimes times out while waiting for the secondary to catch up. Or,
> > at least I think it does, and I'm not too familiar with TAP failure
> > outputs: it returns with error code 29 and logs that I'd expect when
> > the timeout is reached.
>
> It's also failing in the buildfarm, eg
>
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=conchuela&dt=2022-07-28%2020%3A57%3A50
>
> Looks like only conchuela so far, reinforcing the idea that we're
> only seeing it on FreeBSD. I'd tentatively bet on a timing problem
> that requires some FreeBSD scheduling quirk to manifest; we've seen
> such quirks before.
Maybe it just needs a replication slot? I see:
ERROR: requested WAL segment 000000010000000000000003 has already been removed
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-27 18:54 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-28 18:04 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Matthias van de Meent <[email protected]>
2022-07-28 21:57 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2022-07-28 23:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
@ 2022-07-29 05:20 ` Kyotaro Horiguchi <[email protected]>
2022-07-29 10:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Kyotaro Horiguchi @ 2022-07-29 05:20 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers
At Fri, 29 Jul 2022 11:27:01 +1200, Thomas Munro <[email protected]> wrote in
> Maybe it just needs a replication slot? I see:
>
> ERROR: requested WAL segment 000000010000000000000003 has already been removed
Agreed, I see the same. The same failure can be surely reproducible
by inserting wal-switch+checkpoint after taking backup [1]. And it is
fixed by the attached.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
[1]:
--- a/src/test/recovery/t/033_replay_tsp_drops.pl
+++ b/src/test/recovery/t/033_replay_tsp_drops.pl
@@ -30,6 +30,13 @@ sub test_tablespace
my $backup_name = 'my_backup';
$node_primary->backup($backup_name);
+ $node_primary->psql(
+ 'postgres',
+ qq[
+ CREATE TABLE t(); DROP TABLE t; SELECT pg_switch_wal();
+ CHECKPOINT;
+ ]);
+
my $node_standby = PostgreSQL::Test::Cluster->new("standby2_$strategy");
$node_standby->init_from_backup($node_primary, $backup_name,
has_streaming => 1);
Attachments:
[text/x-patch] fix_tsp_drop_test_error.diff (516B, ../../[email protected]/2-fix_tsp_drop_test_error.diff)
download | inline diff:
diff --git a/src/test/recovery/t/033_replay_tsp_drops.pl b/src/test/recovery/t/033_replay_tsp_drops.pl
index 9b74cb09ac..0756ca6c87 100644
--- a/src/test/recovery/t/033_replay_tsp_drops.pl
+++ b/src/test/recovery/t/033_replay_tsp_drops.pl
@@ -20,6 +20,7 @@ sub test_tablespace
$node_primary->psql(
'postgres',
qq[
+ SELECT pg_create_physical_replication_slot('slot1', true);
SET allow_in_place_tablespaces=on;
CREATE TABLESPACE dropme_ts1 LOCATION '';
CREATE TABLESPACE dropme_ts2 LOCATION '';
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-27 18:54 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-28 18:04 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Matthias van de Meent <[email protected]>
2022-07-28 21:57 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2022-07-28 23:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-07-29 05:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
@ 2022-07-29 10:59 ` Alvaro Herrera <[email protected]>
2022-07-30 14:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Alvaro Herrera @ 2022-07-29 10:59 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; pgsql-hackers
On 2022-Jul-29, Kyotaro Horiguchi wrote:
> At Fri, 29 Jul 2022 11:27:01 +1200, Thomas Munro <[email protected]> wrote in
> > Maybe it just needs a replication slot? I see:
> >
> > ERROR: requested WAL segment 000000010000000000000003 has already been removed
>
> Agreed, I see the same. The same failure can be surely reproducible
> by inserting wal-switch+checkpoint after taking backup [1]. And it is
> fixed by the attached.
WFM, pushed that way. I added a slot drop after the pg_stat_replication
count check to be a little less intrusive. Thanks Matthias for
reporting. (Note that the Cirrus page has a download link for the
complete logs as artifacts).
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"I'm always right, but sometimes I'm more right than other times."
(Linus Torvalds)
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-27 18:54 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-28 18:04 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Matthias van de Meent <[email protected]>
2022-07-28 21:57 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2022-07-28 23:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-07-29 05:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-07-29 10:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
@ 2022-07-30 14:37 ` Tom Lane <[email protected]>
2022-07-30 16:51 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2022-07-31 03:46 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-08-01 02:01 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Andres Freund <[email protected]>
0 siblings, 3 replies; 109+ messages in thread
From: Tom Lane @ 2022-07-30 14:37 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; pgsql-hackers
Alvaro Herrera <[email protected]> writes:
> WFM, pushed that way.
Looks like conchuela is still intermittently unhappy.
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=conchuela&dt=2022-07-30%2004%3A57%3A51
regards, tom lane
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-27 18:54 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-28 18:04 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Matthias van de Meent <[email protected]>
2022-07-28 21:57 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2022-07-28 23:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-07-29 05:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-07-29 10:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-30 14:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
@ 2022-07-30 16:51 ` Tom Lane <[email protected]>
2022-07-30 23:08 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-08-05 20:29 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2 siblings, 2 replies; 109+ messages in thread
From: Tom Lane @ 2022-07-30 16:51 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; pgsql-hackers
I wrote:
> Looks like conchuela is still intermittently unhappy.
BTW, quite aside from stability, is it really necessary for this test to
be so freakin' slow? florican for instance reports
[12:43:38] t/025_stuck_on_old_timeline.pl ....... ok 49010 ms ( 0.00 usr 0.00 sys + 3.64 cusr 2.49 csys = 6.13 CPU)
[12:44:12] t/026_overwrite_contrecord.pl ........ ok 34751 ms ( 0.01 usr 0.00 sys + 3.14 cusr 1.76 csys = 4.91 CPU)
[12:49:00] t/027_stream_regress.pl .............. ok 287278 ms ( 0.00 usr 0.00 sys + 9.66 cusr 6.95 csys = 16.60 CPU)
[12:50:04] t/028_pitr_timelines.pl .............. ok 64543 ms ( 0.00 usr 0.00 sys + 3.59 cusr 3.20 csys = 6.78 CPU)
[12:50:17] t/029_stats_restart.pl ............... ok 12505 ms ( 0.02 usr 0.00 sys + 3.16 cusr 1.40 csys = 4.57 CPU)
[12:50:51] t/030_stats_cleanup_replica.pl ....... ok 33933 ms ( 0.01 usr 0.01 sys + 3.55 cusr 2.46 csys = 6.03 CPU)
[12:51:25] t/031_recovery_conflict.pl ........... ok 34249 ms ( 0.00 usr 0.00 sys + 3.37 cusr 2.20 csys = 5.57 CPU)
[12:52:09] t/032_relfilenode_reuse.pl ........... ok 44274 ms ( 0.01 usr 0.00 sys + 3.21 cusr 2.05 csys = 5.27 CPU)
[12:54:07] t/033_replay_tsp_drops.pl ............ ok 117840 ms ( 0.01 usr 0.00 sys + 8.72 cusr 5.41 csys = 14.14 CPU)
027 is so bloated because it runs the core regression tests YA time,
which I'm not very happy about either; but that's no excuse for
every new test to contribute an additional couple of minutes.
regards, tom lane
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-27 18:54 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-28 18:04 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Matthias van de Meent <[email protected]>
2022-07-28 21:57 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2022-07-28 23:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-07-29 05:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-07-29 10:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-30 14:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2022-07-30 16:51 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
@ 2022-07-30 23:08 ` Thomas Munro <[email protected]>
2022-07-30 23:17 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
1 sibling, 1 reply; 109+ messages in thread
From: Thomas Munro @ 2022-07-30 23:08 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Kyotaro Horiguchi <[email protected]>; Matthias van de Meent <[email protected]>; pgsql-hackers
On Sun, Jul 31, 2022 at 4:51 AM Tom Lane <[email protected]> wrote:
> BTW, quite aside from stability, is it really necessary for this test to
> be so freakin' slow? florican for instance reports
>
> [12:43:38] t/025_stuck_on_old_timeline.pl ....... ok 49010 ms ( 0.00 usr 0.00 sys + 3.64 cusr 2.49 csys = 6.13 CPU)
> [12:44:12] t/026_overwrite_contrecord.pl ........ ok 34751 ms ( 0.01 usr 0.00 sys + 3.14 cusr 1.76 csys = 4.91 CPU)
> [12:49:00] t/027_stream_regress.pl .............. ok 287278 ms ( 0.00 usr 0.00 sys + 9.66 cusr 6.95 csys = 16.60 CPU)
> [12:50:04] t/028_pitr_timelines.pl .............. ok 64543 ms ( 0.00 usr 0.00 sys + 3.59 cusr 3.20 csys = 6.78 CPU)
> [12:50:17] t/029_stats_restart.pl ............... ok 12505 ms ( 0.02 usr 0.00 sys + 3.16 cusr 1.40 csys = 4.57 CPU)
> [12:50:51] t/030_stats_cleanup_replica.pl ....... ok 33933 ms ( 0.01 usr 0.01 sys + 3.55 cusr 2.46 csys = 6.03 CPU)
> [12:51:25] t/031_recovery_conflict.pl ........... ok 34249 ms ( 0.00 usr 0.00 sys + 3.37 cusr 2.20 csys = 5.57 CPU)
> [12:52:09] t/032_relfilenode_reuse.pl ........... ok 44274 ms ( 0.01 usr 0.00 sys + 3.21 cusr 2.05 csys = 5.27 CPU)
> [12:54:07] t/033_replay_tsp_drops.pl ............ ok 117840 ms ( 0.01 usr 0.00 sys + 8.72 cusr 5.41 csys = 14.14 CPU)
>
> 027 is so bloated because it runs the core regression tests YA time,
> which I'm not very happy about either; but that's no excuse for
> every new test to contribute an additional couple of minutes.
Complaints about 027 noted, I'm thinking about what we could do about that.
As for 033, I worried that it might be the new ProcSignalBarrier stuff
around tablespaces, but thankfully the DEBUG logging I added there
recently shows those all completing in single digit milliseconds. I
also confirmed there are no unexpected fsync'd being produced here.
That is quite a lot of CPU, but it's a huge amount of total runtime.
It runs in 5-8 seconds on various modern systems, 19 seconds on my
Linux RPi4, and 50 seconds on my Celeron-powered NAS box with spinning
disks.
I noticed this is a 32 bit FBSD system. Is it running on UFS, perhaps
on slow storage? Are soft updates enabled (visible as options in
output of "mount")? Without soft updates, a lot more file system ops
perform synchronous I/O, which really slows down our tests. In
general, UFS isn't as good as modern file systems at avoiding I/O for
short-lived files, and we set up and tear down a lot of them in our
testing. Another thing that makes a difference is to use a filesystem
with 8KB block size. This has been a subject of investigation for
speeding up CI (see src/tools/ci/gcp_freebsd_repartition.sh), but
several mysteries remain unsolved...
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-27 18:54 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-28 18:04 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Matthias van de Meent <[email protected]>
2022-07-28 21:57 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2022-07-28 23:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-07-29 05:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-07-29 10:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-30 14:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2022-07-30 16:51 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2022-07-30 23:08 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
@ 2022-07-30 23:17 ` Tom Lane <[email protected]>
2022-08-04 02:54 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
0 siblings, 1 reply; 109+ messages in thread
From: Tom Lane @ 2022-07-30 23:17 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Kyotaro Horiguchi <[email protected]>; Matthias van de Meent <[email protected]>; pgsql-hackers
Thomas Munro <[email protected]> writes:
> I noticed this is a 32 bit FBSD system. Is it running on UFS, perhaps
> on slow storage? Are soft updates enabled (visible as options in
> output of "mount")?
It's an ancient (2006) mac mini with 5400RPM spinning rust.
"mount" says
/dev/ada0s2a on / (ufs, local, soft-updates, journaled soft-updates)
devfs on /dev (devfs)
regards, tom lane
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-27 18:54 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-28 18:04 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Matthias van de Meent <[email protected]>
2022-07-28 21:57 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2022-07-28 23:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-07-29 05:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-07-29 10:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-30 14:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2022-07-30 16:51 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2022-07-30 23:08 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-07-30 23:17 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
@ 2022-08-04 02:54 ` Thomas Munro <[email protected]>
0 siblings, 0 replies; 109+ messages in thread
From: Thomas Munro @ 2022-08-04 02:54 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Kyotaro Horiguchi <[email protected]>; Matthias van de Meent <[email protected]>; pgsql-hackers
On Sun, Jul 31, 2022 at 11:17 AM Tom Lane <[email protected]> wrote:
> Thomas Munro <[email protected]> writes:
> > I noticed this is a 32 bit FBSD system. Is it running on UFS, perhaps
> > on slow storage? Are soft updates enabled (visible as options in
> > output of "mount")?
>
> It's an ancient (2006) mac mini with 5400RPM spinning rust.
> "mount" says
>
> /dev/ada0s2a on / (ufs, local, soft-updates, journaled soft-updates)
> devfs on /dev (devfs)
I don't have all the details and I may be way off here but I have the
impression that when you create and then unlink trees of files
quickly, sometimes soft-updates are flushed synchronously, which turns
into many 5400 RPM seeks; dtrace could be used to check, but some
clues in your numbers would be some kind of correlation between time
and number of clusters that are set up and torn down by each test.
Without soft-updates, it'd be much worse, because then many more
things become synchronous I/O. Even with write caching enabled,
soft-updates flush the drive cache when there's a barrier needed for
crash safety. It may also be that there is something strange about
Apple hardware that makes it extra slow at full-cache-flush operations
(cf unexplainable excess slowness of F_FULLFSYNC under macOS including
old spinning rust systems and current flash systems, and complaints
about this general area on current Apple hardware from the Asahi
Linux/M1 port people, though how relevant that is to 2006 spinning
rust I dunno). It would be nice to look into how to tune, fix or work
around all of that, as it also affects CI which has a IO limits
(though admittedly a couple of orders of mag higher IOPS than 5400
RPM).
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-27 18:54 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-28 18:04 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Matthias van de Meent <[email protected]>
2022-07-28 21:57 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2022-07-28 23:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-07-29 05:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-07-29 10:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-30 14:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2022-07-30 16:51 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
@ 2022-08-05 20:29 ` Alvaro Herrera <[email protected]>
1 sibling, 0 replies; 109+ messages in thread
From: Alvaro Herrera @ 2022-08-05 20:29 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; pgsql-hackers
On 2022-Jul-30, Tom Lane wrote:
> BTW, quite aside from stability, is it really necessary for this test to
> be so freakin' slow? florican for instance reports
>
> [12:54:07] t/033_replay_tsp_drops.pl ............ ok 117840 ms ( 0.01 usr 0.00 sys + 8.72 cusr 5.41 csys = 14.14 CPU)
>
> 027 is so bloated because it runs the core regression tests YA time,
> which I'm not very happy about either; but that's no excuse for
> every new test to contribute an additional couple of minutes.
Definitely not intended. It looks like the reason is just that the DROP
DATABASE/TABLESPACE commands are super slow, and this test does a lot of
that. I added some instrumentation and the largest fraction of time
goes to execute this
CREATE DATABASE dropme_db1 WITH TABLESPACE dropme_ts1;
CREATE TABLE t (a int) TABLESPACE dropme_ts2;
CREATE DATABASE dropme_db2 WITH TABLESPACE dropme_ts2;
CREATE DATABASE moveme_db TABLESPACE source_ts;
ALTER DATABASE moveme_db SET TABLESPACE target_ts;
CREATE DATABASE newdb TEMPLATE template_db;
ALTER DATABASE template_db IS_TEMPLATE = false;
DROP DATABASE dropme_db1;
DROP TABLE t;
DROP DATABASE dropme_db2;
DROP TABLESPACE dropme_ts2;
DROP TABLESPACE source_ts;
DROP DATABASE template_db;
Maybe this is overkill and we can reduce the test without damaging the
coverage. I'll have a look during the weekend.
I'll repair the reliability problem too, separately.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"This is a foot just waiting to be shot" (Andrew Dunstan)
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-27 18:54 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-28 18:04 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Matthias van de Meent <[email protected]>
2022-07-28 21:57 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2022-07-28 23:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-07-29 05:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-07-29 10:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-30 14:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
@ 2022-07-31 03:46 ` Thomas Munro <[email protected]>
2022-08-02 19:58 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2 siblings, 1 reply; 109+ messages in thread
From: Thomas Munro @ 2022-07-31 03:46 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Kyotaro Horiguchi <[email protected]>; Matthias van de Meent <[email protected]>; pgsql-hackers
On Sun, Jul 31, 2022 at 2:37 AM Tom Lane <[email protected]> wrote:
> Alvaro Herrera <[email protected]> writes:
> > WFM, pushed that way.
>
> Looks like conchuela is still intermittently unhappy.
>
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=conchuela&dt=2022-07-30%2004%3A57%3A51
And here's one from CI that failed on Linux (this was a cfbot run with
an unrelated patch, parent commit b998196 so a few commits after "Fix
test instability"):
https://cirrus-ci.com/task/5282155000496128
https://api.cirrus-ci.com/v1/artifact/task/5282155000496128/log/src/test/recovery/tmp_check/log/033_...
It looks like this sequence is racy and we need to wait for more than
just "connection is made" before dropping the slot?
$node_standby->start;
# Make sure connection is made
$node_primary->poll_query_until('postgres',
'SELECT count(*) = 1 FROM pg_stat_replication');
$node_primary->safe_psql('postgres', "SELECT
pg_drop_replication_slot('slot')");
Why not set the replication slot name so that the standby uses it
"properly", like in other tests?
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-27 18:54 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-28 18:04 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Matthias van de Meent <[email protected]>
2022-07-28 21:57 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2022-07-28 23:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-07-29 05:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-07-29 10:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-30 14:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2022-07-31 03:46 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
@ 2022-08-02 19:58 ` Thomas Munro <[email protected]>
0 siblings, 0 replies; 109+ messages in thread
From: Thomas Munro @ 2022-08-02 19:58 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Kyotaro Horiguchi <[email protected]>; Matthias van de Meent <[email protected]>; pgsql-hackers
On Sun, Jul 31, 2022 at 3:46 PM Thomas Munro <[email protected]> wrote:
> On Sun, Jul 31, 2022 at 2:37 AM Tom Lane <[email protected]> wrote:
> > Alvaro Herrera <[email protected]> writes:
> > > WFM, pushed that way.
> >
> > Looks like conchuela is still intermittently unhappy.
> >
> > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=conchuela&dt=2022-07-30%2004%3A57%3A51
>
> And here's one from CI that failed on Linux (this was a cfbot run with
> an unrelated patch, parent commit b998196 so a few commits after "Fix
> test instability"):
>
> https://cirrus-ci.com/task/5282155000496128
>
> https://api.cirrus-ci.com/v1/artifact/task/5282155000496128/log/src/test/recovery/tmp_check/log/033_...
>
> It looks like this sequence is racy and we need to wait for more than
> just "connection is made" before dropping the slot?
>
> $node_standby->start;
>
> # Make sure connection is made
> $node_primary->poll_query_until('postgres',
> 'SELECT count(*) = 1 FROM pg_stat_replication');
> $node_primary->safe_psql('postgres', "SELECT
> pg_drop_replication_slot('slot')");
>
> Why not set the replication slot name so that the standby uses it
> "properly", like in other tests?
Or to keep doing it this way, does that pg_stat_replication query need
a WHERE clause looking at the state?
^ permalink raw reply [nested|flat] 109+ messages in thread
* Re: standby recovery fails (tablespace related) (tentative patch and discussion)
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-27 18:54 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-28 18:04 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Matthias van de Meent <[email protected]>
2022-07-28 21:57 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
2022-07-28 23:27 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Thomas Munro <[email protected]>
2022-07-29 05:20 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Kyotaro Horiguchi <[email protected]>
2022-07-29 10:59 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-30 14:37 ` Re: standby recovery fails (tablespace related) (tentative patch and discussion) Tom Lane <[email protected]>
@ 2022-08-01 02:01 ` Andres Freund <[email protected]>
2 siblings, 0 replies; 109+ messages in thread
From: Andres Freund @ 2022-08-01 02:01 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; pgsql-hackers
Hi,
On 2022-07-30 10:37:55 -0400, Tom Lane wrote:
> Alvaro Herrera <[email protected]> writes:
> > WFM, pushed that way.
>
> Looks like conchuela is still intermittently unhappy.
>
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=conchuela&dt=2022-07-30%2004%3A57%3A51
CI as well:
https://cirrus-ci.com/task/5295464063959040?logs=test_world#L2671
https://cirrus-ci.com/task/5042590885085184?logs=test_world#L2664
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 109+ messages in thread
end of thread, other threads:[~2022-08-05 20:29 UTC | newest]
Thread overview: 109+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2019-04-17 07:56 standby recovery fails (tablespace related) (tentative patch and discussion) Paul Guo <[email protected]>
2019-04-19 04:38 ` Asim R P <[email protected]>
2019-04-22 04:36 ` Paul Guo <[email protected]>
2019-04-22 07:15 ` Kyotaro HORIGUCHI <[email protected]>
2019-04-22 07:40 ` Kyotaro HORIGUCHI <[email protected]>
2019-04-22 12:19 ` Kyotaro HORIGUCHI <[email protected]>
2019-04-23 02:34 ` Michael Paquier <[email protected]>
2019-04-23 05:02 ` Kyotaro HORIGUCHI <[email protected]>
2019-04-23 05:31 ` Paul Guo <[email protected]>
2019-04-23 07:39 ` Kyotaro HORIGUCHI <[email protected]>
2019-04-24 08:13 ` Kyotaro HORIGUCHI <[email protected]>
2019-04-28 07:33 ` Paul Guo <[email protected]>
2019-04-30 06:33 ` Paul Guo <[email protected]>
2019-05-07 06:47 ` Kyotaro HORIGUCHI <[email protected]>
2019-05-13 09:37 ` Paul Guo <[email protected]>
2019-05-14 03:06 ` Kyotaro HORIGUCHI <[email protected]>
2019-05-27 13:39 ` Paul Guo <[email protected]>
2019-06-19 07:21 ` Paul Guo <[email protected]>
2019-07-08 03:15 ` Thomas Munro <[email protected]>
2019-07-15 10:52 ` Paul Guo <[email protected]>
2019-08-01 22:37 ` Thomas Munro <[email protected]>
2019-08-22 13:13 ` Paul Guo <[email protected]>
2019-08-22 16:13 ` Anastasia Lubennikova <[email protected]>
2019-09-03 15:58 ` Alvaro Herrera <[email protected]>
2019-09-05 07:12 ` Paul Guo <[email protected]>
2019-09-10 11:42 ` Asim R P <[email protected]>
2019-09-11 14:26 ` Anastasia Lubennikova <[email protected]>
2019-09-12 08:35 ` Kyotaro Horiguchi <[email protected]>
2019-09-12 12:02 ` Asim R P <[email protected]>
2019-09-19 11:59 ` Asim R P <[email protected]>
2019-09-20 12:23 ` Asim R P <[email protected]>
2019-12-26 14:45 ` Anastasia Lubennikova <[email protected]>
2020-01-10 00:22 ` Alvaro Herrera <[email protected]>
2020-01-10 13:43 ` Alvaro Herrera <[email protected]>
2020-01-13 10:27 ` Paul Guo <[email protected]>
2020-01-15 10:18 ` Paul Guo <[email protected]>
2020-01-27 15:24 ` Fujii Masao <[email protected]>
2020-03-25 05:52 ` Fujii Masao <[email protected]>
2020-07-07 21:12 ` Daniel Gustafsson <[email protected]>
2020-07-08 12:56 ` Paul Guo <[email protected]>
2021-01-05 01:07 ` Kyotaro Horiguchi <[email protected]>
2021-01-27 08:36 ` Paul Guo <[email protected]>
2021-03-27 14:23 ` Alvaro Herrera <[email protected]>
2021-03-30 07:12 ` Paul Guo <[email protected]>
2021-07-09 19:37 ` Ibrar Ahmed <[email protected]>
2021-08-05 10:20 ` Paul Guo <[email protected]>
2021-08-10 20:56 ` Robert Haas <[email protected]>
2021-08-11 07:58 ` Paul Guo <[email protected]>
2021-08-11 12:59 ` Robert Haas <[email protected]>
2021-09-24 18:14 ` Tom Lane <[email protected]>
2021-11-04 12:34 ` Daniel Gustafsson <[email protected]>
2021-11-08 08:55 ` Kyotaro Horiguchi <[email protected]>
2021-11-09 03:51 ` Michael Paquier <[email protected]>
2021-11-09 08:05 ` Kyotaro Horiguchi <[email protected]>
2021-11-10 08:14 ` Kyotaro Horiguchi <[email protected]>
2021-11-10 12:14 ` Alvaro Herrera <[email protected]>
2021-11-11 02:13 ` Kyotaro Horiguchi <[email protected]>
2021-11-12 07:43 ` Kyotaro Horiguchi <[email protected]>
2021-12-24 10:21 ` Kyotaro Horiguchi <[email protected]>
2022-03-25 12:26 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-03-28 01:01 ` Kyotaro Horiguchi <[email protected]>
2022-03-28 01:34 ` Thomas Munro <[email protected]>
2022-03-28 06:25 ` Kyotaro Horiguchi <[email protected]>
2022-03-29 01:57 ` Michael Paquier <[email protected]>
2022-03-28 08:20 ` Kyotaro Horiguchi <[email protected]>
2022-03-28 14:37 ` Robert Haas <[email protected]>
2022-03-29 01:34 ` Kyotaro Horiguchi <[email protected]>
2022-03-28 16:17 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Robert Haas <[email protected]>
2022-03-29 04:55 ` Kyotaro Horiguchi <[email protected]>
2022-03-29 11:37 ` Alvaro Herrera <[email protected]>
2022-03-29 12:45 ` Robert Haas <[email protected]>
2022-03-29 13:28 ` Alvaro Herrera <[email protected]>
2022-03-29 13:31 ` Robert Haas <[email protected]>
2022-07-13 16:43 Re: standby recovery fails (tablespace related) (tentative patch and discussion) Alvaro Herrera <[email protected]>
2022-07-14 21:47 ` Alvaro Herrera <[email protected]>
2022-07-15 07:30 ` Kyotaro Horiguchi <[email protected]>
2022-07-15 07:56 ` Alvaro Herrera <[email protected]>
2022-07-15 10:58 ` Alvaro Herrera <[email protected]>
2022-07-15 12:03 ` Alvaro Herrera <[email protected]>
2022-07-20 10:50 ` Alvaro Herrera <[email protected]>
2022-07-20 15:25 ` Alvaro Herrera <[email protected]>
2022-07-20 16:34 ` Alvaro Herrera <[email protected]>
2022-07-21 11:01 ` Alvaro Herrera <[email protected]>
2022-07-21 11:14 ` Thomas Munro <[email protected]>
2022-07-21 11:20 ` Alvaro Herrera <[email protected]>
2022-07-21 11:25 ` Alvaro Herrera <[email protected]>
2022-07-22 01:02 ` Kyotaro Horiguchi <[email protected]>
2022-07-22 00:20 ` Kyotaro Horiguchi <[email protected]>
2022-07-22 08:18 ` Alvaro Herrera <[email protected]>
2022-07-22 08:49 ` Kyotaro Horiguchi <[email protected]>
2022-07-27 06:07 ` Alvaro Herrera <[email protected]>
2022-07-22 08:53 ` Thomas Munro <[email protected]>
2022-07-21 11:11 ` Thomas Munro <[email protected]>
2022-07-21 11:17 ` Alvaro Herrera <[email protected]>
2022-07-27 18:54 ` Alvaro Herrera <[email protected]>
2022-07-28 18:04 ` Matthias van de Meent <[email protected]>
2022-07-28 21:57 ` Tom Lane <[email protected]>
2022-07-28 23:27 ` Thomas Munro <[email protected]>
2022-07-29 05:20 ` Kyotaro Horiguchi <[email protected]>
2022-07-29 10:59 ` Alvaro Herrera <[email protected]>
2022-07-30 14:37 ` Tom Lane <[email protected]>
2022-07-30 16:51 ` Tom Lane <[email protected]>
2022-07-30 23:08 ` Thomas Munro <[email protected]>
2022-07-30 23:17 ` Tom Lane <[email protected]>
2022-08-04 02:54 ` Thomas Munro <[email protected]>
2022-08-05 20:29 ` Alvaro Herrera <[email protected]>
2022-07-31 03:46 ` Thomas Munro <[email protected]>
2022-08-02 19:58 ` Thomas Munro <[email protected]>
2022-08-01 02:01 ` Andres Freund <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox