pg.ddx.io pgsql-committers@postgresql.org mailing list archive
help / color / mirror / Atom feedpgsql: Fix pg_basebackup with in-place tablespaces.
9+ messages / 6 participants
[nested] [flat]
* pgsql: Fix pg_basebackup with in-place tablespaces.
@ 2022-03-15 01:31 Thomas Munro <tmunro@postgresql.org>
2022-03-15 20:40 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. David Steele <david@pgmasters.net>
0 siblings, 1 reply; 9+ messages in thread
From: Thomas Munro @ 2022-03-15 01:31 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
Fix pg_basebackup with in-place tablespaces.
Previously, pg_basebackup from a cluster that contained an 'in-place'
tablespace, as introduced by commit 7170f215, would produce a harmless
warning on Unix and fail completely on Windows.
Reported-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/20220304.165449.1200020258723305904.horikyota.ntt%40gmail.com
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/c6f2f01611d4f2c412e92eb7893f76fa590818e8
Modified Files
--------------
src/backend/access/transam/xlog.c | 14 ++++++++++++++
1 file changed, 14 insertions(+)
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: pgsql: Fix pg_basebackup with in-place tablespaces.
2022-03-15 01:31 pgsql: Fix pg_basebackup with in-place tablespaces. Thomas Munro <tmunro@postgresql.org>
@ 2022-03-15 20:40 ` David Steele <david@pgmasters.net>
2022-03-15 21:28 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 9+ messages in thread
From: David Steele @ 2022-03-15 20:40 UTC (permalink / raw)
To: Thomas Munro <tmunro@postgresql.org>; pgsql-committers@lists.postgresql.org
On 3/14/22 19:31, Thomas Munro wrote:
> Fix pg_basebackup with in-place tablespaces.
>
> Previously, pg_basebackup from a cluster that contained an 'in-place'
> tablespace, as introduced by commit 7170f215, would produce a harmless
> warning on Unix and fail completely on Windows.
Perhaps I'm being picky, but seems like this logic should be wrapped in:
if (allow_in_place_tablespaces)
{
<...>
}
I worry about strange effects when this GUC is not enabled.
Regards,
-David
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: pgsql: Fix pg_basebackup with in-place tablespaces.
2022-03-15 01:31 pgsql: Fix pg_basebackup with in-place tablespaces. Thomas Munro <tmunro@postgresql.org>
2022-03-15 20:40 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. David Steele <david@pgmasters.net>
@ 2022-03-15 21:28 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-15 22:13 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 9+ messages in thread
From: Tom Lane @ 2022-03-15 21:28 UTC (permalink / raw)
To: David Steele <david@pgmasters.net>; +Cc: Thomas Munro <tmunro@postgresql.org>; pgsql-committers@lists.postgresql.org
David Steele <david@pgmasters.net> writes:
> On 3/14/22 19:31, Thomas Munro wrote:
>> Fix pg_basebackup with in-place tablespaces.
> Perhaps I'm being picky, but seems like this logic should be wrapped in:
> if (allow_in_place_tablespaces)
> {
> <...>
> }
> I worry about strange effects when this GUC is not enabled.
What would happen if someone created an in-place tablespace and
then turned off the GUC?
regards, tom lane
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: pgsql: Fix pg_basebackup with in-place tablespaces.
2022-03-15 01:31 pgsql: Fix pg_basebackup with in-place tablespaces. Thomas Munro <tmunro@postgresql.org>
2022-03-15 20:40 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. David Steele <david@pgmasters.net>
2022-03-15 21:28 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. Tom Lane <tgl@sss.pgh.pa.us>
@ 2022-03-15 22:13 ` Thomas Munro <thomas.munro@gmail.com>
2022-03-16 05:42 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
0 siblings, 1 reply; 9+ messages in thread
From: Thomas Munro @ 2022-03-15 22:13 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: David Steele <david@pgmasters.net>; Thomas Munro <tmunro@postgresql.org>; pgsql-committers <pgsql-committers@lists.postgresql.org>
On Wed, Mar 16, 2022 at 10:28 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> David Steele <david@pgmasters.net> writes:
> > On 3/14/22 19:31, Thomas Munro wrote:
> >> Fix pg_basebackup with in-place tablespaces.
>
> > Perhaps I'm being picky, but seems like this logic should be wrapped in:
> > if (allow_in_place_tablespaces)
> > {
> > <...>
> > }
> > I worry about strange effects when this GUC is not enabled.
>
> What would happen if someone created an in-place tablespace and
> then turned off the GUC?
Then it would break with unhelpful error messages. I suppose we could
error out explicitly, "in-place tablespace detected, but
allow_in_place_tablespaces not enabled". I'm not sure why we should
suddenly choose to enforce that *here* though.
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: pgsql: Fix pg_basebackup with in-place tablespaces.
2022-03-15 01:31 pgsql: Fix pg_basebackup with in-place tablespaces. Thomas Munro <tmunro@postgresql.org>
2022-03-15 20:40 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. David Steele <david@pgmasters.net>
2022-03-15 21:28 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. Tom Lane <tgl@sss.pgh.pa.us>
2022-03-15 22:13 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. Thomas Munro <thomas.munro@gmail.com>
@ 2022-03-16 05:42 ` Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2022-03-16 14:29 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. David Steele <david@pgmasters.net>
0 siblings, 1 reply; 9+ messages in thread
From: Kyotaro Horiguchi @ 2022-03-16 05:42 UTC (permalink / raw)
To: thomas.munro@gmail.com; +Cc: tgl@sss.pgh.pa.us; david@pgmasters.net; tmunro@postgresql.org; pgsql-committers@lists.postgresql.org
At Wed, 16 Mar 2022 11:13:53 +1300, Thomas Munro <thomas.munro@gmail.com> wrote in
> On Wed, Mar 16, 2022 at 10:28 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > David Steele <david@pgmasters.net> writes:
> > > On 3/14/22 19:31, Thomas Munro wrote:
> > >> Fix pg_basebackup with in-place tablespaces.
> >
> > > Perhaps I'm being picky, but seems like this logic should be wrapped in:
> > > if (allow_in_place_tablespaces)
> > > {
> > > <...>
> > > }
> > > I worry about strange effects when this GUC is not enabled.
> >
> > What would happen if someone created an in-place tablespace and
> > then turned off the GUC?
>
> Then it would break with unhelpful error messages. I suppose we could
> error out explicitly, "in-place tablespace detected, but
> allow_in_place_tablespaces not enabled". I'm not sure why we should
> suddenly choose to enforce that *here* though.
+1. The GUC is only to prevent non-developer users from accidentally
creating in-place tablespaces that is not officieally suported. We
have done nothing about in-place tablespaces other than the things
needed to perform regression test, and I think we won't do that in
future.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: pgsql: Fix pg_basebackup with in-place tablespaces.
2022-03-15 01:31 pgsql: Fix pg_basebackup with in-place tablespaces. Thomas Munro <tmunro@postgresql.org>
2022-03-15 20:40 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. David Steele <david@pgmasters.net>
2022-03-15 21:28 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. Tom Lane <tgl@sss.pgh.pa.us>
2022-03-15 22:13 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. Thomas Munro <thomas.munro@gmail.com>
2022-03-16 05:42 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
@ 2022-03-16 14:29 ` David Steele <david@pgmasters.net>
2022-03-16 19:33 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 9+ messages in thread
From: David Steele @ 2022-03-16 14:29 UTC (permalink / raw)
To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; thomas.munro@gmail.com; +Cc: tgl@sss.pgh.pa.us; tmunro@postgresql.org; pgsql-committers@lists.postgresql.org
On 3/15/22 23:42, Kyotaro Horiguchi wrote:
> At Wed, 16 Mar 2022 11:13:53 +1300, Thomas Munro <thomas.munro@gmail.com> wrote in
>> On Wed, Mar 16, 2022 at 10:28 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> David Steele <david@pgmasters.net> writes:
>>>> On 3/14/22 19:31, Thomas Munro wrote:
>>>>> Fix pg_basebackup with in-place tablespaces.
>>>
>>>> Perhaps I'm being picky, but seems like this logic should be wrapped in:
>>>> if (allow_in_place_tablespaces)
>>>> {
>>>> <...>
>>>> }
>>>> I worry about strange effects when this GUC is not enabled.
>>>
>>> What would happen if someone created an in-place tablespace and
>>> then turned off the GUC?
>>
>> Then it would break with unhelpful error messages. I suppose we could
>> error out explicitly, "in-place tablespace detected, but
>> allow_in_place_tablespaces not enabled". I'm not sure why we should
>> suddenly choose to enforce that *here* though.
>
> +1. The GUC is only to prevent non-developer users from accidentally
> creating in-place tablespaces that is not officieally suported. We
> have done nothing about in-place tablespaces other than the things
> needed to perform regression test, and I think we won't do that in
> future.
Sure, but there is a behavioral change whether the GUC is enabled or
not. Before, if there was clutter in pg_tblspc there would at least be a
warning in the log. Now that logging does not happen.
It seems that the warning at line 8313 is essentially dead code now. I
don't expect test code to have an impact on production systems, even if
the effect is minor.
I'm less worried about what happens when the flag is flipped on then off
because that's not likely to happen in production.
Regards,
-David
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: pgsql: Fix pg_basebackup with in-place tablespaces.
2022-03-15 01:31 pgsql: Fix pg_basebackup with in-place tablespaces. Thomas Munro <tmunro@postgresql.org>
2022-03-15 20:40 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. David Steele <david@pgmasters.net>
2022-03-15 21:28 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. Tom Lane <tgl@sss.pgh.pa.us>
2022-03-15 22:13 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. Thomas Munro <thomas.munro@gmail.com>
2022-03-16 05:42 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2022-03-16 14:29 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. David Steele <david@pgmasters.net>
@ 2022-03-16 19:33 ` Thomas Munro <thomas.munro@gmail.com>
2022-03-16 21:09 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. David Steele <david@pgmasters.net>
0 siblings, 1 reply; 9+ messages in thread
From: Thomas Munro @ 2022-03-16 19:33 UTC (permalink / raw)
To: David Steele <david@pgmasters.net>; +Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Thomas Munro <tmunro@postgresql.org>; pgsql-committers <pgsql-committers@lists.postgresql.org>
On Thu, Mar 17, 2022 at 3:29 AM David Steele <david@pgmasters.net> wrote:
> Sure, but there is a behavioral change whether the GUC is enabled or
> not. Before, if there was clutter in pg_tblspc there would at least be a
> warning in the log. Now that logging does not happen.
If there's clutter that doesn't look like an OID, we already ignore it
silently here:
/* Skip anything that doesn't look like a tablespace */
if (strspn(de->d_name, "0123456789") != strlen(de->d_name))
continue;
> It seems that the warning at line 8313 is essentially dead code now. I
> don't expect test code to have an impact on production systems, even if
> the effect is minor.
It's not dead, it's how we'd report something like EACCES or EIO. Why
we only warn and press on, I'm not sure. (I'm also looking into why
we ignore OS errors for pgwin32_is_junction everywhere.)
> I'm less worried about what happens when the flag is flipped on then off
> because that's not likely to happen in production.
So, concretely, if there is a non-symlink with a name that looks like
an OID under pg_tblspc, previously we'd barf (or apparently press on
with an empty pathname on Windows, which might indicate a lack of
error checking somewhere). Given the policy of quietly ignoring
anything else in the directory, is it really this code's job to sanity
check the cluster layout? Hmm, I guess we could fix the problem on
Windows in a different way so that it behaves like Unix, and then
revert this. You'd get an ignorable not-a-symlink warning as before
(and now it'd work on Windows) when backing up in-place tablespaces,
but I'm not sure it's really an improvement.
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: pgsql: Fix pg_basebackup with in-place tablespaces.
2022-03-15 01:31 pgsql: Fix pg_basebackup with in-place tablespaces. Thomas Munro <tmunro@postgresql.org>
2022-03-15 20:40 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. David Steele <david@pgmasters.net>
2022-03-15 21:28 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. Tom Lane <tgl@sss.pgh.pa.us>
2022-03-15 22:13 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. Thomas Munro <thomas.munro@gmail.com>
2022-03-16 05:42 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2022-03-16 14:29 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. David Steele <david@pgmasters.net>
2022-03-16 19:33 ` Re: pgsql: Fix pg_basebackup with in-place tablespaces. Thomas Munro <thomas.munro@gmail.com>
@ 2022-03-16 21:09 ` David Steele <david@pgmasters.net>
0 siblings, 0 replies; 9+ messages in thread
From: David Steele @ 2022-03-16 21:09 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Thomas Munro <tmunro@postgresql.org>; pgsql-committers <pgsql-committers@lists.postgresql.org>
On 3/16/22 13:33, Thomas Munro wrote:
>
>> It seems that the warning at line 8313 is essentially dead code now. I
>> don't expect test code to have an impact on production systems, even if
>> the effect is minor.
>
> It's not dead, it's how we'd report something like EACCES or EIO. Why
> we only warn and press on, I'm not sure. (I'm also looking into why
> we ignore OS errors for pgwin32_is_junction everywhere.)
I'm also wondering why this is only a warning.
>> I'm less worried about what happens when the flag is flipped on then off
>> because that's not likely to happen in production.
>
> So, concretely, if there is a non-symlink with a name that looks like
> an OID under pg_tblspc, previously we'd barf (or apparently press on
> with an empty pathname on Windows, which might indicate a lack of
> error checking somewhere). Given the policy of quietly ignoring
> anything else in the directory, is it really this code's job to sanity
> check the cluster layout? Hmm, I guess we could fix the problem on
> Windows in a different way so that it behaves like Unix, and then
> revert this. You'd get an ignorable not-a-symlink warning as before
> (and now it'd work on Windows) when backing up in-place tablespaces,
> but I'm not sure it's really an improvement.
I'm not going to press on this, but FWIW this solution sounds better to
me. Either way it is a behavioral change. Windows used to error in this
case and now it does not, Unix used to warn in this case and now it does
not.
For my 2C I'd rather this was a hard error because that makes life a lot
easier down the line. But, that's certainly not the responsibility of
this patch.
Regards,
-David
^ permalink raw reply [nested|flat] 9+ messages in thread
* pgsql: Fix pg_basebackup with in-place tablespaces some more.
@ 2023-04-18 15:35 Robert Haas <rhaas@postgresql.org>
0 siblings, 0 replies; 9+ messages in thread
From: Robert Haas @ 2023-04-18 15:35 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
Fix pg_basebackup with in-place tablespaces some more.
Commit c6f2f01611d4f2c412e92eb7893f76fa590818e8 purported to make
this work, but problems remained. In a plain-format backup, the
files from an in-place tablespace got included in the tar file for
the main tablespace, which is wrong but it's not clear that it
has any user-visible consequences. In a tar-format backup, the
TABLESPACE_MAP option is used, and so we never iterated over
pg_tblspc and thus never backed up the in-place tablespaces
anywhere at all.
To fix this, reverse the changes in that commit, so that when we scan
pg_tblspc during a backup, we create tablespaceinfo objects even for
in-place tablespaces. We set the field that would normally contain the
absolute pathname to the relative path pg_tblspc/${TSOID}, and that's
good enough to make basebackup.c happy without any further changes.
However, pg_basebackup needs a couple of adjustments to make it work.
First, it needs to understand that a relative path for a tablespace
means it's an in-place tablespace. Second, it needs to tolerate the
situation where restoring the main tablespace tries to create
pg_tblspc or a subdirectory and finds that it already exists, because
we restore user-defined tablespaces before the main tablespace.
Since in-place tablespaces are only intended for use in development
and testing, no back-patch.
Patch by me, reviewed by Thomas Munro and Michael Paquier.
Discussion: http://postgr.es/m/CA+TgmobwvbEp+fLq2PykMYzizcvuNv0a7gPMJtxOTMOuuRLMHg@mail.gmail.com
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/363e8f9115469fe3d30a80b694cd60e9db3b2537
Modified Files
--------------
src/backend/access/transam/xlog.c | 110 ++++++++++++---------
src/bin/pg_basebackup/bbstreamer_file.c | 53 +++++++---
src/bin/pg_basebackup/pg_basebackup.c | 22 ++++-
src/bin/pg_basebackup/t/011_in_place_tablespace.pl | 40 ++++++++
4 files changed, 159 insertions(+), 66 deletions(-)
^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2023-04-18 15:35 UTC | newest]
Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-03-15 01:31 pgsql: Fix pg_basebackup with in-place tablespaces. Thomas Munro <tmunro@postgresql.org>
2022-03-15 20:40 ` David Steele <david@pgmasters.net>
2022-03-15 21:28 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-15 22:13 ` Thomas Munro <thomas.munro@gmail.com>
2022-03-16 05:42 ` Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2022-03-16 14:29 ` David Steele <david@pgmasters.net>
2022-03-16 19:33 ` Thomas Munro <thomas.munro@gmail.com>
2022-03-16 21:09 ` David Steele <david@pgmasters.net>
2023-04-18 15:35 pgsql: Fix pg_basebackup with in-place tablespaces some more. Robert Haas <rhaas@postgresql.org>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox