public inbox for [email protected]
help / color / mirror / Atom feedFrom: Thomas Munro <[email protected]>
To: Sami Imseih <[email protected]>
Cc: Tom Lane <[email protected]>
Cc: Tomas Vondra <[email protected]>
Cc: Andres Freund <[email protected]>
Cc: Michael Paquier <[email protected]>
Cc: Andrew Dunstan <[email protected]>
Cc: Amul Sul <[email protected]>
Cc: Zsolt Parragi <[email protected]>
Cc: Robert Haas <[email protected]>
Cc: Chao Li <[email protected]>
Cc: Anthonin Bonnefoy <[email protected]>
Cc: Fujii Masao <[email protected]>
Cc: Jakub Wartak <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Subject: Re: pg_waldump: support decoding of WAL inside tarfile
Date: Fri, 3 Apr 2026 13:07:40 +1300
Message-ID: <CA+hUKGLMkv_fnGXzVRO8qbx5uHs-qMn151GTJYCfn9w1ZamGNg@mail.gmail.com> (raw)
In-Reply-To: <CAA5RZ0tt89MgNi4-0F4onH+-TFSsysFjMM-tBc6aXbuQv5xBXw@mail.gmail.com>
References: <[email protected]>
<[email protected]>
<[email protected]>
<CAD5tBcLVWKnph3iB-VPuPKR0dCckOJRFZW2-4H7HTTmhw8-vOg@mail.gmail.com>
<[email protected]>
<[email protected]!!.pa.us>
<CAD5tBcLsYDz+Nzx8MryjxiKaN3fGKd4ZgXuN1Jn=CYxw9dh+AA@mail.gmail.com>
<[email protected]>
<[email protected]>
<[email protected]>
<x2tknjejjouleunkqrvpnwn2tuulunybinycidefm3wmnsyhht@pw5uo3wrqx43>
<CA+hUKGL2dppjO4o28ZY7n_LTWviKLAi-7KZ=tx5w2HGevCEYPA@mail.gmail.com>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<CA+hUKGJyvdyWMC-RW1njqevD-q_gTbFq+DyDiFpUJVaG+DY20w@mail.gmail.com>
<[email protected]>
<CA+hUKG+Pqz5=YQG_=8ho0YsTfn2HWOsJQWqS4j0q8QQWweJP9w@mail.gmail.com>
<[email protected]>
<[email protected]>
<CA+hUKG+-pn14s_tjEBO6YKHmc=uRhGVn=w2oM91KKnEUc7pH0Q@mail.gmail.com>
<[email protected]>
<CAA5RZ0tt89MgNi4-0F4onH+-TFSsysFjMM-tBc6aXbuQv5xBXw@mail.gmail.com>
On Fri, Apr 3, 2026 at 12:43 PM Sami Imseih <[email protected]> wrote:
> The --format=ustar has a limit of 2^21 (2097151) for UID/GID [1]
> and on my machine the UID is 10012663.
>
> So I found that one way to deal with this is to run the tar command with
> --owner=0 --group=0. As far as I can tell, the owner and group IDs don't
> matter for these tests, so maybe that is OK.
>
> @@ -1333,6 +1333,10 @@ sub tar_portability_options
> == 0)
> {
> push(@tar_p_flags, "--format=ustar");
> + # ustar format supports UIDs only up to 2^21 (2097151).
> + # Override owner/group to avoid failures on systems where
> + # the running user's UID/GID exceeds that limit.
> + push(@tar_p_flags, "--owner=0", "--group=0");
Interesting. BSD tar accepts those too, so here's an update to my
previous patch.
> While this fixes the test, I am now not sure what the broader implications are
> for --format=ustar for pg_waldump in the broader discussion?
I think users who have their own tar scripts will mostly be
unaffected, but a small minority will see the new error if they try to
use pg_verifybackup or pg_waldump, and they'll find their way to
--format=ustar, and then they might see the UID/GID error in their own
.tar production scripts, and find their way to adding those switches
too. Seems OK? Especially with a documentation note once we've
settle all of this.
Attachments:
[text/x-patch] v2-0001-Improve-tar-portability-logic-from-ebba64c0.patch (2.1K, 2-v2-0001-Improve-tar-portability-logic-from-ebba64c0.patch)
download | inline diff:
From a2c065fd5cee53bcde2ccaf92939737e8fc07f06 Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Fri, 3 Apr 2026 12:03:56 +1300
Subject: [PATCH v2] Improve tar portability logic from ebba64c0.
* GNU and BSD tar both understand --format=ustar.
* Windows lacks /dev/null, but perl knows its local name.
* ustar format doesn't like large UID/GID values, so set them to 0.
Backpatch-through: 18
Co-authored-by: Thomas Munro <[email protected]>
Co-authored-by: Sami Imseih <[email protected]>
Discussion: https://postgr.es/m/3676229.1775170250%40sss.pgh.pa.us
Discussion: https://postgr.es/m/CAA5RZ0tt89MgNi4-0F4onH%2B-TFSsysFjMM-tBc6aXbuQv5xBXw%40mail.gmail.com
---
src/test/perl/PostgreSQL/Test/Utils.pm | 20 ++++++++------------
1 file changed, 8 insertions(+), 12 deletions(-)
diff --git a/src/test/perl/PostgreSQL/Test/Utils.pm b/src/test/perl/PostgreSQL/Test/Utils.pm
index 120999f6ac9..370acfcef7e 100644
--- a/src/test/perl/PostgreSQL/Test/Utils.pm
+++ b/src/test/perl/PostgreSQL/Test/Utils.pm
@@ -1328,21 +1328,17 @@ sub tar_portability_options
# GNU tar typically produces gnu-format archives, which we can read fine.
# But some platforms configure it to default to posix/pax format, and
- # apparently they enable --sparse too. Override that.
- if (system("$tar --format=ustar -c -O /dev/null >/dev/null 2>/dev/null")
- == 0)
- {
- push(@tar_p_flags, "--format=ustar");
- }
-
- # bsdtar also archives sparse files by default, but it spells the switch
- # to disable that differently.
- if (system("$tar --no-read-sparse -c - /dev/null >/dev/null 2>/dev/null")
+ # apparently they enable --sparse too. BSD tar does something similar.
+ #
+ # ustar format supports UIDs only up to 2^21 (2097151). Override
+ # owner/group to avoid failures on systems where the running user's UID/GID
+ # exceeds that limit.
+ my $devnull = File::Spec->devnull();
+ if (system("$tar --format=ustar --owner=0 --group=0 -c $devnull >$devnull 2>$devnull")
== 0)
{
- push(@tar_p_flags, "--no-read-sparse");
+ push(@tar_p_flags, "--format=ustar", "--owner=0", "--group=0");
}
-
return @tar_p_flags;
}
--
2.53.0
view thread (87+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: pg_waldump: support decoding of WAL inside tarfile
In-Reply-To: <CA+hUKGLMkv_fnGXzVRO8qbx5uHs-qMn151GTJYCfn9w1ZamGNg@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox