Message-ID: <fd25e8b7-9538-4e44-90e5-8a357ac03eff@dunslane.net>
Date: Wed, 30 Jul 2025 14:51:59 -0400
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Non-text mode for pg_dumpall
From: Andrew Dunstan <andrew@dunslane.net>
To: Noah Misch <noah@leadboat.com>, Tom Lane <tgl@sss.pgh.pa.us>
Cc: Mahendra Singh Thalor <mahi6run@gmail.com>,
 =?UTF-8?Q?=C3=81lvaro_Herrera?= <alvherre@alvh.no-ip.org>,
 jian he <jian.universality@gmail.com>, Srinath Reddy
 <srinath2133@gmail.com>, pgsql-hackers@lists.postgresql.org
References: 
 <CAKYtNAr-6_CCz+tnJ0T2597Ar4iAZXkE1qPimsO9EY-Kn7LvzQ@mail.gmail.com>
 <20250716001957.c6.nmisch@google.com>
 <CAKYtNAqXTvfAw-y4FHvzprg72cFrC63cge9xMVDO_R4=NHc5rA@mail.gmail.com>
 <CAKYtNAppY=vVtTMdkxRPV48cHKyzLeYF6mrPk15Wczme7e9=ww@mail.gmail.com>
 <3f22a8bb-29e8-40cc-97a1-309181da2c13@dunslane.net>
 <20250722005339.ca.nmisch@google.com>
 <a5359ad4-4e2b-432d-92c0-188f9d558f06@dunslane.net>
 <20250725162141.6f.nmisch@google.com>
 <d1bfd0e7-f70c-48ef-ad5c-ac9d251e4b4c@dunslane.net>
 <2225040.1753477169@sss.pgh.pa.us> <20250727235628.e2.nmisch@google.com>
 <8f2ad50b-ebb7-4adc-997e-25e0ad96ff34@dunslane.net>
 <2bed001a-462c-42da-9a6b-3c7884502932@dunslane.net>
Content-Language: en-US
Autocrypt: addr=andrew@dunslane.net; keydata=
 xsBNBE7KWFkBCAClridxur2AIc7eW2AR7izbfp3EnNefie2HbLF0izW5Ik5UjX2HBXBx4syI
 gY6b0ugohXrr274+baoAlvSbq6cAoQuEVrk5IZFzt20b1Xkx65FwGSEj526yiKLocqkJceSq
 Xr9xcA5SGY+FZv441chh5SU92v4q6z+6LPpoHOh97ptAVXZYNTtU0LevyvD5lja0TzbvJm6C
 eFXitJfnm1pLEr0DGJCR/iUOl/N62Kh4855zZC7NHIjQHPOvV5Stz/l5ilDhvGVk+xkXFPys
 SjZoUr1rXhYLpiyi5sR0X9FHXT0KnGuz1F5ERO7ZTLSSQ6fJwPj6gOk9K+vvoKvoeql5ABEB
 AAHNJEFuZHJldyBEdW5zdGFuIDxhbmRyZXdAZHVuc2xhbmUubmV0PsLAlwQTAQgAQQIbAwIX
 gAIZAQULCQgHAwUVCgkICwUWAgMBAAIeBRYhBOQ+WEYd/Hy/RGkVpZn6f8tZ/DuBBQJoGNGd
 BQkdEO8nAAoJEJn6f8tZ/DuBq74H/jkTR4Zi3stbw+xC7v2u3QozssK7MYPL2AsVfh7OealS
 h182fiWXpfvmmAB7WUHbhk9GC2RAOnHI/2d2jgKaMLAHsGYOT0YopTVIwRY43fCw/mK67yxc
 wmDcX+zyKfLaivNbf5A7QPLNwda98bEAMSJ8Sn652Uc6cA8t3uKGsVzbRBQOoYzjgvBCfSrE
 9ql3PDNg0l4BfAqabd2f70ZUm9VAMEPrgv/v2xI7M2XiL4g5BVmqLCOwxLM8RMCotCuoweUr
 VO43DeBCIDwLxotMJKvGWDjBzQYlU1NPUAtNcz/gN9ITUe1VUGjyvGj4u1lxBOcQQUw7l1+T
 5moZ4iZxXzvOwE0ETspYWQEIANGc4zQULOxhbqO2dyD51YhqCNRmm9oKWaqf+wmW4tpDe/VV
 cxAnNizd4LWCHfzpb5cHAtGkOPePMfzWVf6nvdF7d3eglbtf59+zG7O7llV0xSSoFiieQBsr
 GvqDInXYX/4mRRXMtyhM353/tixC9RWLs1oofyYmCPPXXY7h9R7en3B8BoVrRFcdzlIY/NFN
 hFGW/9dkEiGjgna2Rk6e15kln4ZvFBWUg23p93w/pqXcxY6+k/8TEk+C4R+M6w7o2PLGOjdZ
 +kPiUcw5H85zf/yZJwQXzisXaNduwWB6Vads9YC9dj6kPR1c4VGRqAaYL++LAEOqrlvm2Tvq
 QqZRtnEAEQEAAcLAfAQYAQgAJgIbDBYhBOQ+WEYd/Hy/RGkVpZn6f8tZ/DuBBQJoGNI2BQkd
 EODdAAoJEJn6f8tZ/DuBfw0IAKTsfD40teP/pp+bsLLMSxPXUYrrprTj7WFB5v61p6dkpSr/
 qXmMlyahdxQFaPmfVgVirB1Vk/kHiWNnnGjfUV9nB2Zg9LI0Xb9/ts3LsUiRWXzG3tkMY6XL
 vsVOxW4XFRND9l2q+WW93aZ1DZl+fqWfYgMvsusFRhmGFOKTRfKPta2Pkv+AhA24N4+PrR5p
 bU4k2MO8PAGiK8eaYKGFG1bHKuAvoDoF7WXJ3FHxuWqLnKEt4dfOLm5pAe3zq1Lt6q8azT9i
 QWGpSAK5vQUWQHBHpiDjdPeqKZ6HiAXIIKfSmb+jrvXBqoP+D6/K7rUjG2aXiRtTIAXms9sm
 VRu7cmw=
In-Reply-To: <2bed001a-462c-42da-9a6b-3c7884502932@dunslane.net>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Archived-At: 
 <https://www.postgresql.org/message-id/fd25e8b7-9538-4e44-90e5-8a357ac03eff%40dunslane.net>
Precedence: bulk


On 2025-07-29 Tu 4:09 PM, Andrew Dunstan wrote:
>
> On 2025-07-28 Mo 8:04 AM, Andrew Dunstan wrote:
>>
>> On 2025-07-27 Su 7:56 PM, Noah Misch wrote:
>>> On Fri, Jul 25, 2025 at 04:59:29PM -0400, Tom Lane wrote:
>>>> Andrew Dunstan <andrew@dunslane.net> writes:
>>>>> Before we throw the baby out with the bathwater, how about this
>>>>> suggestion? pg_dumpall would continue to produce globals.dat, but it
>>>>> wouldn't be processed by pg_restore, which would only restore the
>>>>> individual databases. Or else we just don't produce globals.dat at 
>>>>> all.
>>>>> Then we could introduce a structured object that pg_restore could 
>>>>> safely
>>>>> use for release 19, and I think we'd still have something useful for
>>>>> release 18.
>>>> I dunno ... that seems like a pretty weird behavior.  People would
>>>> have to do a separate text-mode "pg_dumpall -g" and remember to
>>>> restore that too.  Admittedly, this could be more convenient than
>>>> "pg_dumpall -g" plus separately pg_dump'ing each database, which is
>>>> what people have to do today if they want anything smarter than a flat
>>>> text dumpfile.  But it still seems like a hack --- and it would not be
>>>> compatible with v19, where presumably "pg_dumpall | pg_restore"
>>>> *would* restore globals.  I think that the prospect of changing
>>>> dump/restore scripts and then having to change them again in v19
>>>> isn't too appetizing.
>>> +1
>>
>>
>> OK, got it. Will revert.
>>
>>
>>
>
> here's a reversion patch for master. It applies cleanly to release 18 
> as well. Thanks to Mahendra Singh Thalor for helping me sanity check 
> it (Any issues are of course my responsibility)
>
>
> I'll work on pulling the entry out of the release notes.
>
>
>


OK, now that's reverted we should discuss how to proceed. I had two 
thoughts - we could use invent a JSON format for the globals, or we 
could just use the existing archive format. I think the archive format 
is pretty flexible, and should be able to accommodate this. The downside 
is it's not humanly readable. The upside is that we don't need to do 
anything special either to write it or parse it.

There might also be other reasonable options. But I think we should stay 
out of the business of using custom code to parse text.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com