Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1vFoB5-001QD6-JZ for pgsql-hackers@arkaria.postgresql.org; Mon, 03 Nov 2025 06:36:31 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1vFoB3-0002n6-If for pgsql-hackers@arkaria.postgresql.org; Mon, 03 Nov 2025 06:36:28 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1vFoB2-0002my-RH for pgsql-hackers@lists.postgresql.org; Mon, 03 Nov 2025 06:36:28 +0000 Received: from mail-vk1-xa35.google.com ([2607:f8b0:4864:20::a35]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.96) (envelope-from ) id 1vFoAy-005AVq-1x for pgsql-hackers@lists.postgresql.org; Mon, 03 Nov 2025 06:36:25 +0000 Received: by mail-vk1-xa35.google.com with SMTP id 71dfb90a1353d-55964d4ccc1so49546e0c.3 for ; Sun, 02 Nov 2025 22:36:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enterprisedb.com; s=google; t=1762151783; x=1762756583; darn=lists.postgresql.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=CccwJfoaNEaAteK1ccI3mbifPjyXZaxLXnwCSHCXo/w=; b=HQmPi98hA3m5fVOK0IR/SrrCCZ0g+IYTk91CeXE95XjkZwVzm55GGgMAUpRZDOxft2 PAi/p7/drXYQCwcdC0fHFeiVHTZP/5eRQiQtIu83go8XsjUTizI3VyEUkdi9sWn41/uc 7QjXUIwKt47+1H9AmzW3IobTBqFsMUkcv2nFRRfD4d8gzR5tHR3NkC3MDhHWzmlEgdWo dAH/+54Eb09YheYSsfBV9cFzZ1PWEavvTN/xDBekCsGZ4JwPa8irr4CHDWgv2UoPu95J 6reAchHipDRnm0sLopntBts/eYAEeXvUp8sNZ3kzV+E2rAv+VnKqVvf/Vyz5giC5WCn1 g66A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762151783; x=1762756583; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=CccwJfoaNEaAteK1ccI3mbifPjyXZaxLXnwCSHCXo/w=; b=iLyuy7I871inOvK8I7OMktfP/5SQ8CBux5bcTxcNUO15yAyH3dcFQK/bqC7UTKl4dx bugDkPs4piD7xqVOXTIldHvRkL6NS5ESwMavjRHDR0elxz2cKS5oikclQN2Fsp4EnngW m9k00tModNhXeIiFb7X3bZ/OeK+S8W3sSotD0swBQ93fikrNNGQaI16GGhaaoPG8xqYX F2DJTkhH1u2s9PvshRf/WEyY1mIDvyS1pcAqbyYYqg1SNSRUenvp+oWD4GS57+fyIGAU MdVjKBmWP6fcS185Otm/eEtmniq1KOxgcOfe1UQSnjuUHYq2QWggxIvXk/+IOi2ziWFR 3sFw== X-Gm-Message-State: AOJu0Yx2f67Gvzq3c9PO7B8RNv/TQzOJaJo8+FXvW1Hfa4uCrHO1Pt/J VZ20sKgfzWb9vs7OBjXsJxpBdfk5MOONB8jKeP+/BmoQMI/epOeXjSMVLNsLS53kz6ihCPBqcc1 leV0UhLMTwBGmIPBm7OLrPT7oU0WWWazBGZ1wSOnZ X-Gm-Gg: ASbGncs0yrCyT7OePIkWkUP57aYCkFluefYE5lgyfbH6RSw37ZPs7WV3xxi3W5NObJ3 LTkdQA4PTDQArTSJEhnPYHMbH1RXk3IC5oh/f/L6kuzHZQF4hbQvEGMxwtc666ERUfZ8TuKXBJk NBB6A7es0G078kcxuzkbwbBs+u3tkca4VvVLsrWUC4Nr8GO4prnm0WlhPA0Js5vLVUp8IVbn9rb qJJ51J4rgeUrK8bzgVNl57q8oL12q4+6wA+cPQWvZt1PQaK15S85WsJNOkOWp3LUZYlRx/R X-Google-Smtp-Source: AGHT+IEcjDODfDd9DMm6YieMc50T+EFw51OGLpTOAh/AFuQJ6+ghFYKeTXBIOIRe5L1f4+qLozusFz45uyNdjKxd07U= X-Received: by 2002:a05:6122:3c84:b0:559:65d6:1666 with SMTP id 71dfb90a1353d-55965d61da4mr581237e0c.1.1762151783415; Sun, 02 Nov 2025 22:36:23 -0800 (PST) MIME-Version: 1.0 References: <3f22a8bb-29e8-40cc-97a1-309181da2c13@dunslane.net> <20250722005339.ca.nmisch@google.com> <20250725162141.6f.nmisch@google.com> <2225040.1753477169@sss.pgh.pa.us> <20250727235628.e2.nmisch@google.com> <8f2ad50b-ebb7-4adc-997e-25e0ad96ff34@dunslane.net> <2bed001a-462c-42da-9a6b-3c7884502932@dunslane.net> <20250824010811.4d.nmisch@google.com> <82eb35b8-7f07-493b-b689-0934919e1dc3@dunslane.net> In-Reply-To: From: Vaibhav Dalvi Date: Mon, 3 Nov 2025 12:05:46 +0530 X-Gm-Features: AWmQ_bkH1GR4xSidjskfeb3by3qTsNaJ_MmjhPR6Rkc-YBYq0i9aJ3zY9ayLf1Q Message-ID: Subject: Re: Non-text mode for pg_dumpall To: Mahendra Singh Thalor Cc: pgsql-hackers@lists.postgresql.org, Vaibhav Dalvi Content-Type: multipart/alternative; boundary="0000000000001753800642aaef42" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --0000000000001753800642aaef42 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Mahendra, Thank you for your work on this feature. I have just begun reviewing the latest patch and encountered the following errors during the initial setup: ``` $ ./db/bin/pg_restore testdump_dir -C -d postgres -F d -p 5556 pg_restore: error: could not execute query: ERROR: syntax error at or near "\\" LINE 1: \restrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj... ^ Command was: \restrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb pg_restore: error: could not execute query: ERROR: syntax error at or near "\\" LINE 1: \unrestrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCj... ^ Command was: \unrestrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb pg_restore: error: could not execute query: ERROR: syntax error at or near "\\" LINE 1: \connect template1 ^ Command was: \connect template1 pg_restore: error: could not execute query: ERROR: syntax error at or near "\\" LINE 1: \connect postgres ^ Command was: \connect postgres ``` To cross-check tried with plain dump(with pg_dumpall) and restored(SQL file restore) without patch and didn't get above connection errors. It appears there might be an issue with the dump file itself. Please note that this is my first observation as I have just started the review. I will continue with my assessment. Regards, Vaibhav Dalvi EnterpriseDB On Fri, Oct 31, 2025 at 2:51=E2=80=AFPM Mahendra Singh Thalor wrote: > On Tue, 28 Oct 2025 at 11:32, Mahendra Singh Thalor > wrote: > > > > On Thu, 16 Oct 2025 at 16:24, Mahendra Singh Thalor > wrote: > > > > > > On Wed, 15 Oct 2025 at 23:05, Mahendra Singh Thalor < > mahi6run@gmail.com> wrote: > > > > > > > > On Sun, 24 Aug 2025 at 22:12, Andrew Dunstan > wrote: > > > > > > > > > > > > > > > On 2025-08-23 Sa 9:08 PM, Noah Misch wrote: > > > > > > > > > > On Wed, Jul 30, 2025 at 02:51:59PM -0400, Andrew Dunstan wrote: > > > > > > > > > > OK, now that's reverted we should discuss how to proceed. I had > two thoughts > > > > > - we could use invent a JSON format for the globals, or we could > just use > > > > > the existing archive format. I think the archive format is pretty > flexible, > > > > > and should be able to accommodate this. The downside is it's not > humanly > > > > > readable. The upside is that we don't need to do anything special > either to > > > > > write it or parse it. > > > > > > > > > > I would first try to use the existing archiver API, because that > makes it > > > > > harder to miss bugs. Any tension between that API and pg_dumpall > is likely to > > > > > have corresponding tension on the pg_restore side. Resolving tha= t > tension > > > > > will reveal much of the project's scope that remained hidden > during the v18 > > > > > attempt. Perhaps more important than that, using the archiver AP= I > means > > > > > future pg_dump and pg_restore options are more likely to cooperat= e > properly > > > > > with $SUBJECT. In other words, I want it to be hard to add > pg_dump/pg_restore > > > > > features that malfunction only for $SUBJECT archives. The > strength of the > > > > > archiver architecture shows in how rarely new features need > format-specific > > > > > logic and how rarely format-specific bugs get reported. We've ha= d > little or > > > > > no trouble with e.g. bugs that appear in -Fd but not in -Fc. > > > > > > > > > > > > > > > Yeah, that's what we're going to try. > > > > > > > > > > > > > > > cheers > > > > > > > > > > > > > > > andrew > > > > > > > > > > -- > > > > > Andrew Dunstan > > > > > EDB: https://www.enterprisedb.com > > > > > > > > Thanks Andrew, Noah and all others for feedback. > > > > > > > > Based on the above suggestions and discussions, I removed sql > commands > > > > from the global.dat file. For global commands, now we are making > > > > toc.dat/toc.dmp/toc.tar file based on format specified and based on > > > > format specified, we are making archive entries for these global > > > > commands. By this approach, we removed the hard-coded parsing part = of > > > > the global.dat file and we are able to skip DROP DATABASE with the > > > > globals-only option. > > > > > > > > Here, I am attaching a patch for review, testing and feedback. This > is > > > > a WIP patch. I will do some more code cleanup and will add some mor= e > > > > comments also. Please review this and let me know design level > > > > feedback. Thanks Tushar Ahuja for some internal testing and feedbac= k. > > > > > > > > > > Hi, > > > Here, I am attaching an updated patch. In offline discussion, Andrew > > > reported some test-case failures(Thanks Andrew). I fixed those. > > > Please let me know feedback for the patch. > > > > > > > Hi, > > Here I am attaching a re-based patch as v02 was failing on head. > > Thanks Tushar for the testing. > > Please review this and let me know feedback. > > > > Hi all, > Here I am attaching an updated patch for review and testing. Based on > some offline comments by Andrew, I did some code cleanup. > Please consider this patch for feedback. > > -- > Thanks and Regards > Mahendra Singh Thalor > EnterpriseDB: http://www.enterprisedb.com > --0000000000001753800642aaef42 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Mahendra,

Thank you for y= our work on this feature.
I have just begun reviewing the latest = patch and
encountered the following errors during the initial set= up:

```
$ ./db/bin/pg_restore testdump_d= ir -C -d postgres -F d -p 5556
pg_restore: error: could not execu= te query: ERROR: syntax error at or near "\\"
LINE 1: \= restrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj...
= ^
Command was: \restrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguF= z5qskGaFDygTCjCj9vg3Xxys1b3hb

pg_restore: error: c= ould not execute query: ERROR: syntax error at or near "\\"
=
LINE 1: \unrestrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCj..= .
^
Command was: \unrestrict aO9K1gzVZTlafidF5f= Wx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb

pg_r= estore: error: could not execute query: ERROR: syntax error at or near &quo= t;\\"
LINE 1: \connect template1
^
Command was: \connect template1

pg_restore: err= or: could not execute query: ERROR: syntax error at or near "\\"<= /div>
LINE 1: \connect postgres
^
Command w= as: \connect postgres
```
To cross-check tried with pla= in dump(with pg_dumpall) and
=C2=A0restored(SQL file restore) wit= hout patch and didn't get above
connection errors.
=
It appears there might be an issue with the dump file itself= .
Please note that this is my first observation as I have just
started the review. I will continue with my assessment.
<= br>
Regards,
Vaibhav Dalvi
EnterpriseDB
=

On Fri, Oct 31, 2025 at 2:51=E2=80=AFPM Mahendra Sing= h Thalor <mahi6run@gmail.com&g= t; wrote:
On Tue= , 28 Oct 2025 at 11:32, Mahendra Singh Thalor <mahi6run@gmail.com> wrote:
>
> On Thu, 16 Oct 2025 at 16:24, Mahendra Singh Thalor <mahi6run@gmail.com> wrote:=
> >
> > On Wed, 15 Oct 2025 at 23:05, Mahendra Singh Thalor <mahi6run@gmail.com>= wrote:
> > >
> > > On Sun, 24 Aug 2025 at 22:12, Andrew Dunstan <andrew@dunslane.net> w= rote:
> > > >
> > > >
> > > > On 2025-08-23 Sa 9:08 PM, Noah Misch wrote:
> > > >
> > > > On Wed, Jul 30, 2025 at 02:51:59PM -0400, Andrew Dunsta= n wrote:
> > > >
> > > > OK, now that's reverted we should discuss how to pr= oceed. I had two thoughts
> > > > - we could use invent a JSON format for the globals, or= we could just use
> > > > the existing archive format. I think the archive format= is pretty flexible,
> > > > and should be able to accommodate this. The downside is= it's not humanly
> > > > readable. The upside is that we don't need to do an= ything special either to
> > > > write it or parse it.
> > > >
> > > > I would first try to use the existing archiver API, bec= ause that makes it
> > > > harder to miss bugs.=C2=A0 Any tension between that API= and pg_dumpall is likely to
> > > > have corresponding tension on the pg_restore side.=C2= =A0 Resolving that tension
> > > > will reveal much of the project's scope that remain= ed hidden during the v18
> > > > attempt.=C2=A0 Perhaps more important than that, using = the archiver API means
> > > > future pg_dump and pg_restore options are more likely t= o cooperate properly
> > > > with $SUBJECT.=C2=A0 In other words, I want it to be ha= rd to add pg_dump/pg_restore
> > > > features that malfunction only for $SUBJECT archives.= =C2=A0 The strength of the
> > > > archiver architecture shows in how rarely new features = need format-specific
> > > > logic and how rarely format-specific bugs get reported.= =C2=A0 We've had little or
> > > > no trouble with e.g. bugs that appear in -Fd but not in= -Fc.
> > > >
> > > >
> > > > Yeah, that's what we're going to try.
> > > >
> > > >
> > > > cheers
> > > >
> > > >
> > > > andrew
> > > >
> > > > --
> > > > Andrew Dunstan
> > > > EDB: https://www.enterprisedb.com
> > >
> > > Thanks Andrew, Noah and all others for feedback.
> > >
> > > Based on the above suggestions and discussions, I removed sq= l commands
> > > from the global.dat file. For global commands, now we are ma= king
> > > toc.dat/toc.dmp/toc.tar file based on format specified and b= ased on
> > > format specified, we are making archive entries for these gl= obal
> > > commands. By this approach, we removed the hard-coded parsin= g part of
> > > the global.dat file and we are able to skip DROP DATABASE wi= th the
> > > globals-only option.
> > >
> > > Here, I am attaching a patch for review, testing and feedbac= k. This is
> > > a WIP patch. I will do some more code cleanup and will add s= ome more
> > > comments also. Please review this and let me know design lev= el
> > > feedback. Thanks Tushar Ahuja for some internal testing and = feedback.
> > >
> >
> > Hi,
> > Here, I am attaching an updated patch. In offline discussion, And= rew
> > reported some test-case failures(Thanks Andrew). I fixed those. > > Please let me know feedback for the patch.
> >
>
> Hi,
> Here I am attaching a re-based patch as v02 was failing on head.
> Thanks Tushar for the testing.
> Please review this and let me know feedback.
>

Hi all,
Here I am attaching an updated patch for review and testing. Based on
some offline comments by Andrew, I did some code cleanup.
Please consider this patch for feedback.

--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
--0000000000001753800642aaef42--