MIME-Version: 1.0
References: 
 <CAKYtNAr7ToCBSLO2Mm6NGFTZvbdTAFNgjv_YoRqJJghC3n1MiQ@mail.gmail.com>
 <202503111705.xy7fddu36qae@alvherre.pgsql>
 <CACJufxG44zd+KV4B3_0d+Lf2fwnDDptX-DQRqn-o9ukmXEWeNw@mail.gmail.com>
 <4ef51faa-993f-46ea-9e68-7baf736c07b8@dunslane.net>
 <CAKYtNAoebXn-CC_cgv=-DxL1s7hPvx-1UW7CEfsiJZ8fbD6pjg@mail.gmail.com>
 <a29754b5-e926-4385-a7e4-6a3b4c661fe1@dunslane.net>
 <763d1c6e-9298-4bac-9bea-9331db78a154@dunslane.net>
In-Reply-To: <763d1c6e-9298-4bac-9bea-9331db78a154@dunslane.net>
From: Mahendra Singh Thalor <mahi6run@gmail.com>
Date: Sat, 29 Mar 2025 10:47:21 +0530
Message-ID: 
 <CAKYtNAq8PeyK_bhqX7kR_WPp0x+Z3GAVPnqyBCxe5SVa1MRB7A@mail.gmail.com>
Subject: Re: Non-text mode for pg_dumpall
To: Andrew Dunstan <andrew@dunslane.net>
Cc: jian he <jian.universality@gmail.com>,
	=?UTF-8?Q?=C3=81lvaro_Herrera?= <alvherre@alvh.no-ip.org>,
	Srinath Reddy <srinath2133@gmail.com>, pgsql-hackers@lists.postgresql.org
Content-Type: multipart/mixed; boundary="000000000000e1938b0631744d09"
Archived-At: 
 <https://www.postgresql.org/message-id/CAKYtNAq8PeyK_bhqX7kR_WPp0x%2BZ3GAVPnqyBCxe5SVa1MRB7A%40mail.gmail.com>
Precedence: bulk

--000000000000e1938b0631744d09
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Sat, 29 Mar 2025 at 03:50, Andrew Dunstan <andrew@dunslane.net> wrote:
>
>
> On 2025-03-27 Th 5:15 PM, Andrew Dunstan wrote:
> >
> > On 2025-03-19 We 2:41 AM, Mahendra Singh Thalor wrote:
> >> On Wed, 12 Mar 2025 at 21:18, Andrew Dunstan <andrew@dunslane.net>
> >> wrote:
> >>>
> >>> On 2025-03-12 We 3:03 AM, jian he wrote:
> >>>> On Wed, Mar 12, 2025 at 1:06=E2=80=AFAM =C3=81lvaro Herrera
> >>>> <alvherre@alvh.no-ip.org> wrote:
> >>>>> Hello,
> >>>>>
> >>>>> On 2025-Mar-11, Mahendra Singh Thalor wrote:
> >>>>>
> >>>>>> In map.dat file, I tried to fix this issue by adding number of
> >>>>>> characters
> >>>>>> in dbname but as per code comments, as of now, we are not
> >>>>>> supporting \n\r
> >>>>>> in dbnames so i removed handling.
> >>>>>> I will do some more study to fix this issue.
> >>>>> Yeah, I think this is saying that you should not consider the
> >>>>> contents
> >>>>> of map.dat as a shell string.  After all, you're not going to
> >>>>> _execute_
> >>>>> that file via the shell.
> >>>>>
> >>>>> Maybe for map.dat you need to escape such characters somehow, so th=
at
> >>>>> they don't appear as literal newlines/carriage returns.
> >>>>>
> >>>> I am confused.
> >>>> currently pg_dumpall plain format will abort when encountering dbnam=
e
> >>>> containing newline.
> >>>> the left dumped plain file does not contain all the cluster
> >>>> databases data.
> >>>>
> >>>>
> >>>> if pg_dumpall non-text format aborts earlier,
> >>>> it's aligned with pg_dumpall plain format?
> >>>> it's also an improvement since aborts earlier, nothing will be dumpe=
d?
> >>>>
> >>>>
> >>>> am i missing something?
> >>>>
> >>>>
> >>> I think we should fix that.
> >>>
> >>> But for the current proposal, =C3=81lvaro and I were talking this mor=
ning,
> >>> and we thought the simplest thing here would be to have the one line
> >>> format and escape NL/CRs in the database name.
> >>>
> >>>
> >>> cheers
> >>>
> >> Okay. As per discussions, we will keep one line entry for each
> >> database into map.file.
> >>
> >> Thanks all for feedback and review.
> >>
> >> Here, I am attaching updated patches for review and testing. These
> >> patches can be applied on commit a6524105d20b.
> >
> >
> >
> > I'm working through this patch set with a view to committing it.
> > Attached is some cleanup which is where I got to today, although there
> > is more to do. One thing I am wondering is why not put the
> > SimpleDatabaseOidList stuff in fe_utils/simle_list.{c,h} ? That's
> > where all the similar stuff belongs, and it feels strange to have this
> > inline in pg_restore.c. (I also don't like the name much -
> > SimpleOidStringList or maybe SimpleOidPlusStringList might be better).
> >
> >
> >
>
>
> OK, I have done that, so here is the result. The first two are you
> original patches. patch 3 adds the new list type to fe-utils, and patch
> 4 contains my cleanups and use of the new list type. Apart from some
> relatively minor cleanup, the one thing I would like to change is how
> dumps are named. If we are producing tar or custom format dumps, I think
> the file names should reflect that (oid.dmp and oid.tar rather than a
> bare oid as the filename), and pg_restore should look for those. I'm
> going to work on that tomorrow - I don't think it will be terribly
> difficult.
>

Thanks Andrew.

Here, I am attaching a delta patch for oid.tar and oid.dmp format.

--=20
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com

--000000000000e1938b0631744d09
Content-Type: application/octet-stream;
	name="delta-0001-pg_dumpall-dump-as-tar-and-dmp-file-for-file.patch"
Content-Disposition: attachment;
	filename="delta-0001-pg_dumpall-dump-as-tar-and-dmp-file-for-file.patch"
Content-Transfer-Encoding: base64
Content-ID: <f_m8tret5j0>
X-Attachment-Id: f_m8tret5j0

RnJvbSBiNDNhZTExN2ZlMzgwOWI4MmFiYjViYzg5ZmM2MmQ0NWE1NzA3ZmY2IE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBNYWhlbmRyYSBTaW5naCBUaGFsb3IgPG1haGk2cnVuQGdtYWls
LmNvbT4KRGF0ZTogU2F0LCAyOSBNYXIgMjAyNSAxMDoxNDoxOCArMDUzMApTdWJqZWN0OiBbUEFU
Q0hdIHBnX2R1bXBhbGwgLSBkdW1wIGFzIC50YXIgYW5kIC5kbXAgZmlsZSBmb3IgdGFyIGFuZCBj
dXN0b20KIGZvcm1hdAoKLS0tCiBzcmMvYmluL3BnX2R1bXAvcGdfZHVtcGFsbC5jIHwgIDcgKysr
KysrLQogc3JjL2Jpbi9wZ19kdW1wL3BnX3Jlc3RvcmUuYyB8IDIyICsrKysrKysrKysrKysrKysr
KysrKy0KIDIgZmlsZXMgY2hhbmdlZCwgMjcgaW5zZXJ0aW9ucygrKSwgMiBkZWxldGlvbnMoLSkK
CmRpZmYgLS1naXQgYS9zcmMvYmluL3BnX2R1bXAvcGdfZHVtcGFsbC5jIGIvc3JjL2Jpbi9wZ19k
dW1wL3BnX2R1bXBhbGwuYwppbmRleCA2YWFiMWJmZTgzMS4uMTI5ODNkOTczYmUgMTAwNjQ0Ci0t
LSBhL3NyYy9iaW4vcGdfZHVtcC9wZ19kdW1wYWxsLmMKKysrIGIvc3JjL2Jpbi9wZ19kdW1wL3Bn
X2R1bXBhbGwuYwpAQCAtMTY4NSw3ICsxNjg1LDEyIEBAIGR1bXBEYXRhYmFzZXMoUEdjb25uICpj
b25uLCBBcmNoaXZlRm9ybWF0IGFyY2hEdW1wRm9ybWF0KQogCQkgKi8KIAkJaWYgKGFyY2hEdW1w
Rm9ybWF0ICE9IGFyY2hOdWxsKQogCQl7Ci0JCQlzbnByaW50ZihkYmZpbGVwYXRoLCBNQVhQR1BB
VEgsICJcIiVzXCIvXCIlc1wiIiwgZGJfc3ViZGlyLCBvaWQpOworCQkJaWYgKGFyY2hEdW1wRm9y
bWF0ID09IGFyY2hDdXN0b20pCisJCQkJc25wcmludGYoZGJmaWxlcGF0aCwgTUFYUEdQQVRILCAi
XCIlc1wiL1wiJXNcIi5kbXAiLCBkYl9zdWJkaXIsIG9pZCk7CisJCQllbHNlIGlmIChhcmNoRHVt
cEZvcm1hdCA9PSBhcmNoVGFyKQorCQkJCXNucHJpbnRmKGRiZmlsZXBhdGgsIE1BWFBHUEFUSCwg
IlwiJXNcIi9cIiVzXCIudGFyIiwgZGJfc3ViZGlyLCBvaWQpOworCQkJZWxzZQorCQkJCXNucHJp
bnRmKGRiZmlsZXBhdGgsIE1BWFBHUEFUSCwgIlwiJXNcIi9cIiVzXCIiLCBkYl9zdWJkaXIsIG9p
ZCk7CiAKIAkJCS8qIFB1dCBvbmUgbGluZSBlbnRyeSBmb3IgZGJvaWQgYW5kIGRibmFtZSBpbiBt
YXAgZmlsZS4gKi8KIAkJCWZwcmludGYobWFwX2ZpbGUsICIlcyAlc1xuIiwgb2lkLCBkYm5hbWUp
OwpkaWZmIC0tZ2l0IGEvc3JjL2Jpbi9wZ19kdW1wL3BnX3Jlc3RvcmUuYyBiL3NyYy9iaW4vcGdf
ZHVtcC9wZ19yZXN0b3JlLmMKaW5kZXggNDRhMjQ3OTFhNmUuLjMxY2Q5Yzg0YzVhIDEwMDY0NAot
LS0gYS9zcmMvYmluL3BnX2R1bXAvcGdfcmVzdG9yZS5jCisrKyBiL3NyYy9iaW4vcGdfZHVtcC9w
Z19yZXN0b3JlLmMKQEAgLTExNjQsNiArMTE2NCw4IEBAIHJlc3RvcmVBbGxEYXRhYmFzZXMoUEdj
b25uICpjb25uLCBjb25zdCBjaGFyICpkdW1wZGlycGF0aCwKIAkJIGRiX2NlbGw7IGRiX2NlbGwg
PSBkYl9jZWxsLT5uZXh0KQogCXsKIAkJY2hhcgkJc3ViZGlycGF0aFtNQVhQR1BBVEhdOworCQlj
aGFyCQlzdWJkaXJkYnBhdGhbTUFYUEdQQVRIXTsKKwkJY2hhcgkJZGJmaWxlbmFtZVtNQVhQR1BB
VEhdOwogCQlpbnQJCQluX2Vycm9yczsKIAogCQkvKiBpZ25vcmUgZGJzIG1hcmtlZCBmb3Igc2tp
cHBpbmcgKi8KQEAgLTExODAsNyArMTE4MiwyNSBAQCByZXN0b3JlQWxsRGF0YWJhc2VzKFBHY29u
biAqY29ubiwgY29uc3QgY2hhciAqZHVtcGRpcnBhdGgsCiAJCQlvcHRzLT5jcGFyYW1zLm92ZXJy
aWRlX2RibmFtZSA9IE5VTEw7CiAJCX0KIAotCQlzbnByaW50ZihzdWJkaXJwYXRoLCBNQVhQR1BB
VEgsICIlcy9kYXRhYmFzZXMvJXUiLCBkdW1wZGlycGF0aCwgZGJfY2VsbC0+b2lkKTsKKwkJc25w
cmludGYoc3ViZGlyZGJwYXRoLCBNQVhQR1BBVEgsICIlcy9kYXRhYmFzZXMiLCBkdW1wZGlycGF0
aCk7CisKKwkJLyoKKwkJICogVmFsaWRhdGUgZGF0YWJhc2UgZHVtcCBmaWxlLiAgSWYgdGhlcmUg
aXMgLnRhciBvciAuZG1wIGZpbGUgZXhpc3QKKwkJICogdGhlbiBjb25zaWRlciBwYXJ0aWN1bGFy
IGZpbGUsIG90aGVyd2lzZSBqdXN0IGFwcGVuZCBkYm9pZCB0byB0aGUKKwkJICogZGF0YWJhc2Vz
IGZvbGRlci4KKwkJICovCisJCXNucHJpbnRmKGRiZmlsZW5hbWUsIE1BWFBHUEFUSCwgIiV1LnRh
ciIsIGRiX2NlbGwtPm9pZCk7CisJCWlmIChmaWxlX2V4aXN0c19pbl9kaXJlY3Rvcnkoc3ViZGly
ZGJwYXRoLCBkYmZpbGVuYW1lKSkKKwkJCXNucHJpbnRmKHN1YmRpcnBhdGgsIE1BWFBHUEFUSCwg
IiVzL2RhdGFiYXNlcy8ldS50YXIiLCBkdW1wZGlycGF0aCwgZGJfY2VsbC0+b2lkKTsKKwkJZWxz
ZQorCQl7CisJCQlzbnByaW50ZihkYmZpbGVuYW1lLCBNQVhQR1BBVEgsICIldS5kbXAiLCBkYl9j
ZWxsLT5vaWQpOworCisJCQlpZiAoZmlsZV9leGlzdHNfaW5fZGlyZWN0b3J5KHN1YmRpcmRicGF0
aCwgZGJmaWxlbmFtZSkpCisJCQkJc25wcmludGYoc3ViZGlycGF0aCwgTUFYUEdQQVRILCAiJXMv
ZGF0YWJhc2VzLyV1LmRtcCIsIGR1bXBkaXJwYXRoLCBkYl9jZWxsLT5vaWQpOworCQkJZWxzZQor
CQkJCXNucHJpbnRmKHN1YmRpcnBhdGgsIE1BWFBHUEFUSCwgIiVzL2RhdGFiYXNlcy8ldSIsIGR1
bXBkaXJwYXRoLCBkYl9jZWxsLT5vaWQpOworCQl9CiAKIAkJcGdfbG9nX2luZm8oInJlc3Rvcmlu
ZyBkYXRhYmFzZSBcIiVzXCIiLCBkYl9jZWxsLT5zdHIpOwogCi0tIAoyLjM5LjMKCg==
--000000000000e1938b0631744d09--