Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tVdA0-00A9kg-Vs for pgsql-hackers@arkaria.postgresql.org; Wed, 08 Jan 2025 21:00:17 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1tVdA0-006yBF-6Q for pgsql-hackers@arkaria.postgresql.org; Wed, 08 Jan 2025 21:00:15 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tVd9z-006yAu-JO for pgsql-hackers@lists.postgresql.org; Wed, 08 Jan 2025 21:00:15 +0000 Received: from mail-qv1-xf2c.google.com ([2607:f8b0:4864:20::f2c]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.96) (envelope-from ) id 1tVd9w-000aKQ-2O for pgsql-hackers@postgresql.org; Wed, 08 Jan 2025 21:00:13 +0000 Received: by mail-qv1-xf2c.google.com with SMTP id 6a1803df08f44-6dccccd429eso2228016d6.3 for ; Wed, 08 Jan 2025 13:00:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lelarge-info.20230601.gappssmtp.com; s=20230601; t=1736370012; x=1736974812; darn=postgresql.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=uOrjgdI+7Cl6M1uYxMd7jl5b96XzjbvP3VrJFg13KNY=; b=cwGfM7Wj00C7pzcm+94iXMCu6RrZmdqwXMIgmEqS7kk5VtyF55+naO4VpEbsWhAYvq E9nkYGHYBMR42IWJ3LyM0AdInpWEl/iAP7GQjw0OGul9Lv6SN3KhO0p5sehQJ/OPvldC 1LmIaP2soYCr5UZSQk10V2zL4s/cLKXBYVjEf1Dik1yg81YUSgdu3kX08ZuGIOAcfZTC GCimtekxHD7ispUeTVtM+e5mDDgxoYUX/ozzAlExUZQMGEksqciHM54T0L3pDJ0ik40g A/UJ69Fj9DJhUxlQWCJBLiI3M7Ufd9gLjfLvLKvYp7ZcM5/beaL0azsZ1OhQ0qK6Jbc0 BRMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736370012; x=1736974812; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=uOrjgdI+7Cl6M1uYxMd7jl5b96XzjbvP3VrJFg13KNY=; b=cqZz527ARiWtmOVhzgn3XxUq2PHr5ZAR7lKDGFZ5zxe1HSQgA9Rh/4iYDA2uyhzI+O /b+/ynciYQ8qKbUFwDEwrs1wD1g6VhCN1uE3EIU1pggoAEEiaZwTT5kOh5HIhVzJZUN9 RUFnKOvQ8Q7Wm4Lpfbep5rObHcI4i7AdQxlqctIJQzIf8HGLV5jDE11eK4u2ph4dMuN3 QKX2JX/kYkUL38IbYXHtL6yeuDu5W46GnaX5XVtqMZ4DYEz8OKmqa6igq0dM/5gC30xT PGC2tHlfQpwd7H8aL8tKR5j4SBBBuOTJTCotk+im2IhQDpal0fbk9CQryzRHbamzxJq+ fCGQ== X-Forwarded-Encrypted: i=1; AJvYcCVjWXMnyFaO2USsQVDrmBRvJ4FjFVknRbamBjHNjNDAU9IQsfygnCis50P9EwA675gkhyR7yHvgzrZxeGb8@postgresql.org X-Gm-Message-State: AOJu0YxmyAl7aZNC2TMukFRp2kXpGvf8wMx4TX0ezA3sMt+imcaEfFO5 3s9T9MsvbTTuOmZNX4Jb9GUFQcfJwVfhXkjDPIkvTQdSrKSnbm61zkaSFtRMpUwwaxF2+1pPCk3 cz0V8F2Z4SUuyj7p15dXvLFcWKdWeKCBl68nzVw== X-Gm-Gg: ASbGnct10HUEaGeTfiAJBrwDMSgjr+KnfZTkHz6UErUUfbCwhWjUfLB1Aek1uDBqd47 kWiCFJP8grCij+SDL3BfQr7c9ZQp0qFwyO9L+B+s= X-Google-Smtp-Source: AGHT+IGuz5wIL6zuTVzc2k8x3f0VS3uz0U2wcO/WvK2ghc0oLTUkY54yEt8/3YZWYcpzsclqks4gBAGKulURdwNBmUs= X-Received: by 2002:a05:6214:f2c:b0:6dd:d24:305c with SMTP id 6a1803df08f44-6df9b2b2c4fmr69905126d6.37.1736370012256; Wed, 08 Jan 2025 13:00:12 -0800 (PST) MIME-Version: 1.0 References: <740696.1718036463@sss.pgh.pa.us> In-Reply-To: From: Guillaume Lelarge Date: Wed, 8 Jan 2025 22:00:02 +0100 X-Gm-Features: AbW1kvYkWCT7XTcMizFv0K0lAHpQdiaQp322NTW39QJw_6GrLs7LwMUO_ItMLJI Message-ID: Subject: Re: Non-text mode for pg_dumpall To: Mahendra Singh Thalor Cc: Nathan Bossart , Magnus Hagander , Tom Lane , Andrew Dunstan , PostgreSQL-development , Dilip Kumar Content-Type: multipart/alternative; boundary="000000000000c7afd6062b3825e9" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --000000000000c7afd6062b3825e9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, Le mer. 8 janv. 2025 =C3=A0 17:41, Mahendra Singh Thalor a =C3=A9crit : > On Wed, 8 Jan 2025 at 20:07, Mahendra Singh Thalor > wrote: > > > > Hi all, > > > > On Wed, 8 Jan 2025 at 00:34, Mahendra Singh Thalor > wrote: > > > > > > On Mon, 6 Jan 2025 at 23:05, Nathan Bossart > wrote: > > > > > > > > On Thu, Jan 02, 2025 at 02:05:13AM +0530, Mahendra Singh Thalor > wrote: > > > > > Here, I am attaching an updated patch. I fixed some bugs of v01 > patch and > > > > > did some code cleanup also. > > > > > > > > Thank you for picking this up! I started to review it, but the > > > > documentation changes didn't build, and a few tests in check-world > are > > > > failing. Would you mind resolving those issues? Also, if you > haven't > > > > already, please add an entry to the next commitfest [0] to ensure > that 1) > > > > this feature is tracked and 2) the automated tests will run. > > > > > > Thanks Nathan for the quick response. > > > > > > I fixed bugs of documentation changes and check-world in the latest > patch. Now docs are building and check-world is passing. > > > > > > I added entry into commitfest for this patch.[0] > > > > > > > > > > > + if (dbfile) > > > > + { > > > > + printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin= , > > > > + dbfile, > create_opts); > > > > + appendPQExpBufferStr(&cmd, " -F d "); > > > > + } > > > > > > > > Have you given any thought to allowing a directory of custom format > files, > > > > as discussed upthread [1]? Perhaps that is better handled as a > follow-up > > > > patch, but it'd be good to understand the plan, anyway. > > > > > > I will make these changes and will test. I will update my findings > after doing some testing. > > > > In the latest patch, I added dump and restoring for > directory/custom/tar/plain formats. Please consider this patch for review > and testing. > > > > Design: > > When we give --format=3Dd|c|t then we are dumping all global sql comman= ds > in global.dat in plain sql format and we are making a map.dat file with > dbname and dboid. For each database, we are making separate subdirectory > with dboid under databases directory and dumping as per archive > format(d|c|t). > > While restoring, first we are restoring all global sql commands from > global.dat and then we are restoring one by one all databases. As we are > supporting --exclude-database with pg_dumpall, the same we are supporting > with pg_restore also to skip restoring on some specified database pattern= s. > > If we want to restore a single database, then we can specided particula= r > subdirectory from the databases folder. To get file name, we refer dbname > into map.file. > > > > TODO: Now I will work on test cases for these new added options to the > pg_dumpall and pg_restore. > > > > Here, I am attaching the v04 patch for testing and review. > > Sorry. My mistake. > v04 was the delta patch on the top of v03. > > Here, I am attaching the v05 patch for testing and review. > > Just FWIW, I did a quick test tonight. It applies cleanly, compiles OK. I did a dump: $ pg_dumpall -Fd -f dir and then a restore (after dropping the databases I had): $ pg_restore -Cd postgres -v dir It worked really well. That's great. Quick thing to fix: you've got this error message: pg_restore: error: -d/--dbanme should be given when using archive dump of pg_dumpall I guess it is --dbname, rather than --dbanme. Of course, it needs much more testing, but this feature would be great to have. Thanks for working on this! --=20 Guillaume. --000000000000c7afd6062b3825e9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

Le=C2=A0mer.= 8 janv. 2025 =C3=A0=C2=A017:41, Mahendra Singh Thalor <mahi6run@gmail.com> a =C3=A9crit=C2=A0:
On Wed, 8 Jan 2025 at = 20:07, Mahendra Singh Thalor <mahi6run@gmail.com> wrote:
>
> Hi all,
>
> On Wed, 8 Jan 2025 at 00:34, Mahendra Singh Thalor <mahi6run@gmail.com> wrote:<= br> > >
> > On Mon, 6 Jan 2025 at 23:05, Nathan Bossart <nathandbossart@gmail.com&g= t; wrote:
> > >
> > > On Thu, Jan 02, 2025 at 02:05:13AM +0530, Mahendra Singh Tha= lor wrote:
> > > > Here, I am attaching an updated patch. I fixed some bug= s of v01 patch and
> > > > did some code cleanup also.
> > >
> > > Thank you for picking this up!=C2=A0 I started to review it,= but the
> > > documentation changes didn't build, and a few tests in c= heck-world are
> > > failing.=C2=A0 Would you mind resolving those issues?=C2=A0 = Also, if you haven't
> > > already, please add an entry to the next commitfest [0] to e= nsure that 1)
> > > this feature is tracked and 2) the automated tests will run.=
> >
> > Thanks Nathan for the quick response.
> >
> > I fixed bugs of documentation changes and check-world in the late= st patch. Now docs are building and check-world is passing.
> >
> > I added entry into commitfest for this patch.[0]
> >
> > >
> > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0if (dbfile)
> > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0{
> > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0prin= tfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin, > > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0dbfile, create_opts);
> > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0appe= ndPQExpBufferStr(&cmd, " -F d ");
> > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0}
> > >
> > > Have you given any thought to allowing a directory of custom= format files,
> > > as discussed upthread [1]?=C2=A0 Perhaps that is better hand= led as a follow-up
> > > patch, but it'd be good to understand the plan, anyway.<= br> > >
> > I will make these changes and will test. I will update my finding= s after doing some testing.
>
> In the latest patch, I added dump and restoring for directory/custom/t= ar/plain formats. Please consider this patch for review and testing.
>
> Design:
> When we give --format=3Dd|c|t then we are dumping all global sql comma= nds in global.dat in plain sql format and we are making a map.dat file with= dbname and dboid. For each database, we are making separate subdirectory w= ith dboid under databases directory and dumping as per archive format(d|c|t= ).
> While restoring, first we are restoring all global sql commands from g= lobal.dat and then we are restoring one by one all databases.=C2=A0 As we a= re supporting --exclude-database with pg_dumpall, the same we are supportin= g with pg_restore also to skip restoring on some specified database pattern= s.
> If we want to restore a single database, then we can specided particul= ar subdirectory from the databases folder. To get file name, we refer dbnam= e into map.file.
>
> TODO: Now I will work on test cases for these new added options to the= pg_dumpall and pg_restore.
>
> Here, I am attaching the v04 patch for testing and review.

Sorry. My mistake.
v04 was the delta patch on the top of v03.

Here, I am attaching the v05 patch for testing and review.


Just FWIW, I did a quick test tonight.= It applies cleanly, compiles OK. I did a dump:

$ = pg_dumpall -Fd -f dir

and then a restore (after dr= opping the databases I had):

$ pg_restore -Cd post= gres -v dir

It worked really well. That's grea= t.

Quick thing to fix: you've got this err= or message:
pg_restore: error: =C2=A0-d/--dbanme should be given = when using archive dump of pg_dumpall

I guess it i= s --dbname, rather than --dbanme.

Of course, it ne= eds much more testing, but this feature would be great to have. Thanks for = working on this!

--
Guillaume.
=
--000000000000c7afd6062b3825e9--