Date: Mon, 13 Mar 2023 16:33:14 +0800
From: Julien Rouhaud <rjuju123@gmail.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Andrew Dunstan <andrew@dunslane.net>,
	Alvaro Herrera <alvherre@alvh.no-ip.org>,
	David Rowley <dgrowleyml@gmail.com>, Peter Geoghegan <pg@bowt.ie>,
	Robert Haas <robertmhaas@gmail.com>,
	pgsql-hackers@lists.postgresql.org,
	Andrew <pgsqlhackers@andrewrepp.com>
Subject: Re: pg_dump versus hash partitioning
Message-ID: <20230313083314.pivmgfu36fi4eu2n@jrouhaud>
References: <1425446.1675291093@sss.pgh.pa.us>
 <CAApHDvqJM5HxhZSnxESENAHLxTZ=eXKKHxeHfvVe1E8v28woDA@mail.gmail.com>
 <1440373.1675299790@sss.pgh.pa.us>
 <a7ef6212-f9fe-6826-9640-5dac54327140@dunslane.net>
 <1533086.1675349554@sss.pgh.pa.us>
 <1912821.1676402493@sss.pgh.pa.us>
 <20230311030154.pg63uspio35h4a6t@jrouhaud>
 <1078804.1678504214@sss.pgh.pa.us>
 <20230311033232.g6jgkuzxdawepsbe@jrouhaud>
 <1602203.1678650412@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1602203.1678650412@sss.pgh.pa.us>
Archived-At: <https://www.postgresql.org/message-id/20230313083314.pivmgfu36fi4eu2n%40jrouhaud>
Precedence: bulk

On Sun, Mar 12, 2023 at 03:46:52PM -0400, Tom Lane wrote:
> Julien Rouhaud <rjuju123@gmail.com> writes:
> > The BEGIN + TRUNCATE is only there to avoid generating WAL records just in case
> > the wal_level is minimal.  I don't remember if that optimization still exists,
> > but if yes we could avoid doing that if the server's wal_level is replica or
> > higher?  That's not perfect but it would help in many cases.
>
> After thinking about this, it seems like a better idea is to skip the
> TRUNCATE if we're doing load-via-partition-root.  In that case it's
> clearly a dangerous thing to do regardless of deadlock worries, since
> it risks discarding previously-loaded data that came over from another
> partition.  (IOW this seems like an independent, pre-existing bug in
> load-via-partition-root mode.)

It's seems quite unlikely to be able to actually truncate already restored data
without eventually going into a deadlock, but it's still possible so agreed.

> The trick is to detect in pg_restore whether pg_dump chose to do
> load-via-partition-root.  If we have a COPY statement we can fairly
> easily examine it to see if the target table is what we expect or
> something else.  However, if the table was dumped as INSERT statements
> it'd be far messier; the INSERTs are not readily accessible from the
> code that needs to make the decision.
>
> What I propose we do about that is further tweak things so that
> load-via-partition-root forces dumping via COPY.  AFAIK the only
> compelling use-case for dump-as-INSERTs is in transferring data
> to a non-Postgres database, which is a context in which dumping
> partitioned tables as such is pretty hopeless anyway.

It seems acceptable to me.

> (I wonder if
> we should have some way to dump all the contents of a partitioned
> table as if it were unpartitioned, to support such migration.)

(this would be nice to have)

> An alternative could be to extend the archive TOC format to record
> directly whether a given TABLE DATA object loads data via partition
> root or normally.  Potentially we could do that without an archive
> format break by defining te->defn for TABLE DATA to be empty for
> normal dumps (as it is now) or something like "-- load via partition root"
> for the load-via-partition-root case.  However, depending on examination
> of the COPY command would already work for the vast majority of existing
> archive files, so I feel like it might be the preferable choice.

Given that this approach wouldn't help with existing dump files (at least if
using COPY, in any case the one using INSERT are doomed), so I'm slightly in
favor of the first approach, and later add an easy and non magic incantation
way to produce dumps that don't depend on partitioning.  It would mean that you
would only be able to produce such dumps using pg16 client binaries, but such
version would also work with older server versions so it doesn't seem like a
huge problem in the long run.